The Science Of Fair Evaluations: Why Simple Averaging Hurts Athletes and How Normalization Fixes it

In youth sports, the evaluation process is an organization’s most critical operational event. It sets the stage for athlete development, parent trust, and organizational reputation. The main goal is a fair, transparent, and objective evaluation for everyone involved.

A popular approach is to use simple averages for each athlete. It is easy to implement, sounds fair, and is already familiar to most people. There are places where it works, but in a real-world evaluation it suffers from a serious flaw.

Not every evaluator uses the same scoring scale in the same way.

On a scale from 1 - 10, one coach might rarely score an athlete above a 7, while another regularly gives scores between 7 and 10. When scores are averaged together, the coach who uses a wider scoring range can have more influence on which athletes rank highest. As a result, final rankings may reflect differences in how coaches score athletes, rather than just differences in athlete performance.

A simple, practical way to fix those flaws is with normalization. The math behind it is not complicated, but the concepts of why and how are not as commonly understood as simple averages are.

The following example demonstrates  how and why SkillShark uses normalization to ensure fair athlete evaluation rankings.

Scenario

Both Jim and Steve will score a set of players on a 1 to 10 scale. Jim, who could be called overly harsh, only scores players between 1 and 7. Steve only scores players from 7 to 10, as he is very nice.

Range

How far apart are the highest and lowest scores provided by each of Jim and Steve?

• Jim's range is 6 (7-1)

• Steve's range is 3 (10-7)

Compare the change from the first row, to the second and third. When Jim gives his highest score, the average moves by 3.0, but when Steve gives his highest, it only moves 1.5.

When you use simple averaging, you treat these two distinct sets of data as if the coaches are speaking the same language, that a 7 out of 10 is always 7 out of 10. This is not the case as Jim's 7 is his highest score, and Steve's 7 is his lowest score.

Although the numbers are identical, they represent different judgements relative to the other athletes each coach evaluated.

Normalization

The same set of scores again from Jim and Steve, but this time we will apply normalization to them.

Normalized = (Score - Minimum) / (Maximum - Minimum)

With normalization you use the range of scores used by both Jim and Steve respectively.

• Jim - (Score - 1) / (7 - 1)

• Steve - (Score - 7) / (10 - 7)

This produces normalized scores in a range of 0 to 1. The table puts them onto a 0 to 10 scale for easier reading.

Those normalized scores look very jarring. Jim and Steve's lowest scores are now 0, and highest is 10. Normalizing has turned Steve's 7s into 0s. At first glance, it can seem like a fair score has been erased, ignored the evaluator's intent, and penalized the athlete. This is not true and is the opposite of what has happened.

The normalized scores have accounted for the difference in scoring range since they account for the entire scale of values that were observed. Jim and Steve now influence the total and average equally. Any hidden imbalance is now accounted for and this is the better representation of both evaluators.

It defines excellence by the best actual performance observed, not the abstract idea that the same value in the same overall scale means the same thing to Jim and Steve.

Complications In Real Evaluations

The scenario above illustrates the core difference in range between simple averages and normalization, namely how hidden imbalances are accounted for or not. If you ran thousands of simulated evaluations, you would see normalization cause shifts in ranks compared to the simple averages. Arguing that the differences from normalization are errors misses the entire point. Normalization is the fix, not the flaw. It changes rankings because it actively strips away that hidden spread, instead of allowing it to distort simple averaged rankings.

The scenario only used subjective 0 to 10 measurements. A real evaluation will contain many more measurements, differing scales, and even objective measurements like a timed drill measured in seconds. This is to say nothing of tons more evaluators, athletes, age groups, or holes present in the data. The real thing is, sadly, much messier.

By applying normalization, different ranges are accounted for  and all measurements are adjusted from different scales to a common one. Simple averaging does not account for those differences, meaning they can distort final rankings and team selection decisions.

Normalizing is not perfect or without pitfalls, and the method above is sensitive to extreme scores, flukes, or simple data entry mistakes. It must be paired with guardrails such as a fixed scale of 1 to 10 for the scores or detection of inputs dramatically outside the expected range.

Objective Measurements

A timed drill will provide an objective measurement in seconds. The top and bottom performers may only be separated by 0.5 seconds or less. For timed drills where lower numbers are better, normalization can invert the formula so that the fastest time maps to 10 and the slowest maps to 0. The units pose enough of a problem on their own, and with such a compressed measurement of time, the real observations are vulnerable to being drowned out by simple averaging.

Championing that only objective measurements should be normalized, but not subjective measurements, is a logical contradiction. An objective measure will still have a different range compared to any other measurement, subjective or objective.

By refusing to normalize all measurements, simple averaging tries to protect a scoring range under the illusion it still means what it was originally set to mean. Above, we saw it doesn't account for that anyway; trying to tweak only objective data afterwards just makes it all worse. The differing ranges have to be accounted for correctly no matter what.

Weighting

Through normalization, with a shared scale from 0 to 10 across every measurement, different weights can be applied explicitly and easily. This provides powerful tools to highlight inherent difficulty gaps or philosophies in the measurements based on what you really want to evaluate.

The example used evaluators, but if two different measurements scored by both Jim and Steve needed a 60%/40% split, multiplying each total or average value by 0.6 and 0.4 respectively works in either case. Normalized values produce correct results given the shared range within and between the two measurements. Simple averages, without having accounted for any range, are warped even worse than before. The higher the spread in one set of scores to another, the more your intended percentages will be overridden.

How it Applies Using SkillShark

On the surface, a simple average tries to respect a chosen scale, retain judgments or intents, and produce fair results.

It falls apart in doing any of those due to not taking the spread in the score sets into consideration. Pen and paper evaluations are susceptible to this given they often start with a simple spreadsheet, and as they grow in complexity, this core shortcoming is not addressed.

Over time, the wrong cuts are made, trust erodes in the youth sports organization, and worst of all, the athletes themselves suffer. They are put into tiers too high or too low, which stifles their confidence and development. Then they stop playing.

Frustratingly, some evaluation software still perpetuates these basic statistics problems.

We built SkillShark to help fix that. We utilize normalization as described above for all the reasons outlined here so that the measurements you gather with real people are presented and understood as accurate, making evaluations transparent, fair, and objective for everyone.