What does an optimal grading distribution look like?

Question

Researchers on education must have thought about this, so I hope to find some directions here. Suppose that I am grading students on a scale from 1 to 10. If I have no prior knowledge of the ability of the students but have obtained their test scores (let's say from 1 to 100).

Suppose these test scores have only ordinal meaning. A student with a higher score can be said to have achieved better mastery of the course but having double the score does not mean that one has achieved double as much as another student. We may also assume that a (statistically speaking) large number of students representative of the student population took the exam.

What should be the optimal grading distribution?

We may also assume that the grading distribution/scale should achieve two goals: a) it should be informative about student's grasp of the material, b) it should incentivize students to study the material.

Regarding a) from an information theory perspective, we may want to maximize the information (entropy) of the grade distribution. Thus, we would choose a scale which yields a uniform distribution. However, in practice most teachers implement distributions which are peaked. What is the motivation behind this?

Answer 1

Assigning grades to fit some "optimal distribution" is misguided. We don't want to maximize the entropy of the grades in a particular course. It's not a very useful measure of a "good" set of grades. To quote from an answer by Anonymous Mathematician:

Strictly speaking, Shannon entropy pays no attention to the distance between scores, just to whether they are exactly equal. I.e., you can have high entropy if every student gets a slightly different score, even if the scores are all very near to each other and thus not useful for distinguishing students.

(Note that this was in answer to a question that asked about using the entropy of exam scores as an indication of how good an exam is at distinguishing between different levels of mastery. It wasn't suggesting that grades should be curved a posteriori to maximize entropy.)

What we actually want is for grades to signal as closely as possible the students' mastery of course material. If every student in the course has achieved truly excellent mastery of the course material, they should all get high scores. This grade then seems to carry very little information. But in reality, it is much more useful to say that all students in this particular class achieved excellence and deserve a 10/10, than it would be to maximize the "information" carried by the grade and give some students a 1/10 because their performance was slightly less excellent than the highest level of excellence achieved by a student that year. This scenario (where all students achieve excellent or very good grades) is not even that unusual, as Michael Covington points out:

In advanced courses, it can be quite proper for all students to get A’s and B’s, because weak students would not take the course in the first place.

For an individual student, the grade should depend on that student's demonstrated mastery of course material, and hopefully not at all (or as little as possible) on the other students in the class.

If you insist on thinking about it from an information theoretic perspective, what we really want is to minimize the Kullback–Leibler divergence between the distribution of students' achievements and distribution of students' grades.

If you have no information about the exam and what it measures, you cannot assign meaningful grades based on that exam score. If you know about the exam, you can assign meaningful grades based on exam scores, but not according to any optimal distribution - you would assign scores based on how much of the exam students are expected to know to demonstrate various levels of mastery, not by mathematically shaping the grades into some predetermined "optimal distribution."

Edit: Suppose these test scores have only ordinal meaning. A student with a higher score can be said to have achieved better mastery of the course but having double the score does not mean that one has achieved double as much as another student.

Your edits are not going to change the answer; there's still not going to be an optimal distribution. If I have many excellent students in the class and I do an exceptionally good job teaching them, I'll give out many excellent grades, no matter how they rank with respect to one another. If all of my students are terrible and do poorly, they'll all get low grades even if one manages to get in a few more points than another (although in that case, I'll also take a closer look at the class to see why students are doing so poorly.) If half of my class excels and the other half fails to meet minimum standards, I'll give out 50% top grades and 50% failing grades. If their abilities happen to be normally distributed, their grades will be as well. You get the idea.

The distribution of student grades should follow the distribution of demonstrated achievements. Any grade distribution that doesn't is definitely not optimal.

---

The idea of grading students to some predetermined distribution (of any shape) is known as "norm-referenced grading." For more information about alternatives to norm-referenced grading, see:

• Sadler, D. Royce. "Interpretations of criteria‐based assessment and grading in higher education." Assessment & Evaluation in Higher Education 30.2 (2005): 175-194.
• Aviles, Christopher B. "Grading with norm-referenced or criterion-referenced measurements: to curve or not to curve, that is the question." Social work education 20.5 (2001): 603-608.
• Rose, Leslie. "Norm-referenced grading in the age of carnegie: why criteria-referenced grading is more consistent with current trends in legal education and how legal writing can lead the way." Journal of the Legal Writing Institute 17 (2011): 123.
• Guskey, Thomas R. "Grading policies that work against standards... and how to fix them." NASSP Bulletin 84.620 (2000): 20-29.

These go into some more detail about problems with norm-referenced grading. You asked for a grade distribution that is "informative about student's grasp of the material." The literature I have cited explains that pure norm-referenced grading is not informative about that; it can only inform about student's relative grasp of the material, compared to others in the same group who have taken the same exam. In other words (emphasis mine):

It can be useful for selective purposes (e.g. for the distribution of a scholarship to the 5 best students, or extra tuition to the 5 which are struggling most), but gives little information about the actual abilities of the candidates.

Source: McAlpine, Mhairi. Principles of assessment. CAA Centre, University of Luton, 2002.

In the middle of the 1990s, the Swedish secondary school grading system was changed from a norm-referenced to a criterion-referenced system. Thus it became possible to compare (for the same population) the ability of a norm-referenced grading system and a criterion-referenced grading system to predict academic success.

The first paper looking at this Swedish data set is not written in English. (Cliffordson, C. (2004). De målrelaterade gymnasiebetygens prognosförmåga. [The Predictive Validity of Goal-Related Grades from Upper Secondary School]. Pedagogisk Forskning i Sverige, 9(2), 129-140.) However, in a later paper, the author describes those results as follows:

Cliffordson (2004b) showed in a study of 1st-year achievement in the Master of Science programs in Engineering that the predictive validity of CRIT-GPA was somewhat higher than it was for NORM-GPA.

In that later study, Cliffordson found (consistent with the earlier study)

a somewhat higher prognostic validity for CRIT-GPA than for NORM-GPA.

and that across a variety of disciplines,

Despite differences in both design and purpose, the predictive efficacy of criterion-referenced grading is at least as good, or indeed is somewhat better, than that of norm-referenced grades.

For details, see:

Cliffordson, Christina. "Differential prediction of study success across academic programs in the Swedish context: The validity of grades and tests as selection instruments for higher education." Educational Assessment 13.1 (2008): 56-75.

Answer 2

Without trying to evade your question, it is extremely difficult to quantify the impact of different grading distributions. This makes it not possible to find an optimal solution. You state 2 goals which makes optimization difficult. Further, your goals are not well defined.

On the surface, the goal of motivating students seems laudable, but we need to be careful about what we are motivating. We do not want to motivate students to strive for higher grades (i.e., grade grubbing). Nor do we want to promote a cutthroat environment where students attempt to increase their grade, by sabotaging fellow students. We want to motivate students, at both the individual and group levels, to increase their understanding. Education research is pretty clear that external motivators, like grades, do not promote the type of learning teachers strive for.

Your second goal is for the grades to be informative. But you do not define whom you are trying to inform. Do we want grades to inform students about their level of understanding, or do we want the grades to inform potential employers. From my understanding of education theory, summative assessments are not part of the learning process while formative assessments are. Further, it is not clear if letter/numerical grades are useful for formative assessments.

Answer 3

The optimal grading distribution looks exactly like the distribution of capability in the field that the members of your class have.

Now, that's not really something you can easily or perfectly measure, but that's your target.

If you could follow your students and grade their performance in later life at a few milestones, you could conceivably estimate the information snr in your grading. That's where you'd apply those concepts meaningfully.

That has nothing to do with the shape of the distribution of grades, however.

You really can't assume any specific shape as valid or not unless you know the distribution of the population.

Answer 4

Years ago I was involved in the grading of entrance exams for a well-known university. Our goal, in grading, was to provide the maximum differentiation: as the purpose of the exam was to find the top candidates, whilst recognizing that for some the physics portion of the exam would only carry limited weight, I was asked to ensure that my grading was sufficiently lenient (this was a VERY hard exam) that the median would be 20/40; this required giving partial credit to partially correct answers, and did in fact produce scores from 2 to 39. Yes, that one paper scoring 39/40 on a really hard exam is something I remember 30 years later...

My point is this: "grades" can mean different things to different target groups. Questions one might ask:

1. Is the student sufficiently prepared to take the "next" course?
2. Does the student merit special recognition (scholarships etc)?
3. Does the student qualify for some "limited access" position (a job, entrance into a graduate program, etc...)

The first point should be "objective", without regard for distribution. That is, an exam should be constructed to test the necessary knowledge, and whether the person has it or not should be independent of the grades of all other students. Studying the distribution is possible helpful to the instructor (are you making "good" exams?) - but ultimately asking 10 hard questions, getting on average 4 good answers, and deciding to set the pass grade at 4+ does not guarantee that people who pass the exam have mastered the material.

The second point relies less on a distribution, and more on a "cutoff". Perhaps there are 5 scholarships: you award them to the top 5 scoring individuals. If you aggregate over multiple exams, you take their rankings on all exams (perhaps eliminating N outliers) to come up with a final ranking.

The final point is the only one where a distribution might be helpful.

Unfortunately, most grading schemes don't follow anything like the common sense approach that I describe...

Answer 5

Edwards Deming, a guru in the quality movement, thought that there should be only three outcomes: "Student masters the material", which is the objective of the course and the main responsibility of the instructor, "Student shows exceptional proficiency" (those he regarded as outliers and would hold out as exceptional achievements), and "Student did not master the material" (outliers at the other end). To students in the first group he would give an A (that would be a large majority), students in the second group would get an A+, and students in the third group would get an F.

Answer 6

I agree with other answers: there is little sense to change your grading paradigm a posteriori until you get some distribution.

However, it can be useful to test your assumptions by formulating your expectations before inspecting the actual statistics. What should the overall distribution look like? What distribution should which individual problems exhibit? Should a certain subset of the problems be accessible to almost all students? How many of the problems do you expect a strong, average, weak student to be able to attempt in the alloted time? Do results on some problems correlate more strongly than they should, i.e. do they really test different skills?

Then you can check if the data fit your expectations. If they do not, you did something wrong. Here are two examples.

Multimodal Distributions

Experience suggests that you probably get something like a Bell curve if there is nothing out of the ordinary going on. In other words, if you do not get such a curve further inquiry may be warranted¹. As a specific example, if you get

you want to look for a criterion that separates the participants into two (or more) groups that explain the two bumps:

Possible criteria (which you can likely check) are course of study, participation in exercises (or some other course-related activity), gender, and so on. You can then take measures to ensure your course works equally well for everybody.

Hardness of a problem is off

What you expect to see is this:

That is, most students solve (very) easy problems well (these should form the baseline for passing the exam, if that), average problems distribute normally (these determine everything between passing and B), and few students solve the hard problems (those that do probably get an A).

If you see big derivations from your expectations, you may have missed the mark when formulating a problem (which can influence your grading decisions) and/or your course did not promote the necessary skills as you thought it would.

---

1. The other direction does not work, i.e. there may be issues you do not see in a histogram.

Answer 7

Your question is really impossible to answer because you don't give us the measurable goals of your grading distributions.

Some sort of Bell Curve (example #1)

This is clearly flawed because it poses no distinction of the mastery of a course nor does it show the success or flaws of the professor. If you were a really good teacher and had really good students a "bell curve" approach would mean that students that actually had a really good grasp of the subject material could get an F.

Let's just say it is a rather simple class and you are an incredible teacher. The grades on a percentage basis are 88-100%. Do the 88% students get Fs, the 90% students get Ds, and so on? And if so why would anyone with any brains take a class like this? I would have to rely on others being dumber, to get a better grade? I would think what if everyone got a 100% except for me and I got a 97%? It makes no sense. (And I know I am being arbitrary using %s as a lot of classes aren't graded like this but just using this for an easy example.

Some sort of Bell Curve (example #2)

Well same situation but this time the class isn't a "normal" class. It is at a university that has 50 openings into their engineering school with 1000 kids taking classes to get in and they take a set number of classes. Well then we might want to grade using some sort of bell distribution because it is comparing the students, not their mastery of the course. After doing this for a few courses you could probably weed out to around 50 students.

Other distribution models

The problem with stating that given a large population that you should have a certain amount of certain grades is really - as mentioned above - about comparing the students to each other and has very little to do with their knowledge of the material of the course.

Even with a simple curve model where you give the best student a 100% and grade from there you could run into some major issues while having a poor rendering of grades. If you were a terrible teacher and students had poor mastery the top student might have a 80%. Do you just bump everyone by 20% and feel good about it?

Also when doing things like that how is this reflected in your grading strategy? For instance what if the first half of your tests are relatively easy and every performs at A level. Then the next half is very very hard. The person getting an 80%, gets bumped to 100%. But then the person who got the first half all right but then missed everything on the harder material... they still get a C?

What do you do?

If your course is part of a core piece of learning and taken by most of the student population you want to work with your school on what they expect their students to get. Possibly look at the historical records for like classes. I am sure an Intro to English class has different scores at upper-tier universities compared to state colleges.

From there set objectives for the students and let them know what is expected. If everyone does what is expected and masters the material, they all get As. If no one shows up for your classes and learns nothing, they all get Fs. The hard part is adjusting the class if you feel that the grading scale is too easy or too hard. But there shouldn't be a fixed distribution. If we are talking English 101 at Harvard we are seeing at least 70-80% As.

For a upper level class a lot of the distribution goes out the window. If you are giving a science class in a specific area you must clearly grade according to how well the student grasped the concepts and materials. There grade should be indicative of how well they can expect to perform if they go to the next level. If you were teaching Quantum Physics and had a class of 10 that were absolutely below average would just give some As and Bs out? Then when they take Quantum Physics II the next professor is like "what the hell, these students aren't ready for this class."

Answer 8

The right distribution to use is the one that the students are used to.

I would dispute your assumption that more entropy is always better than less.

Lets imagine a student who typically receives a B grade, with grades going from A through D, and with F as well and various pluses or minuses, and there is a bell curve concentrating the class's results around a C+

They are content with a B, they are happy with a B+ and so on. They know broadly where they sit relative to their cohort.

Then, Professor Entropy decides to give letters all the way through W, with F still being a fail (F is the lowest grade with W-- being the 2nd lowest) and +/- still being used. The bell curving has been replaced with a flat distribution. The student receives a H+ for their work. This has a lot more entropy, but tells the student almost nothing about how well they did compared to how well they should have done, or how they did in their other subjects (C+, B- and A- using the old system). If the student wants to compare their mark to other marks they have received, they will have to convert the mark you have given them into the same distribution they are used to.

You want to compare the students to each other. They want to compare themselves to a theoretical self that studied a bit harder, or DID make time to go to that party, or moved back to subject ABC instead of XYZ where they recently switched to.

Are you the target audience for the grades, or are the students??

Apologies for lack of citation, but this is as much disputing your premise/assumptions as it is an answer.

Answer 9

My undergraduate course tutor's research specialism was statistical mechanics. He wrote the "grade normalisation" software for my university course which was Physics. It would take students raw trades and spit out the final UK grade (1st, Upper 2nd, Lower 2nd, 3rd, Fail). He told me that the idealised distribution should be a normal distribution; students would be randomised around a mean with more or less ability, applying more or less effort, with more or less distractions of being undergrads (aka over partying), and as those factors were random [*].

My course tutor lamented that the problem was that the actual grade distributions on typical exams was more like an inverted normal distribution; bi-modal with a group of folks flunking, a group of folks acing the exams, and very few people doing "average". This was largely because exams are not perfect and hard to pitch correctly; too hard and folks flunk and too easy and too many folks ace them. So the purpose of the software he wrote was to try to renormalise the individual exams against the student population sitting them. Rather than perfect exams grading a student population the imperfect exams can be normalised against the student population. He was a very smart scientist who had been an educator for decades so I suspect that his intuition and approach was sound.

[*] Of the two brightest guys that I still know two decades later with one got a triple 1st and a college medal, the other flunked out due to over-partying; the friend who flunked his undergraduates Physics recently took a professorial role at a respectable university with a research specialism of international development.

Answer 10

My philosophy professor's grading philosophy was that by very definition, most people are average (i.e. C)

Students asked if he graded on a curve and his response: Yes - if my grades don't fall on a bell curve with the bulk of the class making C, several D's and B's and maybe a couple of A's and F's then I've either made my tests too easy (grades skewed to the A) or too hard (grades skewed to F).

I suppose it depends on if you're trying to push a few of your students to succeed, or if you're trying to help as many students as you can get good grades.