# Understanding an abnormal grade distribution

I have three years of experience teaching as part of a team (many teachers, some with more experience, agreeing on a syllabus and preparing the tests together), but this year is the first time I am fully in charge of some courses.

After grading the mid-term exam for one of my class, I noticed I had a weird grade distribution: (If this is useful, there are 24 grades, the set of grades is { 1.2, 1.4, 1.4, 1.9, 2.0, 2.3, 2.6, 2.6, 3.4, 4.2, 4.2, 4.3, 4.6, 4.6, 4.8, 4.8, 4.9, 5.3, 6.0, 6.2, 6.4, 7.1, 7.8, 7.8 }, the average is 4.25 and the standard deviation is 2.01.)

I have looked carefully at all my previous tests, and I can confirm I have never seen a curve like this before.

In my short experience, I have heard, read of reflected that a distribution with two curves would probably either mean that a) a large subgroup of students cheated or b) as a teacher I am mostly addressing the best students and letting the others down.

But this looks like there are actually three curves, and I am wondering which characteristic of my teaching or my students could explain that.

Besides, if someone is aware of any scholarly work on this subject, that would be lovely. I couldn't find anything myself.

• Thank you for this question that adds content for the academia SO site that is not about how a given supervisor is a mean person or sense of self-worth issues. On the topic, I must say that I'm always amazed when I learn an instructor/prof does not care about grade distribution. – Cape Code Mar 31 '14 at 13:50
• Your bins look too small for size and standard-dev or your dataset. Wikipedia has a few suggestions on bin-size: en.wikipedia.org/wiki/Histogram#Number_of_bins_and_width – Fractional Mar 31 '14 at 14:43
• xkcd.com/1347 (Excuse me, but I couldn't resist.) – Piotr Migdal Mar 31 '14 at 19:54
• It's hard to believe anything particularly unusual is happening when adjusting three students by one SD each (move two from 2 to 4 and one from 8 to 6) gives a single-peaked distribution. – David Richerby Mar 31 '14 at 21:45
• That distribution looks very much like the Teacher's T distribution. In other words, sometimes data will just look weird without meaning there's anything weird going on. – E.P. Apr 2 '14 at 10:56

There are several possible factors here: given the relatively small number of points available, lumping can skew how grades are distributed, particularly if they're also awarded in whole number increments. (That is, there's not enough refinement in the model to separate things out.)

Another issue is that the sample size is relatively small; twenty-four students is not a particularly large sample size—your standard deviation here is two points out of 10! Also, you should try plotting the data according to half-integer bins (0.5 to 1.5, 1.5 to 2.5, etc.); you'll end up with a very different distribution.

So, basically, I wouldn't try to draw any definitive conclusions from such a plot or distribution.

• Thank you for your help. It is true that using half-integer bins gives a different result, although it still looks abnormal. It would still be interesting to know what a larger abnormal distribution would mean. – scozy Mar 31 '14 at 13:38

I agree with the other answers that this may be an artifact of the histogram. May I humbly offer a few alternative ways to plot these grades? All of these essentially show that your effects are likely due to small n and possibly an essentially discrete underlying data generating process.

R code:

``````require(hdrcde)
require(Hmisc)
require(denstrip)
require(beanplot)
require(beeswarm)

grades <- c(1.2, 1.4, 1.4, 1.9, 2.0, 2.3, 2.6, 2.6, 3.4, 4.2, 4.2, 4.3, 4.6, 4.6, 4.8, 4.8, 4.9, 5.3, 6.0, 6.2, 6.4, 7.1, 7.8, 7.8)

opar <- par(mfrow=c(1,6), mar=c(3,2,4,1))
main="Density\nplot")
denstrip(grades, horiz=FALSE, at=1, width=1)
par(opar)
``````

EDIT: (sorry, I'm a statistician, I can't help it...) I went and took Jack's kernel density estimate and resampled 24 "students" from it a couple of times. In each case, I plotted a histogram. The result is below. We see that even an innocuous unimodal curve can lead to pretty bumpy histograms, because of the discretization and the small sample size. R code:

``````dens <- density(grades)

opar <- par(mfrow=c(2,4))
for ( ii in 1:8 ) {
samp <- rnorm(length(grades), sample(grades, size = length(grades), replace = TRUE), dens\$bw)
hist(pmin(10,pmax(0,samp)),breaks=0:10,xlab="",ylab="",main="",col="gray")
}
par(opar)
``````
• `shapiro.test(grades)` does not reject the null hypothesis of a normal distribution, p = 0.27. However, as you write, this can be due to the small sample size. We actually know that the data cannot be normal, since (a) grades are bounded between 0 and 10 (?), whereas the normal distribution is unbounded, and (b) grades are discrete through the points given. Which illustrates why the p value > 0.05 is not "proof" of anything and why statisticians are not too keen on NHST ;-) – Stephan Kolassa Apr 1 '14 at 14:48
• @Raphael: yes, of course every sample is bounded and discrete (including samples from the normal), but in this particular case we know that the population is bounded and discrete, so it cannot be normal. So so be nitpickery, the Shapiro-Wilks test asks a question about the data that we know the answer to. But as long as we don't take the p value too seriously, your bigger point is correct: the data don't look "abnormal" enough to be concerned (although this is really not overfitting, which would be a consequence of fitting too complex a model). – Stephan Kolassa Apr 2 '14 at 6:45
• The data don't look multimodal in a meaningful sense to me. If we, e.g., do a kernel density, multimodality (or not) will depend on the bandwidth we use. The bandwidth `density()` chooses gives a curve that is only very slightly multimodal, see the plot in Jack Aidley's answer. A histogram is in the end very similar to such a kernel density estimator, with a very particular choice of smoothing kernel. YMMV, of course. – Stephan Kolassa Apr 2 '14 at 14:21
• You might be right, but I don't think a box-and-whisker plot is any evidence at all for this. – jwg Apr 2 '14 at 14:44
• +1 to your comment about my boxplot. I couldn't agree more. Boxplots are really too crude, and I don't like them much. My three plots on the right are much more informative. Unfortunately, boxplots are very common, and many people understand them who may need quite a bit of explanation to understand the beanplot or density plot. My personal favorite is really the beeswarm, unless there are too many data points. – Stephan Kolassa Apr 2 '14 at 15:19

I did a Kernel density estimate plot of your data, shown below. You have a central lump of candidates with 4-5ish and a second lower lump of students who've done pretty badly. • Not being a statistician, can I ask you to add a couple words about what kernel density estimates are and how they are computed? Thanks. – Willie Wong Apr 3 '14 at 11:36
• A kernel density estimate is an attempt to derive the distribution from which a sample is derived from that sample. As to how they are calculated? Ask someone better at stats than me, perhaps @Stephen Kolassa – Jack Aidley Apr 3 '14 at 21:50
• A quick intro: en.wikipedia.org/wiki/Kernel_density_estimation. KDE involves a subjective bandwidth (smoothing) parameter, with some rules-of-thumb for automatic selection. Would be interesting to know which software was used to create the graph above. – Daniel R. Collins Jan 21 '17 at 22:25
• @DanielRCollins It was done in R but so long ago I couldn't tell you exactly how I made it. I would likely have used its default settings. – Jack Aidley Jan 21 '17 at 22:26

Just adding to the other statistical analyses here...you can't really be sure that this sample doesn't come from a normally distributed population of grades from similar students in similar classes. Here's some more R code for analyses and their output: `x=c(1.2,1.4,1.4,1.9,2.0,2.3,2.6,2.6,3.4,4.2, 4.2,4.3,4.6,4.6,4.8,4.8,4.9,5.3,6.0,6.2,6.4,7.1,7.8,7.8);qqnorm(x);qqline(x)` Compare your grades to: by Skbkekas

Your grades on the left don't fit the QQ line wonderfully, but they're not deviating very systematically. The numbers on the right are from a normal distribution; other than being more numerous, they seem similar.

Your grades are basically not skewed (`skew(x)` = .12). They are platykurtic, but you don't have enough of them to disregard the possibility that this difference from a normal distribution is due to sampling error with much confidence. Here are results of an Anscombe–Glynn kurtosis test (`require(moments);anscombe.test(x)`): kurtosis = 2.03, z = -1.23, p = .22. FWIW, you can also test the null hypothesis that your data came from a normally distributed population using a Shapiro–Wilk test (`shapiro.test(x)`: W = .95, p = .27), but normality testing may be 'essentially useless' (this may apply to dedicated significance tests of kurtosis or skewness too).

You seem to refer to the modes or local maxima as curves. @StephanKolassa, @aeismail, and @JackAidley have already demonstrated how misleading histograms can be in this regard. @RedSirius' comment is on-point as well, and you've acknowledged the effect of bin size in your comment, but haven't edited your question to clarify what this doesn't answer for you (hint hint ;) ;). It's unclear what more needs to be said here. You haven't got much evidence of anything unusual, much less have you given a serious external basis for your proposed interpretations regarding cheating or uneven service to students of different aptitudes, so it seems further speculation could only grasp at proverbial straws.

However, it may still be worthwhile to recite some (maybe insufficiently examined) academic truisms:

1. It's very hard to make one size fit all when students greatly outnumber instructors.
2. In the case of students who exert near-zero effort, you really can't help enough.
3. Cheating probably won't matter enough either if that's the main form of effort exerted.

Remember that the central limit theorem assumes independent samples. This is often a bad assumption for students. Cheating is of course a possibility, but it could also just be that they study in groups (most people find this very helpful). The spikes in your data could just correspond to groups who study together and have similar strengths and weaknesses.

• It seems to me that the distribution could be an artefact of the test as easily as it is an artefact of the teaching or students. – Jack Aidley Mar 31 '14 at 15:40
• @jack: Yes, of course there are many possible explanations. I just offered one. – Nate Eldredge Mar 31 '14 at 20:12

I have a theory that can explain situational anomalies such as this in classroom that's as situationally accurate as I have time to puzzle out.

To simplify the mathematics of the problem, I'm omitting some scale factors from my math which serve as little more than visual clutter.

Let an ideal bell curve be defined by `C(x)`.

Your set of students is `S`, and you have a magical function `Q(s)` for `s ∈ S` which yields the "quality" of a student's work.

Now consider a test. The test consists of a set of problems (call this set `T`). Each problem `p ∈ T` has a difficulty, given by `D(p)`. The probability of a student `s` correctly answering a problem is defined:

Pcorrect(s, p) = It then follows that a student of higher `Q` than the `D` of the problem will be more than likely to solve it, and one with a lower `Q` will be less than likely.

Let's define the ideal score of a student taking a test to be `Si(s, T) = Σ Pcorrect(s, p), p ∈ T`

If you were to take the ideal score on a particular test for each student in your classroom, you would get an ideal distribution, and chances are that if your students actually took the test, you'd get a distribution that at least roughly approximated the ideal one.

The important thing to take away from what we have so far is that, for an population of students taking a test, the difficulty of the problems on the test mathematically affects the grade distribution you're likely to have.

For example, assuming your student population is roughly bell-curved, you might see a grade distribution like your observations if your test questions have roughly these difficulty levels:

``````[2, 2, 5, 6, 6, 7, 7, 8, 10, 10+, 10+]
``````

A large number of students would get the 2 easy questions right, but since there are few questions of low-intermediate difficulty, some of the students on the lower end of the curve wouldn't be able to get any harder ones right. On the very high end, there are some questions that might be extraordinarily difficult for the skill level of the students (this can happen for a number of reasons) that most of the class got wrong (assuming the grade is out of 10 points total).

Assuming your class distribution is something like this,

``````0-2|
3  | 1
4  | =2
5  | ===4
6  | ====5
7  | ====5
8  | ===4
9  | =2
10 | 1
``````

Their ideal distribution (as defined previously, rounded a bit to reduce clumping) would look something like this:

``````0  |
1  |
2  | ===4
3  | ===4
4  | ===4
5  | ===4
6  | ==3
7  | =2
8  | 1
9  |
10 |
``````

Which resembles, in a muted way, the observed curve you experimentally observed.

Also, realistic situations won't have such elegant mathematical solutions (like a student's probability of getting a question correct), so this model should only be viewed as a reasonable, educated approximation.

TL;DR It's possible the questions on this one test were more difficult and less comprehensive than you thought when you handed it out.

If every student were able to solve every question with an independent probability p, then you would expect a normal distribution. But that is actually not a very good model.

Let's say you have a good number of problems in your test that any decent student would be expected to solve. Solving all of them gives you grade 5. Therefore you get lots of students on grade 5; everyone who is doing reasonably well in this course; some are a bit lower because of stupid mistakes which just happen, plus a few who are just badly prepared and have no chance to pass.

Then you have a few really hard problems. The average student solves none of them. Excellent students solve one or two or three, until they run out of time.

That kind of test could produce your distribution even with a very large number of students.

Difficult class scenario with pretty standard student groups:

1. Champions - ambitious and hardworking / clever / interested
2. Mere students - every hour on this subject hurts but I must pass it
3. Underachievers - no time, no effort or no idea how to handle the topic

If the test wasn't important enough or was too difficult, breakdown above explains it all.

Champions didn't manage to max it out but they collected pretty nice scores of around 6-7 points. Mere students managed to learn enough to halve the test. Underachievers found out that without grasping the topic they aren't going to shine here.

Groups 2 and 3 blend together, which is visible in plot from Jack Aidley, hope I can borrow it :) Is the level high? Was the test well-constructed and offering questions of different difficulty?

If so, then I think that's the case and you just managed to see who your students are. You might want to find out whether the problem is lack of motivation or being unable to jump into the subject.

• When I first served as a TA (in 1996), the principal instructor told me that there were 3 types of student: “the ones who are here because they're interested and want to learn; the ones who are here to get the degree to get a good job; and the ones who are here to keep warm.” It's remarkable how well they map to your categories :) – Emmet Apr 3 '14 at 17:03
• Well said @Emmet, or rather well quoted. However, students occasionally change categories when changing subjects. It's relatively easy for a passionate to become underachiever (or the other way) in a particular area. Sometimes students need a nudge to get fascinated by a subject or their willingness to learn can fade away when the first tasks seem unclear or too difficult. – Legat Apr 3 '14 at 22:01
• I have no respect for students who're just "[there] to keep warm" - if they had even a modicum of drive and self-respect they'd knuckle down and badger their parents into moving to warmer climes. Seriously though, +1 for the psychological side. People near a pass will struggle to make it; those in with a chance to excel will be spurred on. – Tony D Apr 4 '14 at 3:36

I think you have over-tailored the difficulty of the questions to the range of abilities in the class. This is a similar theory to @gnasher729's.

Obviously this is all guesswork based on the data, and you will have to decide yourself if it makes sense or not. But it would be consistent with the data if you had two extremely easy questions (which absolutely everyone could solve), three slightly harder, three harder still and two impossible ones which no-one could solve. Everyone fell into one of three levels, with some of the students also making one or two errors on questions which they knew how to solve.

If one point doesn't correspond to one question, than the same thing could still be the case, but with a different amount of questions.

So you tried to spread out the difficulty of the questions (as is normal) but a) you clumped to many questions of the same difficulty together and b) you spread the clumps out too much, with 4 of 10 questions not giving any indication at all as to the relative ability of the students (because everyone got 2 of them, and no-one got the other 2). My guess is that the time-limit of the exam was not a big factor, since people work at different speeds and time limits probably therefore smooth out grades.

There is any easy way to test this theory. Did the students who got 2, 5 and 8 all get the same or very similar sets of questions right? My theory predicts that they did.

## Edit

Looking closer at the numbers (this is still guesswork), I would now say that the 3 different ability groups correspond to something like 7.8 questions (with quite a few making one or two points worth of mistakes), 4.8 questions (with most not making many mistakes) and 2.6 (with most making 0.5-1.5 points worth of mistakes).

I believe that part of your issue is how you're doing the rounding for the graph. Instead of rounding up, let's just round to the nearest integer. This looks far more like a normal distribution, if perhaps a little heavy in the back end. Honestly, rounding like this makes a lot more sense when dealing with a distribution of a class, because without this kind of rounding, something similar to what your graph looks like can crop up. This also explains why many of the non-integer graphs look rather normal as well.

• Something similar to the OP's graph can crop up with rounding more or less as easily. I think @StephanKolassa has already demonstrated how ordinary this is with histograms of small samples. – Nick Stauner Apr 3 '14 at 0:17