Understanding an abnormal grade distribution

Question

I have three years of experience teaching as part of a team (many teachers, some with more experience, agreeing on a syllabus and preparing the tests together), but this year is the first time I am fully in charge of some courses.

After grading the mid-term exam for one of my class, I noticed I had a weird grade distribution:

(If this is useful, there are 24 grades, the set of grades is { 1.2, 1.4, 1.4, 1.9, 2.0, 2.3, 2.6, 2.6, 3.4, 4.2, 4.2, 4.3, 4.6, 4.6, 4.8, 4.8, 4.9, 5.3, 6.0, 6.2, 6.4, 7.1, 7.8, 7.8 }, the average is 4.25 and the standard deviation is 2.01.)

I have looked carefully at all my previous tests, and I can confirm I have never seen a curve like this before.

In my short experience, I have heard, read of reflected that a distribution with two curves would probably either mean that a) a large subgroup of students cheated or b) as a teacher I am mostly addressing the best students and letting the others down.

But this looks like there are actually three curves, and I am wondering which characteristic of my teaching or my students could explain that.

Besides, if someone is aware of any scholarly work on this subject, that would be lovely. I couldn't find anything myself.

Answer 1

There are several possible factors here: given the relatively small number of points available, lumping can skew how grades are distributed, particularly if they're also awarded in whole number increments. (That is, there's not enough refinement in the model to separate things out.)

Another issue is that the sample size is relatively small; twenty-four students is not a particularly large sample size—your standard deviation here is two points out of 10! Also, you should try plotting the data according to half-integer bins (0.5 to 1.5, 1.5 to 2.5, etc.); you'll end up with a very different distribution.

So, basically, I wouldn't try to draw any definitive conclusions from such a plot or distribution.

Answer 2

I agree with the other answers that this may be an artifact of the histogram. May I humbly offer a few alternative ways to plot these grades?

All of these essentially show that your effects are likely due to small n and possibly an essentially discrete underlying data generating process.

R code:

require(hdrcde)
require(Hmisc)
require(denstrip)
require(beanplot)
require(beeswarm)

grades <- c(1.2, 1.4, 1.4, 1.9, 2.0, 2.3, 2.6, 2.6, 3.4, 4.2, 4.2, 4.3, 4.6, 4.6, 4.8, 4.8, 4.9, 5.3, 6.0, 6.2, 6.4, 7.1, 7.8, 7.8)

opar <- par(mfrow=c(1,6), mar=c(3,2,4,1))
    boxplot(grades, col="gray90", main="Standard\nboxplot",yaxt="n")
    hdr.boxplot(grades, main="HDR\nboxplot",yaxt="n")
    bpplot(grades,xlab="",name=FALSE,main="Box-Percentile\nPlot")
    beanplot(grades,col="grey",yaxt="n",main="Bean plot/\nViolin plot",border="black")
    plot(c(0,2),range(grades),type="n",xaxt="n",yaxt="n",xlab="",ylab="",
        main="Density\nplot")
    denstrip(grades, horiz=FALSE, at=1, width=1)
    beeswarm(grades,pch=19,main="Beeswarm\nplot")
par(opar)

EDIT: (sorry, I'm a statistician, I can't help it...) I went and took Jack's kernel density estimate and resampled 24 "students" from it a couple of times. In each case, I plotted a histogram. The result is below. We see that even an innocuous unimodal curve can lead to pretty bumpy histograms, because of the discretization and the small sample size.

R code:

dens <- density(grades)

opar <- par(mfrow=c(2,4))
for ( ii in 1:8 ) {
    samp <- rnorm(length(grades), sample(grades, size = length(grades), replace = TRUE), dens$bw)
    hist(pmin(10,pmax(0,samp)),breaks=0:10,xlab="",ylab="",main="",col="gray")
}
par(opar)

Answer 3

I did a Kernel density estimate plot of your data, shown below. You have a central lump of candidates with 4-5ish and a second lower lump of students who've done pretty badly.

Answer 4

Just adding to the other statistical analyses here...you can't really be sure that this sample doesn't come from a normally distributed population of grades from similar students in similar classes. Here's some more R code for analyses and their output: x=c(1.2,1.4,1.4,1.9,2.0,2.3,2.6,2.6,3.4,4.2, 4.2,4.3,4.6,4.6,4.8,4.8,4.9,5.3,6.0,6.2,6.4,7.1,7.8,7.8);qqnorm(x);qqline(x)

Compare your grades to: by Skbkekas

Your grades on the left don't fit the QQ line wonderfully, but they're not deviating very systematically. The numbers on the right are from a normal distribution; other than being more numerous, they seem similar.

Your grades are basically not skewed (skew(x) = .12). They are platykurtic, but you don't have enough of them to disregard the possibility that this difference from a normal distribution is due to sampling error with much confidence. Here are results of an Anscombe–Glynn kurtosis test (require(moments);anscombe.test(x)): kurtosis = 2.03, z = -1.23, p = .22. FWIW, you can also test the null hypothesis that your data came from a normally distributed population using a Shapiro–Wilk test (shapiro.test(x): W = .95, p = .27), but normality testing may be 'essentially useless' (this may apply to dedicated significance tests of kurtosis or skewness too).

You seem to refer to the modes or local maxima as curves. @StephanKolassa, @aeismail, and @JackAidley have already demonstrated how misleading histograms can be in this regard. @RedSirius' comment is on-point as well, and you've acknowledged the effect of bin size in your comment, but haven't edited your question to clarify what this doesn't answer for you ^{(hint hint ;) ;)}. It's unclear what more needs to be said here. You haven't got much evidence of anything unusual, much less have you given a serious external basis for your proposed interpretations regarding cheating or uneven service to students of different aptitudes, so it seems further speculation could only grasp at proverbial straws.

However, it may still be worthwhile to recite some (maybe insufficiently examined) academic truisms:

1. It's very hard to make one size fit all when students greatly outnumber instructors.
2. In the case of students who exert near-zero effort, you really can't help enough.
3. Cheating probably won't matter enough either if that's the main form of effort exerted.

Answer 5

Remember that the central limit theorem assumes independent samples. This is often a bad assumption for students. Cheating is of course a possibility, but it could also just be that they study in groups (most people find this very helpful). The spikes in your data could just correspond to groups who study together and have similar strengths and weaknesses.

Answer 6

I have a theory that can explain situational anomalies such as this in classroom that's as situationally accurate as I have time to puzzle out.

To simplify the mathematics of the problem, I'm omitting some scale factors from my math which serve as little more than visual clutter.

Let an ideal bell curve be defined by C(x).

Your set of students is S, and you have a magical function Q(s) for s ∈ S which yields the "quality" of a student's work.

Now consider a test. The test consists of a set of problems (call this set T). Each problem p ∈ T has a difficulty, given by D(p). The probability of a student s correctly answering a problem is defined:

P_correct(s, p) =

It then follows that a student of higher Q than the D of the problem will be more than likely to solve it, and one with a lower Q will be less than likely.

Let's define the ideal score of a student taking a test to be Si(s, T) = Σ Pcorrect(s, p), p ∈ T

If you were to take the ideal score on a particular test for each student in your classroom, you would get an ideal distribution, and chances are that if your students actually took the test, you'd get a distribution that at least roughly approximated the ideal one.

The important thing to take away from what we have so far is that, for an population of students taking a test, the difficulty of the problems on the test mathematically affects the grade distribution you're likely to have.

For example, assuming your student population is roughly bell-curved, you might see a grade distribution like your observations if your test questions have roughly these difficulty levels:

[2, 2, 5, 6, 6, 7, 7, 8, 10, 10+, 10+]

A large number of students would get the 2 easy questions right, but since there are few questions of low-intermediate difficulty, some of the students on the lower end of the curve wouldn't be able to get any harder ones right. On the very high end, there are some questions that might be extraordinarily difficult for the skill level of the students (this can happen for a number of reasons) that most of the class got wrong (assuming the grade is out of 10 points total).

Assuming your class distribution is something like this,

0-2| 
3  | 1
4  | =2
5  | ===4
6  | ====5
7  | ====5
8  | ===4
9  | =2
10 | 1

Their ideal distribution (as defined previously, rounded a bit to reduce clumping) would look something like this:

0  | 
1  |
2  | ===4
3  | ===4
4  | ===4
5  | ===4
6  | ==3
7  | =2
8  | 1
9  |
10 |

Which resembles, in a muted way, the observed curve you experimentally observed.

Also, realistic situations won't have such elegant mathematical solutions (like a student's probability of getting a question correct), so this model should only be viewed as a reasonable, educated approximation.

TL;DR It's possible the questions on this one test were more difficult and less comprehensive than you thought when you handed it out.

Answer 7

If every student were able to solve every question with an independent probability p, then you would expect a normal distribution. But that is actually not a very good model.

Let's say you have a good number of problems in your test that any decent student would be expected to solve. Solving all of them gives you grade 5. Therefore you get lots of students on grade 5; everyone who is doing reasonably well in this course; some are a bit lower because of stupid mistakes which just happen, plus a few who are just badly prepared and have no chance to pass.

Then you have a few really hard problems. The average student solves none of them. Excellent students solve one or two or three, until they run out of time.

That kind of test could produce your distribution even with a very large number of students.

Answer 8

Difficult class scenario with pretty standard student groups:

1. Champions - ambitious and hardworking / clever / interested
2. Mere students - every hour on this subject hurts but I must pass it
3. Underachievers - no time, no effort or no idea how to handle the topic

If the test wasn't important enough or was too difficult, breakdown above explains it all.

Champions didn't manage to max it out but they collected pretty nice scores of around 6-7 points. Mere students managed to learn enough to halve the test. Underachievers found out that without grasping the topic they aren't going to shine here.

Groups 2 and 3 blend together, which is visible in plot from Jack Aidley, hope I can borrow it :)

Is the level high? Was the test well-constructed and offering questions of different difficulty?

If so, then I think that's the case and you just managed to see who your students are. You might want to find out whether the problem is lack of motivation or being unable to jump into the subject.

Answer 9

I think you have over-tailored the difficulty of the questions to the range of abilities in the class. This is a similar theory to @gnasher729's.

Obviously this is all guesswork based on the data, and you will have to decide yourself if it makes sense or not. But it would be consistent with the data if you had two extremely easy questions (which absolutely everyone could solve), three slightly harder, three harder still and two impossible ones which no-one could solve. Everyone fell into one of three levels, with some of the students also making one or two errors on questions which they knew how to solve.

If one point doesn't correspond to one question, than the same thing could still be the case, but with a different amount of questions.

So you tried to spread out the difficulty of the questions (as is normal) but a) you clumped to many questions of the same difficulty together and b) you spread the clumps out too much, with 4 of 10 questions not giving any indication at all as to the relative ability of the students (because everyone got 2 of them, and no-one got the other 2). My guess is that the time-limit of the exam was not a big factor, since people work at different speeds and time limits probably therefore smooth out grades.

There is any easy way to test this theory. Did the students who got 2, 5 and 8 all get the same or very similar sets of questions right? My theory predicts that they did.

Edit

Looking closer at the numbers (this is still guesswork), I would now say that the 3 different ability groups correspond to something like 7.8 questions (with quite a few making one or two points worth of mistakes), 4.8 questions (with most not making many mistakes) and 2.6 (with most making 0.5-1.5 points worth of mistakes).

Answer 10

I believe that part of your issue is how you're doing the rounding for the graph. Instead of rounding up, let's just round to the nearest integer.

This looks far more like a normal distribution, if perhaps a little heavy in the back end. Honestly, rounding like this makes a lot more sense when dealing with a distribution of a class, because without this kind of rounding, something similar to what your graph looks like can crop up. This also explains why many of the non-integer graphs look rather normal as well.