How to measure entropy of exam results

In this answer to a quora question, the answerer mentions how the 'entropy' of a set of exam results can be used to measure how well the exam differentiates between students.

Should I be computing the entropy of my students' exam results? How do I do it? How should I interpret the entropy information?

Edit: How is entropy related to standard deviation?

Entropy measures how much information you learn on average about each student from the exam results. For example, imagine an exam on which everyone gets a perfect score. In that case, you would learn nothing, so the entropy is zero. If half the students get one score and half get another, then you learn one bit of information about each of them. If you want to assign meaningful grades on the usual U.S. scale, you'll need at least several bits of entropy, and the 3.5 or 4 bits mentioned in the quora answer sounds reasonable to me.

The idea behind the answer you link to is perfectly reasonable: if your exam results have low entropy, then that basically means they are clumped together on too few possible scores, and you don't have enough ability to distinguish between students. On the other hand, I don't see much point to actually computing a mathematical measure of entropy (e.g., Shannon entropy), except perhaps for fun if you enjoy that sort of thing. Instead, you can just look at the range of scores and judge how well they distinguish between students. Think about how you might assign grades, and you'll rapidly see whether you run into problems, without any need for mathematical computations.

Furthermore, doing it by entropy is a little subtle anyway. Strictly speaking, Shannon entropy pays no attention to the distance between scores, just to whether they are exactly equal. I.e., you can have high entropy if every student gets a slightly different score, even if the scores are all very near to each other and thus not useful for distinguishing students. The quora answer obliquely refers to that (in the discussion of bins), but still this means you can't just compute a single number without thinking.

So I'd view entropy more as a metaphor than a number most professors should compute.

If 100% of your students mastered the material completely, then you don't want some of them to get very low scores and others very high scores. In this situation, if there is a big spread in scores, it means that your exam is bad, not good.

You really want a whole bunch of simultaneous criteria to be satisfied:

1. Your exam questions have what's known as "face validity." That means that an expert, reading them, agrees that they are written so that they correctly test knowledge of the topic.

2. You want scores on your questions to correlate with one another and/or with external measures of your students' knowledge.

3. You want the test to be reliable, in the sense that for a student with a given level of knowledge, the standard deviation of the test result is small. (E.g., you want a decent number of questions.)

4. If student A and student B have different levels of knowledge, then you want your test to distinguish between them.

If a test has low entropy, it could mean that you have a problem with #4.

The practicalities first. Take your list of exam scores and count for each possible value of the score how many of the exams got that score. In Excel the FREQUENCY function is useful for automating this step. To give some names call p_i the number of exams which have score i. From there you just add up -(p_i)log(p_i) over all the scores that actually happened. The base of the logarithm is not particularly important abstractly but what it does is scale your final "entropy" values so be consistent and only compare different classes when you are using the same base.

So far this is just a computation to perform so what does it tell you? It tells you how much information the scores encode. A test that really differentiates between the levels of knowledge that students have will have values that are take more information to predict. That information might be that your students really know/do not know the material. Or it could be 15 of the questions are easy and 5 very hard so scores in the low-80's are going to be more common than they otherwise might be.

What entropy will tell you is a quantified notion for how much more information is in your exam results than just randomly assigning numbers between 0 and 100. Like any attempt to summarize an entire packet of exams and the attached students with a single number, be careful to not push your data too far. The person in your link that was "surprised how few professors compute and report entropy (or even know what it is)" is a person who is in electrical engineering and computer sciences. Both of those fields use the notion of entropy regularly so his surprise maybe not be that surprising.