3

I am interested in different methods for scaling grades from a class test, say. In this case there is only one test and this gives the whole class grade. I remember from years ago a method that took into account the relative strength of the students taking the class by looking at their grades in other classes. However I can't remember the details at all. The idea was that a hard test taken by strong students would be scaled up more than an easy test taken by weak students.

Is anyone familiar with this idea and how it might work mathematically?


In response to comments let me give an example. Say there is an optional class on Semiotics and Structuralism that the strongest students typically choose to take. The mean for this unit is 60% typically, say. Say there is also another optional class on how to spell the words in Harry Potter book one that the weakest students take. Say this gets the same mean of 60% typically. It would be interesting to know what formulas have been used to scale the former class so that the mean is higher than the latter.

  • 3
    Are you sure that the goal is to judge the students on some larger scale, or is it to assess whether they've learned what the course offers? "Education as filter", or "universities as status-gatekeepers" are a bit ugly, I think. Can you clarify your goals? – paul garrett Jun 12 '18 at 0:56
  • 4
    In some sense this kind of comparison is routinely made, at least implicitly, when comparing CVs. Is it better to publish a lot of papers in mediocre journals or a few papers in strong journals? Is a transcript with solid grades in graduate level courses better than one with superlative grades in undergraduate clases? What is not typically done is to rescale grades within a class based on factors external to that class and its evaluation system. In most university systems such a scheme would be against the rules, and would certainly generate a sense of injustice in the students evaluated. – Dan Fox Jun 12 '18 at 5:27
  • 1
    Common and awful. – Daniel R. Collins Jun 12 '18 at 12:14
  • 4
    Handing out different grades for the same results sounds like a great idea. Your students will love it. – Glen Pierce Jun 12 '18 at 14:04
  • 1
    @Anush: Such practices are mostly peculiar to the US, and mostly limited to forcing grades to conform to some preestablished distribution, which is different from rescaling based on factors completely external to the evaluation of the particular class, such as would be a given student's performance in a different class. For example, such a practice would be plainly against the rules (even the law) in most European countries (I can't speak of other areas simply because of ignorance, but I suspect it would be similarly viewed). Even in the US, such a rescaling would be quite unusual. – Dan Fox Jun 12 '18 at 14:57
6

are you asking about just converting everything into standardized scores?

In which case, you take the mean of relative performance (for example, if you are teaching a senior level course, you might take the major gpa distribution rather than overall gpa distribution) assign everyone a z-score based on their position in the calculated normal distribution.

you do the same for your test, standardize the score. For example, if your test average is a 50 with an SD of 10, a person who got a 70 has a z of 2 while a 45 has a z of -.5.

Now if you are planning on basing their recorded grade based on expected performance you are probably opening up a serious can of worms. For example, a student who consistently scores in the z score range of 1.8-2.1 and then only scores at a 1.5 is given an 75 (assuming mean 80 score, and 10 sd) while a student who consistently scores at a -.5 to -.8 range and scores a zero is given an 85 on the test.

If this is your plan, I would seriously rethink it. Not only will it upset students, it penalizes high performing students and rewards low performers.

  • Thank you. I am not sure I understand " it penalizes high performing students and rewards low performers" though. If strong students choose a particular hard optional class, the idea is that their grades would be scaled up. So this benefits strong students. – Anush Jun 11 '18 at 14:45
  • 3
    @Anush Then you risk being unfair in the other direction. Just because students did poorly in the past, and they did well on this test, it doesn't necessarily mean the test was easy. Especially if the sample size is small. – user37208 Jun 11 '18 at 16:09
  • @user37208 You definitely need at least 50 students. But all scaling comes with risks of unfairness I think. – Anush Jun 11 '18 at 16:13
  • 2
    I think you are best served by piloting your course for a semester and then build up a norm for your tests. The reality is, we as academics are trained in our field, not necessarily in writing a good instrument to measure student understanding and knowledge in that field. I firmly believe that if you plan to put this plan into action, that it might be worth reaching out to the psychometricians in your university's college of education. They will be more able to properly guide you to putting your plan into action with fidelity and high probability of good results. – JWH2006 Jun 11 '18 at 18:11
  • Thanks @JWH2006. To be honest it was more out of intellectual interest as I know they used to do this in some major universities I studied at, but I can't remember what the exact formula was. – Anush Jun 11 '18 at 19:29
3

I strongly recommend to look at this idea from a legal standpoint. At least in Germany, we would just not be allowed to do that. The result in one test may never affect the grading in an other course, even if they are closely related (e.g. I'm teaching Intro to CS and Programming for first semester students, both courses are highly interwoven with each other, sometimes even sharing or exchanging time slots, but the results are completely independent from each otherm which sometimes result in students which have to re-take only one of the courses next year).

An other thought: Students might be interested and high performers in topic A, but dislike topic B. If a good grade in A would affect the grading in B, this would be just unfair.

The only thing you can do (in my opinion) is, to give all of them high grades if they deserve it. But you will need your own mental model of which performance is required for each grade. You should define this prior to the exams and change it only if you made serious misassumptions.

This might have an interesting effect btw: The level in your course will lower over time, since "optimizing" students are recognizing, that all peaople in your corse are getting good grades and so they start taking it ;-). But usually you can demonstrate the required performance level by some short tests on the first weeks or so.

  • That's very interesting and slightly surprising. Something similar to what I have described is standard practice in a number of leading universities (although often carried out without an explicit formula). I wonder if this comes under the category of things that are both impossible and commonplace :) – Anush Jun 13 '18 at 7:52
  • 2
    In fact you are implicitely doing it. I'm teaching an elective course which is only taken by very strong students. The grades are usually either very good, or the students quit after a few weeks because the workload is too high. I'm inviting very good students to select this course because I want to have them in - so in fact I'm pre-selecting them based on prior performance which has a similar effect. But I'm defining in the beginning, what are the requirements for good / very good grades. – OBu Jun 13 '18 at 7:59
  • Yes and at some point, if your university is anything like the ones I know, you will be asked by the department to justify why your students get higher grades in this class than the mean over all classes. There is no way to do this other than showing data that your students are stronger than average by using data from other classes that they have taken. – Anush Jun 13 '18 at 8:02
  • 1
    This is the good thing about German universities - at least for now no-one cares ;-))). But I would respont that they should show me statistical evidence that the individual students are getting better grades in my class then they did in others. And if the effect occurs, that it is not because I'm teching better ;-). That should keep them busy ;-). – OBu Jun 13 '18 at 8:09
  • You make German universities sound great :) Your sensible choice of reply to ask them to give statistical evidence that the individual students are getting better grades in my class then they did in others touches on the same as issue as my question. In the end you have to look at the grades those students got in other classes, which is exactly what the formula I can't remember was meant to address. – Anush Jun 13 '18 at 8:11
2

You need to re-use questions over several years

A small subset of well-vetted questions needs to be reused across years to help distinguish the difficulty of tests from the skill of students. Let's say the mean percentage getting these questions right is 60%. In a year with a strong cohort of students, more than 60% will get them right. If these same (strong) students perform poorly on the other questions, it's a sign that the other questions were too hard, and indicates that the scores should be curved.

NB: care must be taken to avoid these questions being "leaked" to other students. In our course (I teach in a large coordinated course) we have a pool of vetted questions that we shuffle every year to reduce the potential for leaking/cheating.

  • 1
    Thanks for this. I am really referring to a class that always attracts stronger students. – Anush Jun 12 '18 at 18:24
  • @Anush ah -- sorry I see your edit and this doesn't address your clarified question – sessej Jun 12 '18 at 18:35
2

[added after the edit to the question] First of all, this is a phylosophical decision. In some countries / universities / models, votes are intended as an absolute measure (how well does this student know Calculus I with respect to an unwritten golden standard of mathematical expertise?). In others, they are a relative measure (how well does this student know Calculus I with respect to the rest of their class?). From what I understand, the 'relative' thinking is more common in the US, and the 'absolute' one in Europe. You need to decide which one you wish to go for. Both have their merits and their drawbacks.

As for "how to do it in practice", I'm definitely not familiar with it, but you might be referring to models like the polytomous Rasch model.

Basically the idea is that the result of a test is given by f(difficulty of the test, ability of the student) + error, and you can obtain the two independent variables by fitting.

This produces an estimate of the difficulty of each test and the ability of each student, which you can then reuse to rescale things at your will.

There is plenty of software to compute these scores, for instance R and Python packages.

1

"Scaling" to a curve, and adjusting the grade to reflect your expectations of a student's performance are two different things. The first is a common practice, though not without some controversy, and the second is a confirmational bias put into practice, and should be avoided. Every student in a course is entitled to the same treatment.

  • Your last sentence is entirely right of course. The question is, what do departments do in practice with a class where the mean is too high or low? Don't they look at the students who took the class and see how strong they are to work out what actions to take? I suppose if they scale to a curve this never happens. But then the department needs to worry that giving the same mean for every class is unfair on those taking the really hard classes. – Anush Jun 13 '18 at 15:20
  • 1
    @Anush -- no. You don't look at who the students are and figure out what grade to give them. If you give an exam, and the mean grade is 95/100, you may well choose to make that mean grade an A- or some such, but not based on who the students are. – Scott Seidman Jun 13 '18 at 15:23
  • I am not sure what the mapping from numbers to letter grades is, sorry. Is that like scaling the mean to the mean over all classes in the year? – Anush Jun 13 '18 at 15:24
  • At least in universities that I know, if the mean is too high for a class there is a departmental discussion about what to do. One argument for not scaling the mean down that is used is that really strong students took the class. I think you would argue that that shouldn't be taken into account but it is something that is commonly said. – Anush Jun 13 '18 at 15:26

Not the answer you're looking for? Browse other questions tagged or ask your own question.