0

At my university, in a Computer Systems course I'm TAing, there are some experiment based assignments in which students are expected to perform experiments, record the results and prepare a report to explain the results.

Post Assignment submission, there's a viva. They're expected to demo the experiments on the same laptops/PCs they did the assignment on.

During the viva, many students are able to reproduce results close to the ones they reported in the assignment submission, while for others there are large deviations.

Slight deviations are okay (and even expected), but large ones are not. Does this constitute cheating?

Clarification:

Demo testing is not running a single instance. Students are expected to report average, standard deviation taken over a very large number of instances of the experiment. On all machines where the TAs have performed the experiments, we have been able to reproduce the results with slight deviations later. That's why I said slight deviations are okay, but
Large deviations from the reported results might mean they've not honestly reported their own results, but instead mentioned copied, slightly-modified results taken from their friends' assignments.

Improve this question
8
  • 1
    I'm confused. What sorts of "experiment"? And if they depend on data, what is the source of it?
    – Buffy
    Jan 18 at 13:52
  • @Buffy An example is measuring how long it takes for compiling a particular (given) C program with different compiler flags
    – whoisit
    Jan 18 at 13:56
  • 1
    Where is the idea that this could be cheating coming from? Your idea? The instructor's?
    – Bryan Krause is on strike ♦
    Jan 18 at 14:33
  • 2
    As a computer scientist, surely you realize that there are all kinds of honest bugs that could manifest like this. Maybe the best test would be if the student is able to explain the discrepancy (but that might turn out to be a lot more difficult than coding the assignment in the first place :)
    – Nate Eldredge
    Jan 18 at 16:26
  • 1
    Hanlon's Razor: Never assume malice when stupidity would suffice. How can you be sure this isn't an 'honest mistake' - the student has mis-copied results/forgotten to update them after improving their code/used the wrong formula/whatever?
    – avid
    Jan 18 at 17:06

4 Answers 4

Reset to default
8

If they can't reproduce the results because they falsified the data, it's cheating. If they can't reproduce the results because they stole the data, it's cheating.

If they can't reproduce the data because they're not good at doing experiments, it's not cheating. If they can't reproduce the data because they're not good at designing experiments, it's not cheating. If they can't reproduce the data because they have a confounding variable they don't know about or don't know how to hold constant, it's not cheating.

Improve this answer
2
  • All of the above non-cheating issues are still reasons to fail the assignment.
    – Brian Borchers
    Jan 19 at 0:32
  • 6
    @BrianBorchers. Can't really comment to that, because I don't know the assignment, but I know investigators who's whole careers have been writing up surprising findings that turned out to be due to confounders.
    – Scott Seidman
    Jan 19 at 1:32
9

I would suggest that in the case of elapsed time as a measure of program efficiency, a single run tells you absolutely nothing, as the load on a modern computer is very (very very) difficult to control, unlike an old IBM 1130. Lots of background processes may be running with no user control (or very little).

That is why we don't measure the efficiency of algorithms in general using elapsed time to run, but with theoretical measures instead (number of comparisons, number of data swaps,...).

This might be especially true of a laptop that is started or awakened in a new (wifi...) environment. The temperature of the CPU might even be a factor. It takes my desktop almost five minutes in the morning to start my browser since it is seldom restarted and the browser needs to swap out other processes to get enough ram to run, especially since the auto-backup program also demands service on awakening.

If time means anything at all (elapsed time), it needs to be averaged over a large number of runs.

That said, if a compiler is poorly written, then the compilation time might differ wildly on similar seeming programs.

So, no, in itself, a slow run in a viva means nothing in itself. But it might indicate lots of things that require further exploration.

To say something explicitly about cheating, I doubt that it is a valid indicator, especially given that the student knows about the viva that will occur. There would be little point and a lot of risk in trying lie or fake something in the written report in that scenario. It might indicate other flaws in the program or the experiment, of course.

Improve this answer
1
  • 2
    Indeed, the assignment is poorly designed at best...
    – Jon Custer
    Jan 18 at 14:30
3

It does not constitute to cheating in and on itself, but it does raise questions in my opinion. The students should be able to explain to you why there are such large deviations, and if they're unable to, it might be worth looking into their work further if you still suspect cheating.

I'm not sure why deviations are expected. (They might be, I'm just not sure what experiments you're conducting.) In my experience, setting seed values for RNGs works very well to ensure that results are consistent across multiple experiments.

Improve this answer
2
  • experiments are not based on RNGs (e.g. measuring how long it takes for compiling a C program with different settings)
    – whoisit
    Jan 18 at 13:07
  • 5
    @whoisit In that case, are you accounting for power plans? If a student times the compilation at submission with their laptop plugged in but does the demo on battery power, there could be significant changes in performance from that alone.
    – Anyon
    Jan 18 at 13:35
3

I'm trying to recall where exactly I got this example from. It may have been in "Code Complete" by Steve McConnell but I don't have a page reference, so I could could be wrong.

The story is, the famous author was being asked to debug a timing issue with a menu system in a major software product. At times it would load quickly, meeting the timing requirements. But at other times it would take ages to load. After some investigation, including using a debugger to crawl through the code, he discovered it was a swap-out issue. If part of the code was not called for a certain time, the system would swap it out of memory. Then, to call the function, the swapped out portion had to be swapped back in. This meant that many meg of code had to be found on the hard drive, swapped into RAM, and then used. And depending on exactly what was swapped in and out, the amount of RAM could vary quite widely. So, getting the File menu to open would, under the "perfect storm" conditions, take 15 seconds.

So you need to be careful about using timing as a metric on code. It requires careful thought to be sure you are not running into such issues. And the reason for changes in timing may be entirely NOT obvious.

Improve this answer
7
  • That is called "thrashing" actually and it is a common issue when memory is a bit tight as in most small systems.
    – Buffy
    Jan 18 at 15:49
  • @Buffy No, sorry, it's not thrashing. Thrashing is when there are competing calls for the same resource from multiple clients. The system begins to spend undue time switching between the clients, and diminishing time actually fulfilling the client requests. This was a single app that got part of itself swapped out.
    – Boba Fit
    Jan 18 at 16:00
  • 2
    My OS class taught me that this is called thrashing. Terminology might have shifted since I was a student, I suppose.
    – Terry Loring
    Jan 18 at 17:05
  • 1
    @Buffy But there was only one call for the resource, from the menu system. there were no competing calls. And it didn't take a long time because it was competing with anything, just because it took a long time to load a big chunk of disk to ram.
    – Boba Fit
    Jan 18 at 20:50
  • 1
    @BryanKrause Note that the definition you linked talks about excessive page faults. The example I gave involved ONE page fault. It's not thrashing.
    – Boba Fit
    Jan 19 at 13:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .