In my master's thesis work, I am using several algorithmic measures to assess the difference between conditions.

One measure in particular is important to predict 'quality'. This may sound weird in abstraction, but there is a standard written by a number of related institutions in my field, and a reference implementation is given as the standard.

During course work, I was given an implementation of the measure which is written in a different programming language. It is labeled as meas_author, where meas is the name of the measure and author is the author of a well-known book in my field. The book also contains measure implementations that are often cited, in the same programming language as the one I received. I thought the versions were identical.

Now I have found a difference in outputs that is non-trivial (meaning that depending on the implementation, the outcome leads to different results and interpretation). After comparing the code, i noticed small comments like JSCHWARZ-04-04-2012:changed something for speed and that the file structure is different.

Personally, this is a Problem because

  • The coursework I did based on the modified implementation has the same results as a paper from someone with comments in the code, suggesting both are not comparable with results obtained from standard implementations (!)
  • The code may have been used in other coursework, theses, or peer-reviewed publications (some of which received awards) from the people involved or using the measure
  • I need to decide whether or not to keep using the measure
  • I need to talk about this with my thesis supervisor at some point, especially about the flawed course work

Question: How do I handle this issue? I don't want to directly escalate it as I of course could be wrong. I could talk to the person supplying the code first, or to my supervisor. The former would be least drastic, but I'm not sure what I would do if he didn't cooperate/agree this is a problem.

  • 1
    The difference in output could be down to a difference in input. Does the modified programme expect inputs in different units to the original, for example?
    – astronat
    Mar 27 at 9:20
  • @astronat My understanding is that both implementations enforce a standard scaling approach before applying the measure, which excludes differences arising from the input file format and similar issues. Ideally, the measure should be relatively robust against something like that. Mar 27 at 9:33
  • 1
    @astronat I think it would make sense to construct a minimum working example without input differences before I talk to anyone about it. Thank you for the remark! Mar 27 at 9:38
  • 2
    that certainly sounds like a good idea. Remember, ninety nine times out of a hundred, mistakes like this are just that: mistakes. There is likely no nefarious intent and it's not an "escalation" to ask a reasonable question about why your results differ.
    – astronat
    Mar 27 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.