8

In computer science, a large portion of people develop algorithms, and demonstrate their effectiveness by running experiments to compare with other existing approaches in their research papers. In these experiments, as far as I know, there are two possible ways to purposely hide data:

  • Ashley has developed algorithm X and decided to experimentally compare it with algorithm Y. She compared them on benchmark set A, B and C. She found that on benchmark C, algorithm Y outperformed X, so she decided to not report on C. In the paper, she also only claims algorithm X outperformed on A and B.

  • Bill has developed algorithm P. He compared P with other algorithms Q, R and S on some benchmark instances. He found that P outperformed Q and R, but not S on these benchmark instances. He decided to not report the comparison between P and S. In the paper, he also only claims algorithm P outperformed Q and R.

Are Ashley and Bill’s actions considered research misconduct?

4
  • "Are Ashley and Bill’s actions considered research misconduct?" Misconduct or detrimental to science? I mean, "misconduct" typically refers to violating some established body of protocols, so while such behavior can be very detrimental to scientific progress, that doesn't necessarily mean that, say, universities or journals object to it. Was this question meant to be more about if such practices are bad for science or if some specific body (e.g. universities or journals) prohibits/discourages it?
    – Nat
    Nov 18 '17 at 23:39
  • To note it, perceptions of this topic vary significantly by field. In higher-stakes contexts, e.g. medical research, not objectively reporting the good-and-bad can result in lives being lost, such that that field's currently receiving more pressure to avoid these tactics. However in lower-stakes contexts, e.g. pure mathematics, there's relatively little call for authors to provide criticism of their own work. CompSci research on algorithms is kinda in-between; people implement algorithms for real-life applications, so it matters, but less directly than medical research.
    – Nat
    Nov 19 '17 at 0:27
  • @Nat Yes, I meant for research misconduct. I'm personally aware of the harmfulness of this practice, but I'm not 100% sure how people in the field accept this.
    – user3294
    Nov 19 '17 at 2:02
  • 2
    These are textbook examples of cherry-picking.
    – Dan Romik
    Nov 19 '17 at 8:51
5

tl;dr- In general, suppressing negative results harms objective analysis but can make the research appear more significant. How people feel about this practice depends on their stake in it.

  • Pure consumers are the most likely to feel that it's misconduct.

  • Pure investees are the most likely to feel that it's acceptable/appropriate.

  • Others in the field likely have mixed feelings since they benefit from the practice as investees but are harmed by it as consumers.

Looking forward, there's a growing understanding that these practices pollute the literature and will need to be weeded out. However, we're not quite there yet.

Pure consumers are likely to consider it misconduct

At one extreme, readers who have no affiliation with the research are the most likely to object. Such readers might be trying to select an algorithm for their own research or business application; they want to know the good and the bad equally, so having the bad omitted is purely detrimental to them.

For example:

Ashley has developed algorithm X and decided to experimentally compare it with algorithm Y. She compared them on benchmark set A, B and C. She found that on benchmark C, algorithm Y outperformed X, so she decided to not report on C. In the paper, she also only claims algorithm X outperformed on A and B.

If a reader is trying to select which algorithm to use, then they'd likely want all relevant benchmarks. Doubly so if Benchmark C is more closely related to their application.

Pure investees are most likely to consider it acceptable/appropriate

At the other extreme, those invested in the research effort itself are most likely to want to see it presented in a positive light. Investees include the researcher themself; their supervisor(s); their institution; and any media services that report on their work (e.g., journals).

Pure investees are those who aren't also consumers. For example, a university's promotional news team is a pretty pure investee, as they basically want to make the research shine.

For example:

Ashley has developed algorithm X and decided to experimentally compare it with algorithm Y. She compared them on benchmark set A, B and C. She found that on benchmark C, algorithm Y outperformed X, so she decided to not report on C. In the paper, she also only claims algorithm X outperformed on A and B.

If Ashley's supervisor feels strongly about the result, they may go into full-promotional-mode, including contacting the university's promotional team and others to advertise the work. Ashley's supervisor and related promoters might wench at any criticism or negative analysis that might detract from their efforts, so they're more likely to appreciate the comparison using Benchmark C not being reported.

Fellow practitioners may have mixed feelings

Other computer scientists may have mixed feelings since they're consumers, but likely engaging in the same behaviors.

In practice, I've seen researchers acknowledge that they understand such behaviors to be detrimental, but still argue that not downplaying/omitting criticism would make their work appear unduly weak compared to other researchers'.

Overall, practitioners seem to generally understand that it's sort-of misconduct in the sense that it shouldn't be done, but that it's acceptable in the sense that authors often feel like they must do it to play on a level playing field with others who do.

4
  • I strongly disagree with the wording of this answer. Although "consumers" will not like the paper, misconduct is a strong word and should at least lead to a formal investigation, retraction of the paper, and possibly to firing of the author(s) in question. This is more in the realm of p-hacking. Personally, I think such "research" is terrible and unethical, but calling it misconduct goes too far.
    – Louic
    Nov 19 '17 at 11:16
  • @louic Definitely, if you're thinking about "misconduct" in sense of formally recognized misconduct. Still, misconduct's a more general concept that applies outside the limited scope of formal proceedings.
    – Nat
    Nov 19 '17 at 13:31
  • Agreed. But we do not know which interpretation of the word OP intended - I just want to make sure it is clear: misconduct or academic misconduct is often used in the context of very serious misbehavior such as data manipulation or plagiarism. With your interpretation of the word "misconduct" this answer is great.
    – Louic
    Nov 19 '17 at 14:33
  • @louic What I meant for misconduct in the question refers to serious misbehavior, and in this context, data manipulation.
    – user3294
    Nov 19 '17 at 23:52
0

Depends on what you want present in the paper and what the methods are.

First, let me divide these comparisons in two groups:

  • Curiosity: There are several "ready to use" methods around, that you spend 5 minutes to fix it to run in your data and see what happens. Sometimes these tests are not even really related to your objective, but they are easy to do, so why not? You don't necessarily need to report everything.

  • Relevant tests: For each problem, in this scenario, there are state of the art solutions (you can't compare your solution if it is the only one). So you need to compare with the state of the art, there is no way around it.

If a paper "skips" on a state of the art mandatory comparison it is not going to get published in a good venue. Any serious reviewer will notice it, raise that question and give it a bad review. Major bad impression...

Personally, I reject papers that only say "we achieve 98% accuracy" without mentioning failing cases and properly explaining why, when, and how the method fails. Which everyone who will consider using the method needs to know... I even have a pre-formatted paragraph saying just that, I use it on almost every review I do... Research is not marketing!

More to the point: IMHO, it is perfectly reasonable to only include the relevant comparisons, regardless of the performance.

3
  • You say: "For each problem, in this scenario, there are state of the art solutions." There are? How do you know? This is only going to work for well-studied problems. Suppose the new algorithm deals with a generalization of a problem that has been studied before. Where do you find the papers on the state-of-the-art solution for that new generalized problem? Suppose an algorithm known to be mediocre for the special case does much better for the general case than anything else, and you don't report it. How do you think the referees are going to catch that? Nov 19 '17 at 19:59
  • @PeterShor by assumption of the problem/question. I agree with your scenario. But, IMHO, in it we would be closer to "new ground" than the scenario the OP is hinting at, where there are several established methods. Nov 19 '17 at 21:56
  • Your comment seems to indicate that there's a clear distinction between new problems and old, well-studied problems. There's not—it's a spectrum, and I would even guess that there are subareas of computer science where most papers fall somewhere in the middle. Dec 2 '17 at 13:01
-1

If the results are reported correctly , then reasons for the varied performance can be evaluated and more progress may be made.

If only selected results are shown then this can cause others to waste time re-evaluating what they think are “curious” results...

IMHO show all the results, as one particular dataset could have a particular variation ie a higher proportion of “odd” numbers which skews the results...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.