In academic papers, is it really a bad thing to not report the relatively not good results?

Question

I am going to submit a computer science paper to a conference. So the thing is fairly simple. On some datasets with 1:1 ratio, our model achieves really good performance. On datasets with 1:5 or 1:10 ratio, our models only reaches mediocre results.

I just want to report the results of datasets with 1:1 ratio in this case since they are pretty good. Putting a mediocre result (on the 1:5 or 1:10 ratio datasets) may extremely lead to a whole paper getting rejected. However, my PhD advisor told me that we have to put the mediocre result as well.

Meanwhile, the PhD advisor seems really angry about this behavior and disappointed about me, and I feel really sad because I didn't cheat, and I just not want to include the relatively mediocre results in my paper.

I think many other computer science papers do something like that, not reporting their results on some datasets if the result is bad, and reporting their results on some datasets if their result is good.

I don't get why my PhD advisor is so angry about that. Could someone tell me? I am really appreciated it.

Currently, I feel wrongly judged by my PhD advisor and afraid of the whole paper getting rejected because of this since we are going to submit to a top-tier conference.

Answer 1

Research is a search for truth, not for happy talk. If you have evidence of some behavior then you should be honest and report it just as it is. Don't try to dress it up.

Giving only the "best" results will sometimes mislead people and will cause them, perhaps, to waste time and effort.

You are currently the best source to give an honest and complete review of the meaning of your results. I strongly suggest you do that. At some level, not doing that is malfeasance.

I suggest that the reviewers themselves probably think like I do and won't reject a paper that is honest and complete, other things being equal. They aren't looking for happy talk either.

But you may also have an opportunity to investigate why you get different outcomes. Is it something about your model, or something special about the data? Inquiring minds want to know.

Answer 2

This is indeed bad. It is worse than it appears from the title of the question.

Your situation is that the model performs well on a very balanced dataset and ceases to do well as soon as imbalance creeps into the data. This seems to be a preliminary model, because there are so many ways to address the imbalance now. You must implement those and then see how the model works.

Presenting only the results on 1:1 data alone conveys the false picture that the model works well universally. Keep in mind that ML/DL are used very commonly now in different fields, and someone not familiar with the nuances of data balancing may pick up your work and start using it directly. I work in a non-CS area that is starting to adopt ML, and poor models can have very bad real-world impact.

I hope you now understand why the advisor is unhappy. In your situation, I would apologise and consider not submitting if I don't get actual robust results (considering that its a top-tier conference you are looking at).

Answer 3

It's bad because large scale behaviour like this pollutes science literature with bogus results, even if researchers are well intentioned.

It's a large scale problem, specially (but not exclusively) in health sciences, where ineffective treatments can actually kill people. See the Chloroquine and Ivermectin fiasco in the current pandemics.

Even completely "negative" results are valuable, because they put the results of other researchers in perspective, and avoid people getting into dead ends. Unfortunately its still hard to publish "negative" results, and it hurts academia and generates perverse incentives.

The must read article regarding this is Why Most Published Research Findings Are False(1), by John Ioannidis:

Summary: There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

There is a XKCD comic that showcases it nicely:

(1) Ioannidis, John P. A. “Why Most Published Research Findings Are False”. PLOS Medicine, vol. 2, no 8, agosto de 2005, p. e124. PLoS Journals, doi:10.1371/journal.pmed.0020124.

Answer 4

The 'mediocre' results are good too. They tell you under what circumstances you get good performance, so people know that if the data has a 1:1 ratio they can use your model, and if it doesn't then they should do something else. That's helpful - it improves overall performance by limiting use to where it is most useful.

Knowing what leads to good/bad performance also gives information about why the model works as it does, and can lead to insights into how to improve it. If all the results are good (or they're all bad), that doesn't tell you anything about how to improve it. The most informative result is one that shows both.

The purpose of a conference is not to boast about your achievements, but to convey useful information to other researchers. So the more informative you are about what works, what doesn't, and what to look at to improve things next, the better it is for the conference's purposes. Imagine you are explaining your work to somebody thinking about working on it themselves - what would be most useful for them to know? How can you help them? That's what you should be putting in your paper. You are giving them a map showing the areas you have explored - how pleased would they be with you if you only marked in the treasure and left off all the booby traps and pitfalls?

There are two definitions of 'good' here. You are thinking only of whether the model is good - how accurate it is. But you also need to consider whether the research is good, and the paper. This is about how much detail the paper gives about the performance, about when and why it performs well or badly. A paper that reports in detail exactly why a particular type of model doesn't work may be considered excellent from a scientific point of view, and tremendously useful to other researchers. The science can be good, even if the model is bad. Your priority should be to do good science. Producing models with good performance is useful too, but secondary.

Answer 5

There are many answers who address the underlying issues of why such a paper needs publishing all the relevant results and I agree with most of them. I'll simply add a blunt and clear statement of why as a supervisor I would be disheartened or angry with you:

A PhD is not about publishing x papers in the champions league of journals, a PhD is about learning how to be a good scientist! Hiding results for personal fame (publishing in top tier journals) by giving the impression of an awesome breakthrough when you know it is not the case, is an absolute counter-indication of a good scientist.

This attitude leads to the dark side. Next you will omit some "risky" tests right from the start, because you assume they will look bad, select data that you know your algorithm will work good on in the first place or outright doctor it to fit, because, well,you want to show the good results, so why not prepare a test case that fits the strengths of the algorithm, right?! Who cares about the rest when the paper gets accepted.*

From a good supervisor's point of view they see that they failed in teaching you so far and try to correct that now pretty late.

If someone published 10 papers at top journals, but omitted data, and another one did average work on average projects, but following proper scientific standards of full transparent disclosure, I'd take the second one any day over the first when I look for a good scientist. The first one is a self-seller, but they haven't earned their PhD. I'd hire them perhaps for the marketing department, if I'd work at a company that doesn't care about how they sell their stuff from an ethical point of view.

Personally I'd have felt ashamed and would have feared to be expelled had I ever brought up the idea to "just omit" bad results to my PhD supervisor. You don't necessarily need to focus on them, if you want to argue for whatever use-case the test scenario is not so relevant, but you definitely need to report them. I.e. interpretation can have focus, data presentation needs to be complete.

* There is a difference between 1) selecting data sets that match the problem scenario the algorithm tries to solve best and indicating no effort was spent yet on going for other data sets either in the same category (where results are unknown so far) or another category where one might state it could be interesting (potentially perform worse) because the algorithm doesn't fit that data and 2) leaving out data sets because one knows of quirks in the algorithm that make worse performance likely but not mentioning it, 3) already having results for data sets that fit well but omitting them because the algorithm behaves bad on them without even mentioning it and 4) other scenarios I didn't include because they weren't in my mind at the time and it wasn't the purpose of this answer to give a full list (those scenarios might be either okay or not).

Answer 6

Failure to report the mediocre results is unethical

When writing an academic paper, pretending that you did not get bad results for some cases that you were hoping to get good results from is unethical and damaging, both to your credibility/reputation, and to the field, as others may waste time trying to use your code in cases that it doesn't work for.

But take note...

There's a difference between reporting your mediocre results, and emphasising them. Your method works well when the datasets have a 1:1 ratio. This should be the central focus of your paper. Focus the paper on that aspect - your new method is great for those cases!

The mediocre results are simply other results. You have shown that it works well for 1:1 ratios. You have also found that it does not do well for 1:5 ratios. This is also useful information. Make a note of it - you have examined the performance of your approach for cases with 1:5 ratios, and found that it does not perform as well.

Not only is this a way to be ethical while also focusing on the good, but it gives you the opportunity to point towards further research possibilities - why doesn't your method work for 1:5 ratios? This is clearly an area for further investigation! If you know what's causing the problem, then simply point to it and note that further work is needed to extend the approach to work for these cases.

It's to your benefit!

Your first instinct might be to think that people will dismiss your work because you mention some weaknesses. This would be folly. People will see your approach, think of a new way to adapt your approach to other cases, and publish... referencing YOUR paper as they do so. And that increases your citations.

If you don't mention the weaknesses, then it's less likely that people will spend time thinking about how to adapt your approach - instead, they'll dismiss your approach as faulty when it doesn't work for their situations, and ignore it.

Get the balance right

You need to mention the mediocre results, but they shouldn't be at all central to the paper. The trick to this is to frame the paper as focused on the 1:1 ratio cases. It is here that the model works well, and thus you are reporting a new approach to 1:1 ratio cases. When discussing the results of the model, you include a small section discussing what happens for 1:5 and 1:10 ratios - this should not be to the same level of detail as the 1:1 ratio cases, however!

"When applied to more unbalanced ratios, such as 1:5 ratio cases, it is found that the approach does not perform as well, achieving only XXX where other approaches can achieve YYY. This can be seen, for example, in case ZZZ,..."

Answer 7

Do the right thing and put all the results on the paper. It's part of ethics in research, and the work of a researcher is (or should be!) producing new knowledge. Obscuring your results, putting only what looks good is indeed a violation of the ethical code of the research.

On the other side, your supervisor should not be too mad at you. Part of doing research is failing...

If you have such bad results you should also consider submitting the paper to a less prestigious conference, you might get published even without outstanding outcomes.

Answer 8

Putting a mediocre result (on the 1:5 or 1:10 ratio datasets) may extremely lead to a whole paper getting rejected.

That is a secondary problem that is generally a result of bad refereeing or a bad attitude from the journal. If you have developed a method that is good in some narrow area, but mediocre in another area, it still sounds like a useful method and it is still useful to hear about its performance across various tasks. Good academic journals should generally be open to papers talking about "negative results" where we looked for some effect and didn't find it, or developed a new method but it didn't work well on some kind of problem. This is useful information because it allows researchers who follow in your footsteps to know what has already been tried, even if it failed. (Refusal to publish papers like this leads to a result that is somewhat similar to the injunction, "those who do not known history are condemned to repeat it".)

Answer 9

I don't have anything intelligent of my own to add, but I think Richard Feynman pretty much addressed this:

“…the idea that we all hope you have learned in studying science in school — we never explicitly say what this is, but just hope that you catch on by all the examples of scientific investigation.  It is interesting, therefore, to bring it out now and speak of it explicitly.  It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty — a kind of leaning over backwards.

For example, if you’re doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked — to make sure the other fellow can tell they have been eliminated.

Details that could throw doubt on your interpretation must be given, if you know them.  You must do the best you can — if you know anything at all wrong, or possibly wrong — to explain it.  If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it.”

— Richard P. Feynman [Caltech commencement address, 1974, quoted in Surely You're Joking, Mr Feynman!: Adventures of a Curious Character] (my emphasis)

(He was of course talking about scientific theories, not data models; but since both are aiming to expand human knowledge, I think the same principle applies.)

Answer 10

Well, if the question is why your supervisor is unhappy with you, I don't think we can help you, but in general, the situation is not that terrible at all. It is quite common, for example, that a result which is going to be published is impressive only within a specific region of parameters. The model is working here, and fails there and everywhere else, for example. Of course there is a temptation to hide those less impressive sides of the findings, and of course, sometimes people do this in publications (although they should not).

As a non-specialist in CS, I do not know how bad for your publication is what you said about your model, but your supervisor should know it. If they say the results are publishable, they probably are. If not, well, tough luck. The inclusion of mediocre results does not necessarily make your paper worse; a lot depends on the wording and subtle details. Usually, there is more than one way to publish a particular result and to build up a paper around it. As a PhD student, it is totally fine if you are not sure how to present your results better, and this is where your supervisor should be able help you. A honest discussion between you and them (and possibly other co-authors, if any) on how to improve the paper is probably necessary at some point.

Answer 11

I agree with the previous answers that it is unethical. However, even publishing a paper that hides 'mediocre' data can backfire: suppose someone replicates the process but performs those tests you omitted and find that the outcome is mediocre. That person might publish and your paper will be no longer be 'famous'.

Don't cheat to get fame because it won't get you any in the short or long run

Answer 12

It's possible to lie by omission too. You're talking about deliberately leaving out a key fact in order to make it appear that your method works better than it does. Even if everything you say is technically true, you'd still be deliberately deceiving the reviewers in order to manipulate them into signing off on a publication that you think that they wouldn't agree with if they knew the full truth about it.

From the Wikipedia article I linked to:

Lying by omission, also known as a continuing misrepresentation or quote mining, occurs when an important fact is left out in order to foster a misconception. Lying by omission includes the failure to correct pre-existing misconceptions. For example, when the seller of a car declares it has been serviced regularly, but does not mention that a fault was reported during the last service, the seller lies by omission. It may be compared to dissimulation. An omission is when a person tells most of the truth, but leaves out a few key facts that therefore, completely obscures the truth.

Besides, there's no expectation that every method will work equally well in every situation. The fact that an algorithm doesn't work well in certain circumstances is potentially useful information, too (especially if you can identify an interesting reason that that's the case).

Answer 13

Another perspective: I see an XY problem. OP asks if the not reporting the relatively not good results solution is bad.

One could also ask:

Should academia require that results be reported without bias?

or

How did and should academia change to reduce bias in reporting?

or

Some of my results are positive and others are negative. What should I do?

Maybe a paper exploring the question of what to present and why would be better than a paper presenting some or all of the results.

It seems to still be the case or perception in much of academia, that surface-level-success in academia is still facilitated by not reporting the relatively not good results.

In that part of academic publishing that involves reporting human clinical trial* results, there is a push to require pre-registration and subsequent reporting of clinical endpoints, but this is not (!) enforced. In very important trials. The solution doesn't seem to be working when it's most important that it does.

*_{(Not all clinical trials are in humans. Veterinary clinical trials are a thing.)}

How can we make it so, in academic papers, is is not a bad thing to report the relatively not good results?

Now there's a topic worth researching!