39

I am going to submit a computer science paper to a conference. So the thing is fairly simple. On some datasets with 1:1 ratio, our model achieves really good performance. On datasets with 1:5 or 1:10 ratio, our models only reaches mediocre results.

I just want to report the results of datasets with 1:1 ratio in this case since they are pretty good. Putting a mediocre result (on the 1:5 or 1:10 ratio datasets) may extremely lead to a whole paper getting rejected. However, my PhD advisor told me that we have to put the mediocre result as well.

Meanwhile, the PhD advisor seems really angry about this behavior and disappointed about me, and I feel really sad because I didn't cheat, and I just not want to include the relatively mediocre results in my paper.

I think many other computer science papers do something like that, not reporting their results on some datasets if the result is bad, and reporting their results on some datasets if their result is good.

I don't get why my PhD advisor is so angry about that. Could someone tell me? I am really appreciated it.

Currently, I feel wrongly judged by my PhD advisor and afraid of the whole paper getting rejected because of this since we are going to submit to a top-tier conference.

18
  • 74
    What is your objective, trying to win support for your model by presenting selective information or giving an informative and useful analysis of the model? To be more blunt, is the paper intended to be an advertisement for your model or is the paper intended to be a scientific analysis of the model? Apr 24 at 0:11
  • 23
    If you omit the poor performance data, what do you expect will happen when another researcher reads your paper and discovers the method doesn't work on 1:5 or 1:10 ratio data sets? They will publish another paper saying that your method doesn't work. Or they will publish an improvement to your method that does work, and your paper is then a useless bit of history.
    – alephzero
    Apr 24 at 11:29
  • 110
    Finally a supervisor is mentioned on SE who does not try to 'beautify' the data. Finally a supervisor who has an upright ethical standard. OP should be happy they have a supervisor with integrity and honesty. Learn from them, these are unfortunately rarer than one would wish for! Apologize and accept what they say, they are an example for how a scientist should be. You must be absolutely trustworthy in your reporting and your supervisor needs now to be convinced that you are. You have to put the work in to gain their trust back. Apr 24 at 12:05
  • 3
    Report your findings. It's ok if a model works in some situations and not others. That is expected. It will be helpful to future readers to know this so that they don't waste time trying to apply your model to a situation where it is already known to not work. Every famous model has limitations. Imagine if those limitations were kept secret! It's ok to make mistakes, this is what learning is!
    – jdods
    Apr 24 at 21:29
  • 7
    Consider this question posed by a PhD student who found belatedly that their work was based on a fraudulent paper, and spent time fruitlessly trying to replicate its claims: "I am still quite angry about this issue as this has cost me about a year and a half of my PhD": academia.stackexchange.com/questions/166813/… Apr 26 at 23:53

14 Answers 14

126

Research is a search for truth, not for happy talk. If you have evidence of some behavior then you should be honest and report it just as it is. Don't try to dress it up.

Giving only the "best" results will sometimes mislead people and will cause them, perhaps, to waste time and effort.

You are currently the best source to give an honest and complete review of the meaning of your results. I strongly suggest you do that. At some level, not doing that is malfeasance.

I suggest that the reviewers themselves probably think like I do and won't reject a paper that is honest and complete, other things being equal. They aren't looking for happy talk either.

But you may also have an opportunity to investigate why you get different outcomes. Is it something about your model, or something special about the data? Inquiring minds want to know.

1
  • Comments are not for extended discussion; this conversation has been moved to chat. Please read this FAQ before posting another comment. We can only move comments to chat once.
    – Massimo Ortolano
    Apr 25 at 16:56
70

This is indeed bad. It is worse than it appears from the title of the question.

Your situation is that the model performs well on a very balanced dataset and ceases to do well as soon as imbalance creeps into the data. This seems to be a preliminary model, because there are so many ways to address the imbalance now. You must implement those and then see how the model works.

Presenting only the results on 1:1 data alone conveys the false picture that the model works well universally. Keep in mind that ML/DL are used very commonly now in different fields, and someone not familiar with the nuances of data balancing may pick up your work and start using it directly. I work in a non-CS area that is starting to adopt ML, and poor models can have very bad real-world impact.

I hope you now understand why the advisor is unhappy. In your situation, I would apologise and consider not submitting if I don't get actual robust results (considering that its a top-tier conference you are looking at).

2
  • 1
    Apologize for what?
    – yarchik
    Apr 24 at 13:11
  • 43
    @yarchik - For attempting to selectively report results. At best its careless, so no big deal, it is a learning opportunity. Apr 24 at 16:35
54

It's bad because large scale behaviour like this pollutes science literature with bogus results, even if researchers are well intentioned.

It's a large scale problem, specially (but not exclusively) in health sciences, where ineffective treatments can actually kill people. See the Chloroquine and Ivermectin fiasco in the current pandemics.

Even completely "negative" results are valuable, because they put the results of other researchers in perspective, and avoid people getting into dead ends. Unfortunately its still hard to publish "negative" results, and it hurts academia and generates perverse incentives.

The must read article regarding this is Why Most Published Research Findings Are False(1), by John Ioannidis:

Summary: There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

There is a XKCD comic that showcases it nicely:

enter image description here

(1) Ioannidis, John P. A. “Why Most Published Research Findings Are False”. PLOS Medicine, vol. 2, no 8, agosto de 2005, p. e124. PLoS Journals, doi:10.1371/journal.pmed.0020124.

6
  • 11
    +1 especially for "Even completely "negative" results are valuable, because they put the results of other researchers in perspective".
    – Buffy
    Apr 24 at 17:09
  • 1
    The Ioannidis paper is great, but I don't think the xkcd comic is exactly the same thing Apr 24 at 17:52
  • 10
    @AzorAhai-him- Surely, but it showcases one of the problems raised by Ioannidis, how the bias toward only publishing positive results lead people to believe in bogus findings.
    – ksousa
    Apr 24 at 17:54
  • 11
    Loving the xkcd item, actually. Should be "required reading" for every doctoral student doing statistical work.
    – Buffy
    Apr 24 at 18:36
  • 2
    I think the OP's comment that including "mediocre result may lead to [the] whole paper getting rejected" sums up the situation very well. It's basically saying, "but if people knew the whole truth about my work, they wouldn't care!" Apr 26 at 14:26
28

The 'mediocre' results are good too. They tell you under what circumstances you get good performance, so people know that if the data has a 1:1 ratio they can use your model, and if it doesn't then they should do something else. That's helpful - it improves overall performance by limiting use to where it is most useful.

Knowing what leads to good/bad performance also gives information about why the model works as it does, and can lead to insights into how to improve it. If all the results are good (or they're all bad), that doesn't tell you anything about how to improve it. The most informative result is one that shows both.

The purpose of a conference is not to boast about your achievements, but to convey useful information to other researchers. So the more informative you are about what works, what doesn't, and what to look at to improve things next, the better it is for the conference's purposes. Imagine you are explaining your work to somebody thinking about working on it themselves - what would be most useful for them to know? How can you help them? That's what you should be putting in your paper. You are giving them a map showing the areas you have explored - how pleased would they be with you if you only marked in the treasure and left off all the booby traps and pitfalls?

There are two definitions of 'good' here. You are thinking only of whether the model is good - how accurate it is. But you also need to consider whether the research is good, and the paper. This is about how much detail the paper gives about the performance, about when and why it performs well or badly. A paper that reports in detail exactly why a particular type of model doesn't work may be considered excellent from a scientific point of view, and tremendously useful to other researchers. The science can be good, even if the model is bad. Your priority should be to do good science. Producing models with good performance is useful too, but secondary.

1
  • 1
    In many cases, readers would only need to know one thing about many of the negative results: how many there were. If one formulates dozens of hypotheses until one is found that seems to hold up with 98% confidence, knowing how many hypotheses were tested would provide a reader with some clue about whether that 98% result is likely to be significant or spurious, or whether more research is warranted.
    – supercat
    Apr 26 at 23:04
18

There are many answers who address the underlying issues of why such a paper needs publishing all the relevant results and I agree with most of them. I'll simply add a blunt and clear statement of why as a supervisor I would be disheartened or angry with you:

A PhD is not about publishing x papers in the champions league of journals, a PhD is about learning how to be a good scientist! Hiding results for personal fame (publishing in top tier journals) by giving the impression of an awesome breakthrough when you know it is not the case, is an absolute counter-indication of a good scientist.

This attitude leads to the dark side. Next you will omit some "risky" tests right from the start, because you assume they will look bad, select data that you know your algorithm will work good on in the first place or outright doctor it to fit, because, well,you want to show the good results, so why not prepare a test case that fits the strengths of the algorithm, right?! Who cares about the rest when the paper gets accepted.*

From a good supervisor's point of view they see that they failed in teaching you so far and try to correct that now pretty late.

If someone published 10 papers at top journals, but omitted data, and another one did average work on average projects, but following proper scientific standards of full transparent disclosure, I'd take the second one any day over the first when I look for a good scientist. The first one is a self-seller, but they haven't earned their PhD. I'd hire them perhaps for the marketing department, if I'd work at a company that doesn't care about how they sell their stuff from an ethical point of view.

Personally I'd have felt ashamed and would have feared to be expelled had I ever brought up the idea to "just omit" bad results to my PhD supervisor. You don't necessarily need to focus on them, if you want to argue for whatever use-case the test scenario is not so relevant, but you definitely need to report them. I.e. interpretation can have focus, data presentation needs to be complete.

* There is a difference between 1) selecting data sets that match the problem scenario the algorithm tries to solve best and indicating no effort was spent yet on going for other data sets either in the same category (where results are unknown so far) or another category where one might state it could be interesting (potentially perform worse) because the algorithm doesn't fit that data and 2) leaving out data sets because one knows of quirks in the algorithm that make worse performance likely but not mentioning it, 3) already having results for data sets that fit well but omitting them because the algorithm behaves bad on them without even mentioning it and 4) other scenarios I didn't include because they weren't in my mind at the time and it wasn't the purpose of this answer to give a full list (those scenarios might be either okay or not).

2
  • Doesn't whether the advisor is appropriately angry depend on whether the student has had a chance to learn about this aspect of scientific ethics? Really depends on whether this issue has been clearly communicated to the student previously ...
    – Ben Bolker
    Apr 24 at 23:24
  • @BenBolker "why as a supervisor I would be disheartened or angry" "From a good supervisors point of view they see that they failed in teaching you so far and try to correct that now pretty late" should imply that the exact emotion depends on the context (and still might be irrational) and that not only OP would be to "blame", which doesn't change the why. Though it is also disheartening by itself if that is something that is not inherently clear at least on some level, but indeed, then it should be taught ideally. Apr 25 at 7:41
12

Failure to report the mediocre results is unethical

When writing an academic paper, pretending that you did not get bad results for some cases that you were hoping to get good results from is unethical and damaging, both to your credibility/reputation, and to the field, as others may waste time trying to use your code in cases that it doesn't work for.

But take note...

There's a difference between reporting your mediocre results, and emphasising them. Your method works well when the datasets have a 1:1 ratio. This should be the central focus of your paper. Focus the paper on that aspect - your new method is great for those cases!

The mediocre results are simply other results. You have shown that it works well for 1:1 ratios. You have also found that it does not do well for 1:5 ratios. This is also useful information. Make a note of it - you have examined the performance of your approach for cases with 1:5 ratios, and found that it does not perform as well.

Not only is this a way to be ethical while also focusing on the good, but it gives you the opportunity to point towards further research possibilities - why doesn't your method work for 1:5 ratios? This is clearly an area for further investigation! If you know what's causing the problem, then simply point to it and note that further work is needed to extend the approach to work for these cases.

It's to your benefit!

Your first instinct might be to think that people will dismiss your work because you mention some weaknesses. This would be folly. People will see your approach, think of a new way to adapt your approach to other cases, and publish... referencing YOUR paper as they do so. And that increases your citations.

If you don't mention the weaknesses, then it's less likely that people will spend time thinking about how to adapt your approach - instead, they'll dismiss your approach as faulty when it doesn't work for their situations, and ignore it.

Get the balance right

You need to mention the mediocre results, but they shouldn't be at all central to the paper. The trick to this is to frame the paper as focused on the 1:1 ratio cases. It is here that the model works well, and thus you are reporting a new approach to 1:1 ratio cases. When discussing the results of the model, you include a small section discussing what happens for 1:5 and 1:10 ratios - this should not be to the same level of detail as the 1:1 ratio cases, however!

"When applied to more unbalanced ratios, such as 1:5 ratio cases, it is found that the approach does not perform as well, achieving only XXX where other approaches can achieve YYY. This can be seen, for example, in case ZZZ,..."

9

Do the right thing and put all the results on the paper. It's part of ethics in research, and the work of a researcher is (or should be!) producing new knowledge. Obscuring your results, putting only what looks good is indeed a violation of the ethical code of the research.

On the other side, your supervisor should not be too mad at you. Part of doing research is failing...

If you have such bad results you should also consider submitting the paper to a less prestigious conference, you might get published even without outstanding outcomes.

6

Putting a mediocre result (on the 1:5 or 1:10 ratio datasets) may extremely lead to a whole paper getting rejected.

That is a secondary problem that is generally a result of bad refereeing or a bad attitude from the journal. If you have developed a method that is good in some narrow area, but mediocre in another area, it still sounds like a useful method and it is still useful to hear about its performance across various tasks. Good academic journals should generally be open to papers talking about "negative results" where we looked for some effect and didn't find it, or developed a new method but it didn't work well on some kind of problem. This is useful information because it allows researchers who follow in your footsteps to know what has already been tried, even if it failed. (Refusal to publish papers like this leads to a result that is somewhat similar to the injunction, "those who do not known history are condemned to repeat it".)

6

I don't have anything intelligent of my own to add, but I think Richard Feynman pretty much addressed this:

“…the idea that we all hope you have learned in studying science in school — we never explicitly say what this is, but just hope that you catch on by all the examples of scientific investigation.  It is interesting, therefore, to bring it out now and speak of it explicitly.  It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty — a kind of leaning over backwards.

For example, if you’re doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked — to make sure the other fellow can tell they have been eliminated.

Details that could throw doubt on your interpretation must be given, if you know them.  You must do the best you can — if you know anything at all wrong, or possibly wrong — to explain it.  If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it.”

— Richard P. Feynman [Caltech commencement address, 1974, quoted in Surely You're Joking, Mr Feynman!: Adventures of a Curious Character] (my emphasis)

(He was of course talking about scientific theories, not data models; but since both are aiming to expand human knowledge, I think the same principle applies.)

4

Well, if the question is why your supervisor is unhappy with you, I don't think we can help you, but in general, the situation is not that terrible at all. It is quite common, for example, that a result which is going to be published is impressive only within a specific region of parameters. The model is working here, and fails there and everywhere else, for example. Of course there is a temptation to hide those less impressive sides of the findings, and of course, sometimes people do this in publications (although they should not).

As a non-specialist in CS, I do not know how bad for your publication is what you said about your model, but your supervisor should know it. If they say the results are publishable, they probably are. If not, well, tough luck. The inclusion of mediocre results does not necessarily make your paper worse; a lot depends on the wording and subtle details. Usually, there is more than one way to publish a particular result and to build up a paper around it. As a PhD student, it is totally fine if you are not sure how to present your results better, and this is where your supervisor should be able help you. A honest discussion between you and them (and possibly other co-authors, if any) on how to improve the paper is probably necessary at some point.

1

I agree with the previous answers that it is unethical. However, even publishing a paper that hides 'mediocre' data can backfire: suppose someone replicates the process but performs those tests you omitted and find that the outcome is mediocre. That person might publish and your paper will be no longer be 'famous'.

Don't cheat to get fame because it won't get you any in the short or long run

1

It's possible to lie by omission too. You're talking about deliberately leaving out a key fact in order to make it appear that your method works better than it does. Even if everything you say is technically true, you'd still be deliberately deceiving the reviewers in order to manipulate them into signing off on a publication that you think that they wouldn't agree with if they knew the full truth about it.

From the Wikipedia article I linked to:

Lying by omission, also known as a continuing misrepresentation or quote mining, occurs when an important fact is left out in order to foster a misconception. Lying by omission includes the failure to correct pre-existing misconceptions. For example, when the seller of a car declares it has been serviced regularly, but does not mention that a fault was reported during the last service, the seller lies by omission. It may be compared to dissimulation. An omission is when a person tells most of the truth, but leaves out a few key facts that therefore, completely obscures the truth.

Besides, there's no expectation that every method will work equally well in every situation. The fact that an algorithm doesn't work well in certain circumstances is potentially useful information, too (especially if you can identify an interesting reason that that's the case).

1

Another perspective: I see an XY problem. OP asks if the not reporting the relatively not good results solution is bad.

One could also ask:

Should academia require that results be reported without bias?

or

How did and should academia change to reduce bias in reporting?

or

Some of my results are positive and others are negative. What should I do?

Maybe a paper exploring the question of what to present and why would be better than a paper presenting some or all of the results.

It seems to still be the case or perception in much of academia, that surface-level-success in academia is still facilitated by not reporting the relatively not good results.

In that part of academic publishing that involves reporting human clinical trial* results, there is a push to require pre-registration and subsequent reporting of clinical endpoints, but this is not (!) enforced. In very important trials. The solution doesn't seem to be working when it's most important that it does.

*(Not all clinical trials are in humans. Veterinary clinical trials are a thing.)

How can we make it so, in academic papers, is is not a bad thing to report the relatively not good results?

Now there's a topic worth researching!

-1

Your question is mostly on why your supervisor is angry with you.

You acknowledge that you only wanted to include the results when operating on datasets where the model handled them well as you perceive such results as carrying more cachet with conference editors. Your post suggests that you see little wrong with doing this. Your supervisor impressed on you the scientist's obligation to report all work done, the bad as well as the good. He "seems really angry about this behavior and disappointed".

I think that I know why. In every working relationship there is a basic trust demanded. Clearly with your supervisor, he trusted you to appreciate the need to report the full facts as they occurred - not as you or he may have liked them to have occurred. Somehow when he took you on as a PhD student, he thought he could rely on you having that much integrity. How or why he thought he could rely on you, we don't absolutely know. But we can hazard a guess that other things you did - and perhaps to a greater extent how you did them - assured him that you would be sound on data reporting integrity.

I'll be fully honest about what I think here. I believe that you have been playing your supervisor rather than having honest exchanges of view with him. From the outset, I believe that you adopted a nice smooth manner with him and deftly ignored any likely areas of conflict. You picked up from your undergraduate days how academics tend to assume that people with agreeable manner would share their views and standards. In reality, life isn't so simple. Many people with forthright - even tactless - manners can often have very high standards and many with great social finesse show little scruple when their own immediate desires are involved.

Now that you have shown your true ethical colors, the professor realizes he has not been vigilant on the professional character of his student. He must be very angry and disappointed with himself for being weak and falling for your pretences. On top of this his own reputation in the eyes of peers would be stained by association with someone withholding relevant facts in a conference paper.

You may not realize it but your career in research could be nearly over. A single nuance in how a supervisor's reference is phrased could eliminate you from any serious scientific position after your PhD. You have to accept that, notwithstanding the growth of the "executive academic" in the last 30 years, research is still largely a humdrum occupation demanding more of our endurance than of our creativity. If color, glamor, travel to exotic convention capitals is what you seek you need to move into marketing. I know a few PhDs from my own era who did that and are now multi-millionaires. But if - after a period of contemplation - you decide you want to stay in research then you have to learn to take the bad with the good as far as presenting results are concerned.

Lately I had an idea related to a field that has 60 years of global research activity. It was counterintuitive and yet still hard to believe that so many others could have missed it. After a literature check to see if anyone had reported anomalous results, I found a few who had done the very experiments that would bear out what my idea predicted. It would have been easier for these researchers to just not mention these experiments and focus on the more agreeable data from other experiments. (In fact very difficult in the case of one of them, a PhD student who had to present plausible reasons for his observations to some external examiner.) Yet they did not and I appreciate their honesty and integrity.

EDIT

I also note from here that this supervisor took over your programme after 1 year. You should appreciate the fact that someone was prepared to "stand" for you when you changed supervisor. You therefore owe him some consideration even if you see nothing wrong with your own intention.

5
  • 3
    Your third full paragraph here ("I'll be fully honest about what I think here") leaps to a lot of conclusions for which I don't think you have any evidence one way or the other. How can you know this stuff from a post that is only a few hundred words long?
    – Ben Bolker
    Apr 25 at 15:46
  • No one knows how the supervisor-student relation was. But I'm basing my opinion on what I believe based on experience in 6 institutions in 2 countries. You may have other opinions - that is your right.
    – Trunk
    Apr 25 at 16:13
  • I downvoted because I feel that this answer is unhelpful. As Ben pointed out, there are a lot of assumptions here, but even if we take them as given, I find it overly harsh to say the OP is "showing their true colors" and that their advisor must regret working with them in the first place. There's a lot of pressure on PhD students, rather than berating them about being how they are unethical, I'd rather focus on teaching them to be better in the future. Having said that, I agree with the paragraph about how this behavior can affect a research career.
    – Andrew
    Aug 21 at 21:30
  • Expressing disappointment in this situation is fine (I would certainly do so as the advisor), but to me this answer essentially conveys "you showed your true colors and have been completely dishonest the whole time and your advisor will regret ever taking you on as a student and who knows what you will do once you will be released into the world", which is too much, based on the original question.
    – Andrew
    Aug 21 at 21:33
  • @Andrew Let's look at what we do know. OP won't dare ask (as he would if his ignorance were genuine) his professor what's wrong with omitting adverse data. Neither will he go immediately (as he should if his prof was still too tender on the subject) to his HoD or PGT. He comes here to us instead. Of course we have no say in how this will work out. So OP puts assurance of his own conduct ahead of any rational or sensible action to evaluate it. This would concern me. Character is more important than ability.
    – Trunk
    Aug 22 at 11:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.