23

I am currently working (not in academia) after having graduated from college recently. I hope to apply to PhD programs at the end of the year.

I worked with a professor on a paper about a year ago and it got published in a top-5 AI conference. I was hoping to ask them to write my letter of recommendation when I apply for PhD programs. In fact, my first choice would be to work in their lab doing research similar to what I did with them earlier.

I wanted to explore the idea in my paper a little further and see if I could improve it on my own time. My paper presented a new AI algorithm to do a task that performed better than the baseline method. While doing this exploration, I realized that the results obtained in my paper with my novel algorithm could also be obtained with a slightly different, trivial baseline that behaves randomly. The baseline used in the paper behaved uniformly.

So, basically, my novel algorithm does just as well as a baseline method performing randomly. Nothing in the paper that got accepted to the conference is wrong, but it seems like the new algorithm I presented actually isn't all that effective.

What do I do? I'm obviously hesitant to tell the professor I worked with since they would be my strongest recommender and I want to work in their lab. I also know of other groups that want to use this algorithm. Should I tell them that they might as well use this random baseline?

1
  • 11
    Write a blog post about it
    – Andrew
    Feb 19, 2023 at 18:13

5 Answers 5

38

If the results in the paper are correct, I don't see an issue here.

It sometimes happens that a simple algorithm can do the work of a previously known complicated approach.

If you can prove that the simple baseline method outperforms your previous complicated approach, then any reasonable researcher would see this as a net positive.

This is the point of research - you discover something, it works, then you continue exploring it and discover that something else works better. If I were the professor I would laud you for discovering a simpler approach to solve a problem!

8
  • 2
    If bigiftrue does prove this, how would you suggest they publish that result?
    – wizzwizz4
    Feb 19, 2023 at 17:24
  • 2
    So a researcher publishes complex method A which outperforms trivial baseline 1. Then they write another paper that says their own complex method A is bad because it is outperformed by trivial baseline 2, which they probably should have tested before. Not only does this sequence make them look bad, by shooting down their own work, it can also invite accusations of salami slicing (real or not). They published two papers on something that doesn't work.
    – user71659
    Feb 19, 2023 at 21:03
  • 1
    @user71659 you can always accuse someone of something. I could accuse op of having known the algorithm was "bad" and still having published it. Doesn't make it any more true, though.
    – DonQuiKong
    Feb 19, 2023 at 22:45
  • 14
    (+1) You don't look bad by "shooting down your previous work" --- you look bad by clinging to wrong ideas out of pride or embarrassment. I agree with Spark that both findings are laudable and both should be published.
    – Ben
    Feb 20, 2023 at 1:58
  • 1
    No. OP had the wherewithal to do a basic (even trivial) sanity check which "the professor" either forgot to do (not speaking well for his competence) or just conveniently "forgot" to do (not speaking well for his ethics).
    – Deipatrous
    Feb 20, 2023 at 7:46
9

If the original paper was strictly correct, then it should be left alone.

What I would do is continue to investigate your complex algorithm with an eye to determine when it outperforms your second randomized baseline.

Is there some specific type or subset of data where the complex algorithm outperforms baseline 2? What types of inputs break baseline 2, and how does your complex algorithm respond to those? Is there a modification to your algorithm that can make it beat baseline 2 in those cases?

If you can answer those questions, you can publish another paper saying that although simple baseline 2 beat your algorithm, it still works better in these cases or with these changes.

That way you're reporting on improvements and evolution of your algorithm. This is how many things work: AI and artificial neural networks were first proposed in the 1960's and they clearly didn't work very well until multiple decades of advances got them to where they are now.

You also want to report the work in this way to your professor. You can say that it appears our algorithm doesn't work well against baseline 2, but preliminary results show that these changes would fix the problem, and here's a plan for further investigation (if you hire me). That's very impressive as it shows the skills of independence, drive, and "stick-to-it-iveness" needed for a PhD.

Even if your complex algorithm is unsalvageable, you now have a new task: do better than baseline 2. Your next paper could be our complex algorithm A was defeated by baseline 2, but now complex algorithm B does much better than A and 2. If you keep going down paths like this, you now have a career in research.

3

I suspect some of the other answers are (over)generalizing from other fields. In AI, publishing two papers on something that doesn't work is indeed unfortunate. All the worse if the baseline propagated by the second paper is something that you should have tried while writing the first paper. Fairly or not, publishing a second paper with the "trivial baseline that behaves randomly" is a little like saying "Breaking news: I just realized that the algorithm I was hocking last year, and some of you bought, is total snake oil!" Not a great look, even if you are acting in good faith.

I would consider posting an updated version of the paper on the arXiv. Not a new paper, nor an erratum, but simply a new version with a brief new section showing the new results. You can then write to the other researchers who you know are using your method, and point them to this new version. This is probably the cleanest way to fulfill your ethical obligations while drawing minimal attention to this unfortunate incident.

3
  • I don't disagree with this answer, but the impact is not that bad as this answer paints, as OP's first paper had gone through peer review, and supposedly if the random baseline is that trivial it should have been caught there. And even if it's indeed trivial, OP shouldn't feel too bad because at least 3-4 other researchers also miss this (including OP's advisor, the 2-3 reviewers, possibly the area chairs, anyone OP consulted their paper with before publications, etc.).
    – justhalf
    Feb 20, 2023 at 9:52
  • If OP takes the suggested course of action, then I agree the impact is not so bad: OP still has a paper in a top journal, and the record is clear. But if they publish a second paper that essentially nullifies the first, this would be much more noticeable, and the time spent on the second paper would (presumably) impede more impactful work.
    – cag51
    Feb 20, 2023 at 21:41
  • Yep. I guess it depends on how much value OP can get from realizing the existence of this random baseline. It might highlight something in the method that can be improved over this random baseline, in which case, it can produce more impactful work just as well.
    – justhalf
    Feb 21, 2023 at 4:18
2

Write a new paper showing this result

It sounds to me like you have a topic for a valuable new paper. In your previous paper you showed that there is a novel method X that solves a problem better than existing method A. In your next paper you can show that method X unfortunately does not perform better than existing method B, so it does not appear to be effective after all. Both results are publishable and both advance the discipline. Contrary to your question title, both results are still "substantial" --- they introduce a new method and show how it compares with various other methods.

0

Remember that your previous work was examined by peer researchers during the peer review, and if none of them suggested to try the random baseline you described here, then it means either this random baseline is not that common in your field (in which case your random baseline is a new contribution to the field, or at least to make it more known, so that other researchers will always think about using this strong baseline), or that probably it's common, but the reviewers also missed that (in which case, you shouldn't feel too bad, since it's not only you who missed this baseline).

At any case, one thing that you shouldn't do is to not mention this to your advisor. You should definitely mention this and discuss the implication. Sometimes, when a method that seems good gets outperformed by a simple baseline, it indicates that the thought process of coming up with the model might have something that needs fixing. And this may or may not spark new discussions and new models/algorithms.

I understand that having our work be defeated by a simple baseline feels devastating. We spent much of our time proudly working and proving and testing that method to work, and wouldn't like to see that the method is apparently not that good. But that's okay, we can treat this as getting new knowledge that perhaps our the mental model of how we approach the problem might be incomplete. We can think of this as a possible new direction where we might contribute to.

So, in terms of actions, the normal course of action would usually be:

  1. Mention this random baseline with your advisor and discuss it
  2. Think of another new method that improves over this random baseline
  3. In the paper describing this newer method, also mention the fact that this baseline is quite strong and other researcher should use it. (or, if the task is kinda solved with this random baseline, then you've shown that it's not an interesting task, and maybe move on with other more interesting tasks)

Step 2 and 3 would probably be suggested by your advisor too after you do step 1. =)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .