36

There is a theoretical computer science paper that is published in a very good conference. The paper essentially comes up with an algorithm and an explicit function (mentioned in the paper) of some 20 parameters, f(r1, ..., r20), which ties the performance guarantee of their algorithm to the value generated by this function.

For example, if they find some values r1, r2, ..., r20 where their f function gives a result of 0.8, then their algorithm performance is at least 0.8.

The problem is that in the paper, the authors claim there exists a set of 20 r values where f(r1, ..., r20) >= 0.8. However, they do not write down explicitly in the paper the 20 r values that they used.

I tried using an optimizer to find 20 values where their stated function might indeed give 0.8 but I cannot find any. The closest I got was 0.74. This is not an improvement over prior algorithms. I emailed the authors 2 weeks ago for the 20 r values that they claim to have used, but they did not reply.

My question is, what options do I have? When should I send another email? If they do not respond, do I have any options? To be frank, I'm surprised that none of the reviewers asked for the 20 parameters that they claim to have used to be written explicitly in the paper.

Improve this question
11
  • 17
    There are lots of situations in mathematics (of which theoretical computer science is arguably a branch) where you can prove that an object exists but cannot name it explicitly. There's nothing inherently wrong with that. It is quite possible that the set of values is outside the range you searched, possibly outside the range of values that your program can handle (e.g. they might be trillion-digit numbers for all we know).
    – Nate Eldredge
    Jun 5 at 4:30
  • 4
    So in my opinion, the way to check their result is to read their proof. If you find a gap, then indeed their paper is incorrect. If you find a step that you can't verify, despite having done enough background research that you should be in a position to do so, then start by contacting the authors to ask for further explanation.
    – Nate Eldredge
    Jun 5 at 4:46
  • 25
    @NateEldredge The numbers (ri) are between 0 and 1, so no magnitude issue. My issue is that they say that they found these numbers numerically. I am completely ok with the idea that they have a better optimiser/searching a larger search space/having a better optimizer than I do, but I still can't find values anywhere close to what they're claiming. It also seems strange the authors don't have 10 minutes to copy paste 20 values between 0 and 1 which significantly improve a known bound.
    – user3508551
    Jun 5 at 5:00
  • 4
    If the authors are not responding, then consider asking on pubpeer.
    – Dimitri Vulis
    Jun 6 at 12:39
  • 3
    @ThePhoton The function is smooth, continuous, differentiable, and fairly well behaved.
    – user3508551
    Jun 8 at 6:32

4 Answers 4

Reset to default
33

The conference format historically has not served theoretical computer science well. Some thirty years ago, I heard from well-known practitioners in the field that many published results were false, including sometimes their own. I hope things are better now. So, it is quite possible that there is an error that went unnoticed, because no referee had sufficient time to check the results.

Unfortunately also, making results replicable is still not always standard in CS. I certainly cannot find code for work I did some twenty years ago. This is definitely getting better.

The reward structure of computer science also does not help, as tender loving care of work stops after publication. After publication, the team often goes its own ways, and if the main work was done by a graduate student, that student might become hard to find after graduation. Also, results like those obtained by using an optimizer might not be reproducible because notes were not taken sufficiently accurately.

You end up now with a lot of scenarios that explain why you do not get an answer:

  1. Nobody feels particularly responsible to talk to you as the person that made the discovery is elsewhere.
  2. They genuinely believe in the result, but cannot reproduce it themselves because they forgot to write down the parameters used or because there is a newer version of the software used and the older one is gone.
  3. They are embarrassed at having been found out making a mistake.
  4. They are busy with the end of the quarter or are already in summer mode.
  5. Each author thinks that some other author should answer.

It could also be of course that the fault is entirely yours for using the wrong tool or using it in a naïve manner.

In an ideal world, you would spend more time trying to follow their procedure, and, if you cannot reproduce their result, submit a comment to the same conference. But we are not in an ideal world and conferences are bad at accepting rebuttals and do not have letters to the editor.

Improve this answer
6
  • 17
    If you are a grad student, I would ask someone higher up for help. If they have connections and can forge an introduction your chances of an answer will increase. (It shouldn't be that way, but it's only human to give increased attention to people you know.)
    – Marianne013
    Jun 5 at 11:13
  • 1
    I am curious, in that field, are those known false publications officially retracted?
    – Chuu
    Jun 5 at 16:16
  • 2
    There's also the additional scenario in which this was not a "mistake", but where the authors (or at least one of them) outright invented the 0.8 claim.
    – Dan Romik
    Jun 5 at 17:56
  • 2
    Another possibility is simply that OP's message ended up in the authors' spam folder.
    – Matt
    Jun 6 at 15:14
  • 1
    @Chuu, known false results are not usually retracted in theoretical CS---at least within the conference circus. Polite authors attach comments to their arxiv versions. Others don't..
    – Oxonon
    Jun 7 at 2:56
7

To add to the otherwise excellent answer of @Thomas Schwarz, in general people tend to be extremely slow to react in such cases, if they even react. I have also sent many messages about supposed errors/missing details during my early years, sadly nobody ever answered. If you "need" your result for your research I would say be on the cautious side and assume the "real" result is the one you can reproduce yourself.

Another point, also important, is how old is the paper. The older it is the lower the chance of response for myriads of reasons. A real possibility is that the result in the meantime, correct or wrong, might have been improved by another paper which makes the whole situation "waste of time".

Improve this answer
0

Wait for two more weeks. Two weeks is nothing for busy people.

If you fail to get these parameters in the long run, you might want to publish this as a small technical report, citing the paper with the missing parameters. Before publishing it (or even writing it), talk to your supervisor, as this might be perceived as a hostile act. Also share a draft version with the authors first to give them the change to provide the right parameters. They might be more motivated facing a negative follow-up article.

Improve this answer
7
  • Writing a technical report might work in some cases. However, OP should make sure that the result still stands at this point and it is not obsolete by possibly some other paper(s), otherwise it would seem weird "look I improved this 1990 paper's parameter from 0.8 to 0.74, but since then people know how to go even further to 0.6 or 0.5" (I assume lower values are better)
    – PsySp
    Jun 7 at 8:29
  • 4
    @PsySp It seems like larger numbers are better--the paper is claiming a very good result, better than other existing methods, but the OP is not able to reproduce this very good result. I am guessing something like "we are able to achieve 80% accuracy, which is an advance in the best possible results" vs. "actually you aren't achieving 80% accuracy, but at most 74%, which 1, we already had by other methods, and 2, I wish this did work because I really do need 80% for some important thing that I can't do otherwise".
    – user3067860
    Jun 7 at 13:31
  • 2
    How good are you at optimization problems with 20 floating variables. Optimizations are tricky. Have you done this with starting guesses that completely cover your solution space? In response to your question, it doesn't feel to me like much of a response is required in any case, and scale that to the confidence you hold in your own abilities in optimization.
    – Scott Seidman
    Jun 7 at 16:17
  • @user3067860 Sure, but this doesn't change what I have elaborated in my comment and in my answer.
    – PsySp
    Jun 7 at 16:25
  • 1
    @PsySp Your example does not fit. OP cannot reproduce a result. If OP is not able to trivially get the parameter by himself and they are not given in the paper, what is he supposed to do? Reaching out to the authors gave no answer. Writing the paper informs the scientific community that OP struggled to use the published result. This might triggere a follow-up paper by the original authors or a third group explicitly stating the required parameters. This outcome would serve the scientific community as a whole.
    – usr1234567
    Jun 8 at 14:16
0

Probably the best thing to do is write a research article countering their claim, say they may have been mistaken given your new research, and try to just get the real result out there.

Its better as an opportunity for yourself to show that you can get the right idea. Not everything in science is accurate first time, we need people to notice incorrect claims and then to publish this information as a letter to the editor, or even journal article if there's a lot to say. This does happen even with the most famous scientists, many years later, and has an impact on the field.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .