21

I was reading a conference paper relevant to my area of study, and I found it interesting - they did what I was planning to do anyways, and had some extremely promising results.

Now, I don't really believe their claims, as the success seems a bit extreme (300-400x speed up for an algorithm by switching from CPU to GPU). It feels like they compared optimized GPU code to an unoptimized CPU algorithm. Nevertheless, I am in general a reasonable person, so I wanted to see what they did and how they did it.

Unfortunately, the paper is very vague on the implementation used. There is a snippet of pseudocode that doesn't really share anything new (it is obvious). I emailed the authors and asked them for source code, if at all possible. The first author replied to me, and provided a GitHub link. The code in the repository is undocumented, and isn't exactly for the implementation I asked about anyways. It is for another implementation they did (also distributed computing).

I tried contacting the author with some questions, but he has never responded to me again... So that link has not been successful.

The paper doesn't have many citations, and all of them but one are self-references. I haven't found evidence that anyone built on these findings so far.

So: I figured I would just move ahead with my idea and not rely on their implementation. However, do I need to address this paper somehow in my thesis? I assume so - the committee could ask me about it, and how my work is different. But how can I compare my work to something else that I have no source code for? Do I take their word for it and compare my results to their published results? Am I allowed to criticize work that provides no supporting code? Probably not...

Should I pester the authors some more? They are probably sick of me by now.

  • 2
    Just an anecdote, but 300-400x speed up going from CPU to GPU is common for many problems, provided they are sufficiently parallelizable. If the algorithm can be translated to a series of matrix multiplies with little or no branching then chances are the GPU will smoke the CPU every time. This is or is at least becoming pretty well known in programming circles, cutting across a lot of domains, genetics, finance, etc. Doesn't mean the paper is accurate, and papers should provide implementations but the magnitude of their results alone doesn't raise alarm bells at all. – Joseph Garvin Apr 30 '15 at 15:59
  • @JosephGarvin, I could see that being the case with a highly parallelizable algorithm, but this one is not. The reason it is not normally done in parallel is because there is no uniform pattern for it - there isn't a big matrix to work with. There are lots of cross-iteration dependencies, which in my opinion would easily prevent this 300-400x speed-up. Still, what do you think of the Intel white paper Debunking the 100x speed-up myth? Not attacking your point of view, just curious what you think. – Mewa Apr 30 '15 at 16:37
  • "It feels like they compared optimized GPU code to an unoptimized CPU algorithm." And now you know the dirty little secret of the GPU fan club. (So yes, this very well should raise alarm bells.) – user4512 Apr 30 '15 at 22:20
  • I'd like to read the paper, but I don't have membership. That said, I am skeptical because the abstract does not mention the cost of adding a GPU versus a CPU. Apparently they don't even try multiple card or CPU models. You have to compare hardware at similar price points, and a quick search of benchmarks for the GTX suggests it was not at the same pricing tier when the paper was published, the CPU was 4x more expensive, and a multiple GPU motherboard will be much cheaper than a multiple CPU motherboard. If they didn't think of these things, what are the odds they actually measured correctly? – Joseph Garvin May 1 '15 at 20:29
  • @JosephGarvin, I think this version is open-access. I agree in general, and I think so did Nvidia, given the publicity mess that the paper resulted in. Furthermore, written by Intel engineers, the CPU code was probably intensely optimized and the GPU might not have been. However, all the problems that people have with that paper, I think, outline a big issue with optimization papers in general - it is difficult to tell if the CPU and GPU implementations were on the same level. Thanks for the response! – Mewa May 1 '15 at 21:01
33

In your thesis, I think it should be safe to say something along the lines of

"John Doe et. al. claim in [22] that the algorithm would run 400X faster with their implementation, however, the details are unclear."

and leave it there. If you are asked more about it in your defense, you may politely state that you tried to approach the authors for more information but could not get it.

  • Definitely my line of action on these cases in my field, but over time it tends to attract animosity. Said authors will eventually read and might take that 'exposure' as a personal attack. What most people do, they just ignore that work to avoid bashing it, which I find bad for the literature. – Scientist Jan 23 '18 at 2:41
13

First off, let me give you some context and help you set expectations. It's very common in computer science that authors are not willing to share their code (for any number of reasons). I'm not saying this is a good thing; I'm just saying this is how it is.

If they gave you a pointer to a Github repository containing their code, that is far above average. You should consider yourself lucky to have their code. Undocumented? Par for the course -- it's research code, you're lucky that they're giving you their code at all. Now if the code they sent is you is not the code they used for this paper, but code for something entirely different, that's a different matter -- then the code may not be useful. However, the question is not entirely clear on this point, so I'm not in a position to judge how relevant what they sent you was.

If they sent you their code for this paper, and if it's highly relevant to your thesis, you should study it carefully. It's your job to be an expert in this area. You need to study their paper and their code to understand the basis for those results, and see if you can replicate them. You need to make every effort to try to understand what's going on with that paper. If someone asks you about it and you respond that you asked for their code but the code they gave you had no documentation so you didn't look at it, that's not going to look good.

Second, regardless of whether they sent you the code, you probably should be trying to reproduce their results, if there is enough detail in the paper to understand what they did, and if it's highly relevant to your thesis.

Now if they didn't send you the code for the paper, and if the paper doesn't include enough information to reproduce their results, then the best you can do is mention in your thesis that "Smith et al claim to achieve a 400x speedup using a GPU [1], however some details of their approach do not seem to be publicly available." and leave it at that.

Finally: You should be talking to your research advisor about this. This is what your advisor is for. Talk to him/her. He/she will likely have useful advice for you.

  • They sent me code for something else, not this particular paper's code. I have no problem trying to replicate other people's results, but some of the stuff they say is so vague... For example (I'm not quoting, because I don't want to point to the paper, but this is paraphrased): "The CPU decides whether the operation to be performed would be more suitable for the GPU or the CPU". I can see many ways this particular "decision" could be implemented, and I'm sure some of them could be much worse than others. Do I just wing it and implement? – Mewa Apr 30 '15 at 16:40
  • [continued] And then at which point does it become my implementation rather than theirs? Because if I spend a month or two trying to get their results (without seeing what they did exactly), how do I know that I'm not doing it better than they did in the first place? – Mewa Apr 30 '15 at 16:41
  • @Mewa, you can try to implement yourself; in places where it's ambiguous, you can make the best guess you can, and then see what kind of performance speedup you get. If you are unable to replicate their 400x speedup, then you know what to write in your thesis: "Smith et al claim a 400x speedup, but some details of the approach do not seem to be publicly available and I was not able to replicate their results." You might be expected to make reasonable efforts to reproduce their results, but no one will expect heroic efforts. Your advisor can help calibrate you on the expectations. – D.W. Apr 30 '15 at 19:49
  • (continued) See Peteris's answer for a good explanation of the principles. For how it applies to your specific situation, only someone who knows your specific situation can give very detailed advice, so for that you might need to rely upon your advisor -- but read Peteris's answer to understand the general criteria and way to think about this. – D.W. Apr 30 '15 at 19:50
3

Attempt to replicate

If the topic is within the area that you're deeply studying, then it could be reasonable to attempt to replicate the results of their paper, even if (or especially because) the source is not available. The approach is described in the paper, as are the expected performance results. Where the details are vague, fill them in reasonably - after all, you're claiming to be an expert of the area as well. If they don't document relevant details (e.g. the hardware used), document these details for your replication as they might explain any notable differences.

If the replication results are significantly different than their claims, then this result may be publishable separately; if the results are comparable with theirs then it goes against your current expectations and fixes a potential flaw in your thesis. So either way the replication results would contribute to your goals.

In general, for a thesis direction "approach Y to achieve X" it's quite reasonable to [re]implement or review + re-test + verify multiple other approaches for achieving X, not just your 'in-house' approach. Having good quality open source implementations of other approaches makes this job easier, but not having such implementations doesn't mean that this job can be skipped.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.