Is it ethical to present the performance of an implementation that uses undisclosed optimizations?

Question

Our research group sometimes does the following:

• There is some problem P, and we are writing a paper about a novel technique T.
• In one of the sections of the paper, we try to empirically show that T is a great technique.
• To do this, we create an implementation I of technique T, and also look at other existing software products/artifacts J, K, L which also solve problem P (but not using technique T).
• Generally, our implementation I is better than J, K, L, and this serves as good empirical evidence for the hypothesis that T is a great technique

However, our implementation I usually uses several other optimizations T1, T2, T3 and the paper does not discuss these optimizations. Now, I believe that we would not have beat implementations J, K, L if we only used T without T1, T2, T3. I also believe that just adding the optimizations T1, T2, T3 to any of J, K, L would still be not as good as I (which uses T + T1, T2, T3).

I think these are (one or more of) the possible rationalizations of doing this:

• Technique T1, T2, T3 are minor optimizations. (This is subjective, but I generally disagree that they are minor. My opinion is that our PI is a gifted programmer and consistently underestimates how difficult it is to implement something.)
• Talking about technique T1, T2, T3 is "bloat" for the paper, and takes away attention from T, which is the main idea. If we do this, this might distract the reviewers, and there is a possibility that they wouldn't agree with the hypothesis that T is a great technique.
• Sometimes the software artifacts J, K, L are very well-known and well-engineered tools. If this is the case, one may say that our results are good evidence for the hypothesis T is a great technique anyway, since the implementation I is an underdog in terms of man-hours spent in producing it.
• If they are not minor, it may be worthwhile keeping T1, T2, T3 for a different paper, or an extended version of the paper, or a "tool" paper.

But more importantly, this feels unethical to me because:

• We are misrepresenting the impact of technique T, which is what the paper is about.
• It reduces the "utility" of our paper. If someone needs to solve P, and discovers our paper, they will be in for some unpleasant surprises if they try to implement technique T.
• It also doesn't help that we rarely publish our source code

Is it ethical to present the performance of an implementation that uses more optimizations than we discuss?

Answer 1

The problem is that I cannot trust your results at all.

You claim that you have improvements due to T. But since you added optimisations T1, T2 and T3, there is no way to verify that any improvements are indeed caused by T. Neither for me nor for you. You should only publish this if you can also provide either the unoptimised version, and the unoptimised version with T, so we can check the improvement from unoptimised to (unoptimised with T), or your current version with T removed, so we can compare (optimised without T) to (optimised with T).

Of course you could just create a paper that demonstrates T + T1 + T2 + T3 being an improvement to having neither of them.

Answer 2

If the other optimizations are responsible for the result and they aren't disclosed, then, yes, it is unethical. If, in fact, your results show that that T alone isn't better than the other alternatives and you misrepresent that it is better, you are being unethical.

You need full disclosure here and you should probably view it as an opportunity. Academic publishing should be about reality, not marketing some technique.

I would think that a paper that demonstrated the combination of things, and not just "T" alone was superior would be a better paper then one that misrepresents the value of T.

As it is, you seem to be wanting to publish a "hoped for" result, not the actual result. Yes, ethical problem.

Answer 3

In general, claiming that X is evidence of Y while not disclosing Z, a fact that makes it clear that X is not quite as strong evidence for Y as you are claiming, is dishonest and unethical.

Your situation clearly falls within this set of behaviors. So yes, it is unethical.

Thinking about the bigger picture here, what seems to me to be the larger problem is that your group is publishing code-based papers without disclosing the code. If you were required to release the code, the sort of gamesmanship you are talking about would be much more difficult to engage in, because the “hidden” optimizations would be easy for others to detect.

Conversely, in an environment where authors are not required to release their code but can still claim breakthroughs evidenced by their private code, it is almost certain that various forms of cheating and gamesmanship would take place; the temptation would be too great, and making performance claims that are unverifiable by anyone who doesn’t have access to your code is just too easy for it not to become commonplace.

So, while your group’s behavior is indeed unethical, I’d say the larger culprit here is the bad publication culture of your field, in which papers can easily get away with making unethical and dishonest claims.

Answer 4

I hope you can see that your back-to-back statements are contradictory:

Now, I believe that we would not have beat implementations J, K, L if we only used T without T1, T2, T3. I also believe that just adding the optimizations T1, T2, T3 to any of J, K, L would still be not as good as I (which uses T + T1, T2, T3).

They correspond respectively to T < J and T + T* > J + T* -- that is, the unoptimized new method loses to the unoptimized packages and the optimised new method would still beat the hypothetically optimised packages.

I suppose that could theoretically be true since computational gains aren't always additive. But it's not obvious how that could be true, and I don't think you are sure whether it is true or not, and if you are not sure about the subject of your expertise then doesn't that make it fertile grounds for research? Shouldn't your group also investigate whether T* are truly optimisations and publish that they are or aren't?

So, ethical concerns aside (and they are real!) your group is simply leaving easy papers unwritten if you keep not documenting all the improvements you're making. If your group is really a paper-focused group, why not write those papers?