Our research group sometimes does the following:
- There is some problem P, and we are writing a paper about a novel technique T.
- In one of the sections of the paper, we try to empirically show that T is a great technique.
- To do this, we create an implementation I of technique T, and also look at other existing software products/artifacts J, K, L which also solve problem P (but not using technique T).
- Generally, our implementation I is better than J, K, L, and this serves as good empirical evidence for the hypothesis that T is a great technique
However, our implementation I usually uses several other optimizations T1, T2, T3 and the paper does not discuss these optimizations. Now, I believe that we would not have beat implementations J, K, L if we only used T without T1, T2, T3. I also believe that just adding the optimizations T1, T2, T3 to any of J, K, L would still be not as good as I (which uses T + T1, T2, T3).
I think these are (one or more of) the possible rationalizations of doing this:
- Technique T1, T2, T3 are minor optimizations. (This is subjective, but I generally disagree that they are minor. My opinion is that our PI is a gifted programmer and consistently underestimates how difficult it is to implement something.)
- Talking about technique T1, T2, T3 is "bloat" for the paper, and takes away attention from T, which is the main idea. If we do this, this might distract the reviewers, and there is a possibility that they wouldn't agree with the hypothesis that T is a great technique.
- Sometimes the software artifacts J, K, L are very well-known and well-engineered tools. If this is the case, one may say that our results are good evidence for the hypothesis T is a great technique anyway, since the implementation I is an underdog in terms of man-hours spent in producing it.
- If they are not minor, it may be worthwhile keeping T1, T2, T3 for a different paper, or an extended version of the paper, or a "tool" paper.
But more importantly, this feels unethical to me because:
- We are misrepresenting the impact of technique T, which is what the paper is about.
- It reduces the "utility" of our paper. If someone needs to solve P, and discovers our paper, they will be in for some unpleasant surprises if they try to implement technique T.
- It also doesn't help that we rarely publish our source code
Is it ethical to present the performance of an implementation that uses more optimizations than we discuss?