We have added a new experimental feature to an existing software project (e.g. on GitHub), and we would like to publish a paper about it. However, we want to publish it in a venue which uses double-blind review. What is the correct method of anonymization in this case?
I see several ways which all seem to have major issues:
cite the project, clearly state which part is our contribution, include it as a supplementary material anonymizing our contribution -> the issue here is that the reviewers will either already know the software project (and then likely know that such a feature has been added), or they will not know about it, look at it, and likely find out that this feature has been added, breaking anonymity. This seems to be the best solution to me, however, since the rules for anonymity tend to be quite strict, we are not sure whether this would be acceptable.
anonymize the whole project -> this is basically (self)-plagiarism (especially if there are already papers published about the original project).
do not include the implementation, only the results -> this makes it less reproducible, and more difficult for the referees to review. Also it might be still good to cite the original project, with the drawbacks of (1).
This would not be a problem for a normal peer review (not double blind), or if the project is completely original (there are several questions on this site about this case).