As a PhD researcher in computer science, my scientific results often take the form of algorithms, which in turn come with implementations. I am convinced that releasing these implementations under an open-source model has various benefits especially for scientific codes, one of which is reproducibility of results. I also believe that "Release Often, Release Early" applies to science, as it opens up the possibility of feedback that ultimately leads to better software and better research.
However, my advisor is concerned with the danger of plagiarism in case the code is released before the respective paper is published, and therefore disagrees with that philosophy. To me it seems like a very theoretical threat since I have never heard of a comparable case of plagiarism. However, we just got a paper rejected in which one major criticism of the reviewer was that it was impossible to reproduce results since the code was not available. My advisor thinks that this is unfair criticism because the conference has not explicitly asked for a release of the source code.
What do you think? Is it sensible to keep implementations secret until the paper is through, or is open-source actually a requirement for reproducible computer science?