Let me start with a disclaimer. I generally subscribe to the free software community perspective that proprietary software is questionable ethically, and best avoided if possible. I realise this perspective is not commonly held in scientific circles. Having said that, sometimes proprietary software is a necessary, or at least not easily avoided evil, and I'm generally pragmatic about using proprietary software when no good alternatives exist. I've used proprietary software in the past, though currently the only proprietary software I'm currently (sporadically) using is Skype, for which no good free alternatives exist.
However, special considerations apply in a scientific context. One of thse has already been covered by @David, namely that in general you can't "see inside" proprietary software to see how something is implemented. Having said that, sometimes proprietary software is written in an interpreted language, as in Splus, and one may be able to see part or all of an algorithm implementation. Regardless, the point holds generally.
A separate and obvious issue, which I don't think anyone has raised, is that using proprietary software forces others who want to use your software to buy the proprietary product you use. These products can be quite expensive, especially for people from poor countries. For example, Matlab, which has been mentioned in this thread, runs to thousands of dollars if one has to pay for a license oneself. Western academic institutions often have site licenses for such popular software, so researchers don't have to pay for it themselves. I personally am quite unhappy when I am expected to use a piece of software written using some proprietary language or package that has to be purchased.
A related issue is that much, if not most research, is done using public funding, i.e. taxpayer money. It seems undesirable to me to use such funds to buy proprietary software, thus adding to the profit of some corporation. In general, there is some movement to make academic work that is done using public funding free. And one can easily make the argument that the usage of proprietary software makes ones scientific product less free. For example, I believe the NIH now has some such policies in place. Similar arguments could be applied to the usage of software tools.
A tangential technical issue is that it is often difficult to get proprietary software to play nice on free software platforms such as the free Unix-like systems currently popular in scientific circles, e.g. the Linux based systems, and the BSD systems. These difficulties include, but are not restricted to
a) ABI problems. If one wants to compile a C/C++ extension for Matlab, for example, one has to use exactly the version of the compiler that the Matlab program has been compiled with
b) The program requires obsolete libraries or requires libraries to be in non-standard places.
I mention this issue in part because my understanding of the question is that it is asking about proprietary vs free in the context of pragmatic usage.
So, to respond to the question directly:
Assuming I'm starting a new project and I wish to make it as
reproducible as possible. Should I be using relatively unpopular free
software or extremely popular proprietary ones?
I don't think there is a clear answer. If there is no viable alternative, then one would have to use the proprietary software, as I do with Skype. If there a viable free version, I would use it. Bear in mind that if more people start using the "relatively unpopular free software" it will become more popular. :-)