What to do when people in a research group use different programming languages for computation (e.g., Mathematica, Python, Matlab)?

Question

I am a very happy user (in 98% of the time) of Python/Numpy/Scipy (anaconda distribution). I switched from Matlab and do not regret the decision. I have reached a level of expertise, which enables me to help others with their computational tasks and encourage people to use Python.

Unfortunately, the situation is not that easy. A senior researcher in a very closely related working group is using Mathematica. One of his students is using Mathematica, one is not sure yet but honestly it would be stupid to use a different software.

If I switch to Mathematica, everyone may benefit from it (code sharing and building up knowledge). I think you can understand that my motivation to do so is not the highest (not again another language; I use Python, Fortran and a bit C++). Additionally, the Python user community is very vibrant, and they frequently come up with interesting projects.

In order to avoid a nonsense discussion about what software to use, let me rephrase my questions as follows:

Have you ever been in a similar situation (either as student or supervisor)? And if you have, did you try to get everyone to use the same language? Did any situation occur where it was good that not everyone was using the same tool?

EDIT:

My field is biotechnology. We do calculate: ODE'S, PDE'S,fractals, system of equations (ODE's, DAE'S, algabraic). Most of the time we do some rapid prototyping (e.g calculating linear pH gradients in chromatography, some combinations of reactors etc., using/extending chromatographic models). I am a PhD student and will continue as a post doc at the same institute including occasional lab exchanges abroad.

Answer 1

This is very familiar to me - people come into my field from a number of different places, and each has not only their own preferred software, but the software they think "everyone" uses - which invariably isn't true.

At the moment for example, I have implementations of various bits of my work in MATLAB, Mathematica, C++, Python, Maple, R, SAS...

Have you been in a similar Situation (either as Student or Supervisor) ?

I've been in groups that successfully united everyone under the same language banner, and some groups that did not (intentionally or otherwise).

And if so, did you tried to get everyone using the same language ?

I have definitely tried, and occasionally failed. You've mentioned some of the benefits, but beyond merely code sharing, everyone using different languages makes it extremely hard to learn from others, share solutions, or collaborate really in any way. If you have a problem, and it's written in another language than what other folks are using, that problem is entirely your own to deal with - even if folks want to help, they may not be able to.

Did situations occur, at which it was good that not everyone is using the same tool ?

The only time is when the "usual tool" is somehow terrible at what's needed for someone's work. For example, a few years ago, when Python's statistics ecosystem was much worse, it was good to have people who knew R. But assuming they can all achieve roughly the same thing, I've never gone "Oh thank god we're all writing in different languages!"

The one exception is I did encounter someone whose ability to parse whitespace-based code is...less than stellar, which made me glad I could run things in MATLAB as well as Python.

Answer 2

First, you have to understand a very important point: most programming languages serve their own niche purposes. As a corollary, there is no one language that does everything best, or even does everything well. Therefore unifying into a single language is often not even an option on the table.

That said, there are cases where the same task can be accomplished in a variety of settings. I'll go through some examples in my experience (as a grad student in computational astrophysics).

• For hardcore, peg-the-CPU, million-core computations, your only options are Fortran, C, or C++. Such codes are often the central workhorses of the group, and it probably doesn't help to be diverse here. If everyone is coding in the same language, routines developed in one place can be used elsewhere with little cost. Pick one, and enforce stringent style guides on any collaborative code.

• For scripting, you have Unix shell scripts, Python, Perl, Ruby, and many others. Here it depends on who is meant to use the script. If writing something for your own personal workflow, there's nothing wrong with being different from everyone. On the other hand, if I am writing a script meant to be run, understood, and modified by others in the group (such as the configure script for the workhorse code), it had better be in an agreed-upon language.

• For light numerics, including matrix manipulation, there are proprietary programs like Matlab and IDL, and also free languages like Python and Octave. Since tasks using these programs can be a bit more involved than simple scripts, it helps to have others to get help from. I was once the sole Matlab user in an IDL group, and so there was little help I could give or receive with regard to numerics.

• For symbolic manipulation, as with Mathematica or Maple, I think the same considerations for light numerics apply.

• For data visualization, there are many options, including Matplotlib, Matlab, IDL, VisIt, Paraview, MayaVi, and yt. All of these are used in my group, where I'm solidly in the Matplotlib camp. Here we decided that visualization is as much an art as a science, and everyone has their own tools they are most comfortable with. If one person makes their best plots in Matlab, and someone else is a natural at VisIt, why force them to make poor use of IDL? In fact, having a multilingual group has proved beneficial, since not all options are always available, and it helps to have someone to get quick help from. For example, only some of the above can work on massively parallel visualization clusters to render terabyte datasets in reasonable time.

Personally, I've championed Python for scripting and small-scale visualization. In the scripting case, I demonstrated its strength by rewriting a configure script the group was using into a more versatile and readable one, and now we use Python for that shared script. That is, it wasn't an issue of "which language is intrinsically better?" but rather of "with which language can we do better?" For visualization, I'm not trying to forcibly convert anyone, but I simply share my scripts and knowledge with anyone who wants to learn what I know how to do.

In summary, how beneficial unification turns out to be depends on what the task is. Code that can be shared or reused helps to be in one language; everyone writing their own version of the same code would be wasted effort. Personal codes, on the other hand, work well when written in whatever language works best for the user; forcing everyone to use the same code (especially using the wrong code for the job, like a symbol manipulation package for data visualization) leads to individuals being less efficient.

Answer 3

The only restrictions that I place on my group members when it comes to the software that they use in their research work are:

• that they not use proprietary software for which the group doesn't own a license
• that their work can be shared or reused by other members of the group in the future

Requiring strict use of one set of tools is, I think, counterproductive, as it can force people to spend a lot of time learning things that won't necessarily be helpful to them in their research or later in their careers.

If they're just getting started on the programming side of things, however, I'll ask that they start with Python and the other standard codes that we use, because it makes life simpler for everybody in the long run.

Answer 4

A piece of wisdom my supervisor shared with me a few months back, when I told him about how cool Julia seemed. Was (paraphrasing):

We (often) aren't really in a position to choice our language for the task. We use what ever the best tools are being developed in. Before that was C++, then it was Matlab, now it seems to be python. Maybe by the time your PhD is done we will all be using Julia.

Point being, that learning a language is easy. You just do it, so you can use the best libraries.

---

One of the things I am loving about Julia is that because of its Foreign Language interface, you can call libraries written is many different languages. (I am aware of working code to call: C, Fortran, Rust, Python, Java, Matlab, Mathematica, and C++). Thus having maintained access to the "Best tools"

This is not to say you should convert everyone to julia. The first point stands. Learning a language is easy. Getting the best tools (or in your case perhaps collaborators), is not.

Answer 5

Yes I have been in a similar situation both as a student and as a supervisor.

As a student I was the only person in my lab to use Python (others were using Matlab or even Excel macros). Yes I tried to convince them to switch, with little success! But that was not a major problem because I did all the programming work on my own for my projects. I occasionally helped on other projects using whatever the main researcher on the project was using.

As a supervisor this was little bit more complicated, because my student was requiring a lot of micro-management and help on simple programming tasks in a language that I never used. He chose this language because he wanted to extend a program written by another research group. This was very frustrating for both of us, and I was hard for me to decide when to spend time figuring out simple things and when to tell him to RTFM!

In the future I would warn the student before starting the project: we agree that either he manages simple programming problems on its own, or he uses a language that I know well.

Answer 6

In my case, I'm using a mixture of R, C++ and Python whereas the rest of my lab uses Matlab. And it's going pretty fine, though I have to admit we don't need to share lots of code. Some observations:

• Having implementations in two languages that are supposed to work the same way helps finding bugs in the code.

• I can quickly evaluate new tools in R which weren't implemented in Matlab. Others can do the same with Matlab code that doesn't yet have an implementation in R. This already proved useful, as R is much easier to use in machine learning algorithms, and Matlab has a great library for polynomial interpolation.

• If you plan for interoperability early, it's not that hard. We decided on a single file format for raw data that's parseable by all our tools, and in case we needed to make scripts in both languages to interoperate, we can do so too (so far it wasn't necessary). However I admit that having skills in polyglot programming is somewhat necessary for that to happen.

Answer 7

Have you been in a similar Situation (either as Student or Supervisor) ?

Yes, frequently - in my current department, people use (depending both on their personal preference and on external requirements) Java, C#, C++, C, JavaScript, Flash, and probably a few more.

It even varies a lot, as students may want to use yet another technology or language for their projects such as Bachelor or Master theses.

And if so, did you tried to get everyone using the same language ?

Never, unless interoperability was an explicit requirement. It may have to do with the fact that I'm in a CS field (i.e. where programming is more at the "core" than a mere tool), but there is what could be called an unwritten rule that you do not prescribe others what technologies they use. At best, it could be interpreted as an immature attempt of starting a flame-war on a "nerdy" topic, at worst, as a violation of other researchers'/developers' personal autonomy by micro-management.

Concerning the aforementioned students, we do make it clear that we cannot provide any technical support if they choose a technology that none of us has any experience with (though it should be noted that we won't provide too much support, anyway, given that Bachelor and Master theses are supposed to focus on the conceptual and methodical aspects in my place, and students are supposed to (show that they are able to) deal with low-level problems while programming their prototypical proof-of-concept on their own.)

Did situations occur, at which it was good that not everyone is using the same tool ?

To avoid making this sound overly one-sided or negative, I'm going to list perceived advantages and disadvantages here:

Disadvantages:

• Artifacts by different persons won't necessarily work together.
  • This has even led to complete reimplementations of prototypes, just so a widget by one person could be used in the application by another person who used a different framework and/or UI toolkit.
• Maintenance of existing code-base is not a given. Once a colleague leaves, their code might not be touched again on the single reason that no-one is acquainted with the technology used for that particular code.

Advantages:

• There is no way to guarantee using just one language, anyway; there are just too many external factors for this. It can more or less be taken for granted that as soon as one has succeeded at bringing the whole department to one single programming language, the next project with external partners will end up in the consortium choosing a different language for one reason or another. Having a diverse department where know-how on different technologies and languages is present, on the other hand, can help when working on such a project.
• As extensively described in Chris White's answer, different programming languages are often suitable for different goals, so depending on what you are doing, a switch of languages might be required.
• Seeing permanent change and diversity makes it less likely to "get stuck" with one technology. Creating a growing collection of reusable code is certainly advantageous, but if that results in the use of outdated technologies because "everything so far has been written in the 1982 dialect of a proprietary language that is not updated any more" and the expectation is that a switch would require porting the entire codebase, this does not exactly increase the research output. As research departments usually do not have to produce production-level foolproof code, but just prototypes and demonstrations of concepts, absolute stability should be a lesser concern, and thus, constant "quick-and-dirty" rewrites of some components are acceptable.

EDIT: Every time I re-read this answer of mine, the developer in myself shudders in horror. Thus, let me clarify my view on the advantages: Yes, maintaining, updating and extending an existing code-base over long periods of time is great. In my opinion, an ideal mix is for small groups of people within one department to share a particular technology and thus have an option of exchanging some code (being the only one bound to a given system can be dire), while at the same time making sure there is some slow, gradual flow in which technologies get phased out and replaced over time with new ones.

Answer 8

Have you ever been in a similar situation (either as student or supervisor)?

Yes, twice. In my previous group, everyone used MATLAB, but I had to learn Python because MATLAB's multiprocessing was prohibitively expensive. After learning Python, I preferred it and stuck to it. In my current group, everyone besides me uses MATLAB.

For my own personal data analysis code, I use Python, but for code meant to be shared across the group, I use MATLAB.

If you have a programming background, you would think that this would lead to less shared code for data analysis. The problem is that nobody else in the group has a programming background, and they neither know nor care about good programming practices (I haven't even had any luck convincing them to use functions rather than copying and pasting code blocks within their tens of thousands of lines scripts, not to mention adding comments). That means there is almost no code sharing even amongst MATLAB users, and everyone just ends up writing their own data analysis code from scratch. So due to the culture of the lab, nobody would even have realize I used Python if I hadn't told them.

So how big of an affect your choice of language has depends to a large degree on the culture of the lab, or perhaps more properly on how familiar they are with programming.

And if you have, did you tried to get everyone to use the same language?

No. I have suggested that people learn Python in addition to MATLAB, without much success. But people who have asked me what they should learn, I have suggested learn MATLAB first, simply because they can be sure to find someone who has at least used it before. I am the only one with a real programming background, and there is a ton of legacy MATLAB code, so it just isn't feasible to switch at this point. That is why I always suggest people learn Python as a second language.

Did any situation occur where it was good that not everyone is using the same tool?

Yes, two cases come to mind.

First, is the fact that I needed to write a new importer for our proprietary data format. Due to a flaw in the format, it could cause overruns in the file, which were still readable in principle but that crashed both the native C file reader and the MATLAB-based file reader. My Python implementation, however, was more flexible, and could handle the data. Someone else in my group ended up routinely getting these overflows, so I wrote a simple wrapper script that would read the data in then convert it ta a MATLAB file, saving his project from disaster.

The second is the fact that the MATLAB-based tool for something we wanted to do is much, much, much more complicated, hard to use, and finicky than the Python equivalent. This probably saved a good month of work, and resulted in something with much better perfomance.

Answer 9

Besides working out a way to use multiple programming languages (e.g. through foreign function interfaces / compiling stuff in libraries / creating scripts which can be called by other languages) I think you could eventually agree on one programming language.

I recently participated in a hackathon. We were three programmers and had a task to solve. We "filtered" the programming languages like this:

1. Which languages do we know which are similar appropriate for the task (in terms of ease-to-use / libraries / community stuff like tutorials / learning curve)
2. How well do the team members know "their" language? (It is better to have one person who is an expert in a language than two who know a little bit, because that way you can ask the expert any question.)

(The hackathon was a success and I learned a new programming language and a framework in another programming language.)

Answer 10

I suggest to go with different languages: it provides cross-validation of your calculations. A software/hardware/algorithm might be wrong. The best way to check the correctness is to repeat the calculation in a completely different environment. So I think it is actually good that members of the lab use different languages.

Even if all members of the group will switch to the same language — the coding style might be different, and will not help for code transfer. It will only work if all of you code in the same langauge and also hold the same views on the coding itself. For example I prefer maximum modularity, while somebody else might prefer the fastest approach instead.