38

I am a very happy user (in 98% of the time) of Python/Numpy/Scipy (anaconda distribution). I switched from Matlab and do not regret the decision. I have reached a level of expertise, which enables me to help others with their computational tasks and encourage people to use Python.

Unfortunately, the situation is not that easy. A senior researcher in a very closely related working group is using Mathematica. One of his students is using Mathematica, one is not sure yet but honestly it would be stupid to use a different software.

If I switch to Mathematica, everyone may benefit from it (code sharing and building up knowledge). I think you can understand that my motivation to do so is not the highest (not again another language; I use Python, Fortran and a bit C++). Additionally, the Python user community is very vibrant, and they frequently come up with interesting projects.

In order to avoid a nonsense discussion about what software to use, let me rephrase my questions as follows:

Have you ever been in a similar situation (either as student or supervisor)? And if you have, did you try to get everyone to use the same language? Did any situation occur where it was good that not everyone was using the same tool?

EDIT:

My field is biotechnology. We do calculate: ODE'S, PDE'S,fractals, system of equations (ODE's, DAE'S, algabraic). Most of the time we do some rapid prototyping (e.g calculating linear pH gradients in chromatography, some combinations of reactors etc., using/extending chromatographic models). I am a PhD student and will continue as a post doc at the same institute including occasional lab exchanges abroad.

  • 3
    Another avenue is to try to make everything modular, so that, for example, a high-level Python script calls more dedicated Python/Mathematica/etc. functions. This can be done directly in a programming language, or you can try to use something sort of workflow management tool (one that I have tried minimally is called Kepler, but it doesn't have Mathematica as a native language). – dang Apr 14 '15 at 21:25
  • 1
    If you do want to make your case for why Python is better though then you could focus on it being open source and having a very active community. I've run into licensing issues trying to use MATLAB on national HPC systems (I think it is harder/more expensive to license software that can be used by researchers from multiple institutions). One reason I learned R and Python (I also started with MATLAB) is that I might not have access to expensive licensed software in the future. – user49483 Apr 14 '15 at 22:28
  • 6
    I think it worth to remark what field you are in: CS? Physics? Different fields have different cultures and level of "reasonable programming skill" also very different. In this particular situation, I also would remark that Python/scipy and Mathematica are very-very different animals, and if one is a good tool for a given problem, the other most probably not. I.e. you can have very good reasons to use both, depending on the problem. Mathematica is a far better symbolic calculator, and scipy is far better number cruncher. – Greg Apr 15 '15 at 12:10
  • 1
    @Szabolcs Considering the following scenario: "Hey, the model I'm running is producing some really weird convergence errors, can you come be an extra set of eyes?" or "Man, that analysis you did was perfect for my project, how did you do it?" Code sharing and collaboration go beyond writing an application. – Fomite Apr 16 '15 at 17:44
  • 1
    Following up on @dang 's point, if there are "natural" places in your workflow to test or checkpoint-to-file or preferably both!, they should be good places to let anyone change languages (and if you make the file formats and assumptions explicit to enable this, well, needed to be done anyway). – cphlewis Apr 16 '15 at 18:33

10 Answers 10

20

This is very familiar to me - people come into my field from a number of different places, and each has not only their own preferred software, but the software they think "everyone" uses - which invariably isn't true.

At the moment for example, I have implementations of various bits of my work in MATLAB, Mathematica, C++, Python, Maple, R, SAS...

Have you been in a similar Situation (either as Student or Supervisor) ?

I've been in groups that successfully united everyone under the same language banner, and some groups that did not (intentionally or otherwise).

And if so, did you tried to get everyone using the same language ?

I have definitely tried, and occasionally failed. You've mentioned some of the benefits, but beyond merely code sharing, everyone using different languages makes it extremely hard to learn from others, share solutions, or collaborate really in any way. If you have a problem, and it's written in another language than what other folks are using, that problem is entirely your own to deal with - even if folks want to help, they may not be able to.

Did situations occur, at which it was good that not everyone is using the same tool ?

The only time is when the "usual tool" is somehow terrible at what's needed for someone's work. For example, a few years ago, when Python's statistics ecosystem was much worse, it was good to have people who knew R. But assuming they can all achieve roughly the same thing, I've never gone "Oh thank god we're all writing in different languages!"

The one exception is I did encounter someone whose ability to parse whitespace-based code is...less than stellar, which made me glad I could run things in MATLAB as well as Python.

  • What was your experience with those groups that sucessfully united everyone under the same language banner ? – Moritz Apr 14 '15 at 19:11
  • 2
    @Moritz From the perspective of collaboration? Better, vastly, but not perfect (different coding styles, etc. still make things hard). – Fomite Apr 14 '15 at 19:12
  • 6
    In theory, writing in different languages is good because you can write two versions of the really important stuff, run them on the same test data, and have a better chance of finding subtle bugs. In practice, who takes the time? – cphlewis Apr 14 '15 at 21:39
  • 1
    @cphlewis I did exactly that. – TheBlackCat Apr 16 '15 at 14:13
  • 1
    @cphlewis Subtle bugs and not-so-subtle ones, I am afraid. Unit testing is a great thing. – TheBlackCat Apr 16 '15 at 18:18
20

First, you have to understand a very important point: most programming languages serve their own niche purposes. As a corollary, there is no one language that does everything best, or even does everything well. Therefore unifying into a single language is often not even an option on the table.

That said, there are cases where the same task can be accomplished in a variety of settings. I'll go through some examples in my experience (as a grad student in computational astrophysics).

  • For hardcore, peg-the-CPU, million-core computations, your only options are Fortran, C, or C++. Such codes are often the central workhorses of the group, and it probably doesn't help to be diverse here. If everyone is coding in the same language, routines developed in one place can be used elsewhere with little cost. Pick one, and enforce stringent style guides on any collaborative code.

  • For scripting, you have Unix shell scripts, Python, Perl, Ruby, and many others. Here it depends on who is meant to use the script. If writing something for your own personal workflow, there's nothing wrong with being different from everyone. On the other hand, if I am writing a script meant to be run, understood, and modified by others in the group (such as the configure script for the workhorse code), it had better be in an agreed-upon language.

  • For light numerics, including matrix manipulation, there are proprietary programs like Matlab and IDL, and also free languages like Python and Octave. Since tasks using these programs can be a bit more involved than simple scripts, it helps to have others to get help from. I was once the sole Matlab user in an IDL group, and so there was little help I could give or receive with regard to numerics.

  • For symbolic manipulation, as with Mathematica or Maple, I think the same considerations for light numerics apply.

  • For data visualization, there are many options, including Matplotlib, Matlab, IDL, VisIt, Paraview, MayaVi, and yt. All of these are used in my group, where I'm solidly in the Matplotlib camp. Here we decided that visualization is as much an art as a science, and everyone has their own tools they are most comfortable with. If one person makes their best plots in Matlab, and someone else is a natural at VisIt, why force them to make poor use of IDL? In fact, having a multilingual group has proved beneficial, since not all options are always available, and it helps to have someone to get quick help from. For example, only some of the above can work on massively parallel visualization clusters to render terabyte datasets in reasonable time.

Personally, I've championed Python for scripting and small-scale visualization. In the scripting case, I demonstrated its strength by rewriting a configure script the group was using into a more versatile and readable one, and now we use Python for that shared script. That is, it wasn't an issue of "which language is intrinsically better?" but rather of "with which language can we do better?" For visualization, I'm not trying to forcibly convert anyone, but I simply share my scripts and knowledge with anyone who wants to learn what I know how to do.

In summary, how beneficial unification turns out to be depends on what the task is. Code that can be shared or reused helps to be in one language; everyone writing their own version of the same code would be wasted effort. Personal codes, on the other hand, work well when written in whatever language works best for the user; forcing everyone to use the same code (especially using the wrong code for the job, like a symbol manipulation package for data visualization) leads to individuals being less efficient.

  • 1
    "For hardcore, peg-the-CPU, million-core computations, your only options are Fortran, C, or C++." Strongly disagree with this one. Of course, these languages are likely to be the fastest if you are an experienced programmer in these languages, but you are unlikely to need this level of optimisation, and CPU time is cheap compared to programming time. – MJeffryes Apr 15 '15 at 9:37
  • 15
    @MJeffryes OP's argument is that "hardcore computations" are a niche with very specific needs. Telling "People are unlikely to be in that niche and to have those needs" does not really counter their argument. – Federico Poloni Apr 15 '15 at 10:37
  • @FedericoPoloni Yes I take your point. Obviously different languages are suited for different tasks which serve different niches, but I think there is something to be said for avoiding prematurely optimising through choice of programming language for the sake of consistency. If everyone is using Python then you should probably consider using it for HPC rather than switching to FORTRAN if no one else understands it. – MJeffryes Apr 15 '15 at 10:49
  • 1
    On the other hand, you need to consider the basic suitability of the language to the task at hand, in the same way that while a skateboard might be better getting from one side of a campus to another, it's not the most suitable way of travelling from state to state, no matter how good you are with it. – Gwyn Evans Apr 15 '15 at 17:21
  • 1
    @MJeffryes That depends on whether Python is fast enough for your needs, and has the tools you need, and has the data structures you need, etc. There are lots of reasons for using another languages. Granted, for Python these will generally come down to performance, while for other languages (MATLAB, Mathematica), these may come down to cost issues or more fundamental problems with the language. But even Python will not be the ideal language for everything, otherwise Python libraries wouldn't have so much C and Fortran code under the hood. – TheBlackCat Apr 16 '15 at 14:16
12

The only restrictions that I place on my group members when it comes to the software that they use in their research work are:

  • that they not use proprietary software for which the group doesn't own a license
  • that their work can be shared or reused by other members of the group in the future

Requiring strict use of one set of tools is, I think, counterproductive, as it can force people to spend a lot of time learning things that won't necessarily be helpful to them in their research or later in their careers.

If they're just getting started on the programming side of things, however, I'll ask that they start with Python and the other standard codes that we use, because it makes life simpler for everybody in the long run.

10

A piece of wisdom my supervisor shared with me a few months back, when I told him about how cool Julia seemed. Was (paraphrasing):

We (often) aren't really in a position to choice our language for the task. We use what ever the best tools are being developed in. Before that was C++, then it was Matlab, now it seems to be python. Maybe by the time your PhD is done we will all be using Julia.

Point being, that learning a language is easy. You just do it, so you can use the best libraries.


One of the things I am loving about Julia is that because of its Foreign Language interface, you can call libraries written is many different languages. (I am aware of working code to call: C, Fortran, Rust, Python, Java, Matlab, Mathematica, and C++). Thus having maintained access to the "Best tools"

This is not to say you should convert everyone to julia. The first point stands. Learning a language is easy. Getting the best tools (or in your case perhaps collaborators), is not.

  • 2
    That depends on the language. Learning some languages are easier than learning others, and some languages teach you skills that are more generalizable than others. And lots of languages have foreign function interfaces. – TheBlackCat Apr 16 '15 at 14:18
  • FFI generally refers to being able to interface with C. But indeed it is getting more and more common for wider interfacing, because it is a great idea. I would be interested to see a answer to this question that recommends using FFI. – Lyndon White Apr 17 '15 at 2:20
7

Yes I have been in a similar situation both as a student and as a supervisor.

As a student I was the only person in my lab to use Python (others were using Matlab or even Excel macros). Yes I tried to convince them to switch, with little success! But that was not a major problem because I did all the programming work on my own for my projects. I occasionally helped on other projects using whatever the main researcher on the project was using.

As a supervisor this was little bit more complicated, because my student was requiring a lot of micro-management and help on simple programming tasks in a language that I never used. He chose this language because he wanted to extend a program written by another research group. This was very frustrating for both of us, and I was hard for me to decide when to spend time figuring out simple things and when to tell him to RTFM!

In the future I would warn the student before starting the project: we agree that either he manages simple programming problems on its own, or he uses a language that I know well.

6

In my case, I'm using a mixture of R, C++ and Python whereas the rest of my lab uses Matlab. And it's going pretty fine, though I have to admit we don't need to share lots of code. Some observations:

  • Having implementations in two languages that are supposed to work the same way helps finding bugs in the code.

  • I can quickly evaluate new tools in R which weren't implemented in Matlab. Others can do the same with Matlab code that doesn't yet have an implementation in R. This already proved useful, as R is much easier to use in machine learning algorithms, and Matlab has a great library for polynomial interpolation.

  • If you plan for interoperability early, it's not that hard. We decided on a single file format for raw data that's parseable by all our tools, and in case we needed to make scripts in both languages to interoperate, we can do so too (so far it wasn't necessary). However I admit that having skills in polyglot programming is somewhat necessary for that to happen.

  • 2
    Key advice in this answer: Plan for interoperability early!! Golden. – Mindwin Apr 15 '15 at 20:55
  • On second thought, having a back-to-back comparison like bullet point #1 is good too. – Mindwin Apr 15 '15 at 20:57
3

Have you been in a similar Situation (either as Student or Supervisor) ?

Yes, frequently - in my current department, people use (depending both on their personal preference and on external requirements) Java, C#, C++, C, JavaScript, Flash, and probably a few more.

It even varies a lot, as students may want to use yet another technology or language for their projects such as Bachelor or Master theses.

And if so, did you tried to get everyone using the same language ?

Never, unless interoperability was an explicit requirement. It may have to do with the fact that I'm in a CS field (i.e. where programming is more at the "core" than a mere tool), but there is what could be called an unwritten rule that you do not prescribe others what technologies they use. At best, it could be interpreted as an immature attempt of starting a flame-war on a "nerdy" topic, at worst, as a violation of other researchers'/developers' personal autonomy by micro-management.

Concerning the aforementioned students, we do make it clear that we cannot provide any technical support if they choose a technology that none of us has any experience with (though it should be noted that we won't provide too much support, anyway, given that Bachelor and Master theses are supposed to focus on the conceptual and methodical aspects in my place, and students are supposed to (show that they are able to) deal with low-level problems while programming their prototypical proof-of-concept on their own.)

Did situations occur, at which it was good that not everyone is using the same tool ?

To avoid making this sound overly one-sided or negative, I'm going to list perceived advantages and disadvantages here:

Disadvantages:

  • Artifacts by different persons won't necessarily work together.
    • This has even led to complete reimplementations of prototypes, just so a widget by one person could be used in the application by another person who used a different framework and/or UI toolkit.
  • Maintenance of existing code-base is not a given. Once a colleague leaves, their code might not be touched again on the single reason that no-one is acquainted with the technology used for that particular code.

Advantages:

  • There is no way to guarantee using just one language, anyway; there are just too many external factors for this. It can more or less be taken for granted that as soon as one has succeeded at bringing the whole department to one single programming language, the next project with external partners will end up in the consortium choosing a different language for one reason or another. Having a diverse department where know-how on different technologies and languages is present, on the other hand, can help when working on such a project.
  • As extensively described in Chris White's answer, different programming languages are often suitable for different goals, so depending on what you are doing, a switch of languages might be required.
  • Seeing permanent change and diversity makes it less likely to "get stuck" with one technology. Creating a growing collection of reusable code is certainly advantageous, but if that results in the use of outdated technologies because "everything so far has been written in the 1982 dialect of a proprietary language that is not updated any more" and the expectation is that a switch would require porting the entire codebase, this does not exactly increase the research output. As research departments usually do not have to produce production-level foolproof code, but just prototypes and demonstrations of concepts, absolute stability should be a lesser concern, and thus, constant "quick-and-dirty" rewrites of some components are acceptable.

EDIT: Every time I re-read this answer of mine, the developer in myself shudders in horror. Thus, let me clarify my view on the advantages: Yes, maintaining, updating and extending an existing code-base over long periods of time is great. In my opinion, an ideal mix is for small groups of people within one department to share a particular technology and thus have an option of exchanging some code (being the only one bound to a given system can be dire), while at the same time making sure there is some slow, gradual flow in which technologies get phased out and replaced over time with new ones.

1

Have you ever been in a similar situation (either as student or supervisor)?

Yes, twice. In my previous group, everyone used MATLAB, but I had to learn Python because MATLAB's multiprocessing was prohibitively expensive. After learning Python, I preferred it and stuck to it. In my current group, everyone besides me uses MATLAB.

For my own personal data analysis code, I use Python, but for code meant to be shared across the group, I use MATLAB.

If you have a programming background, you would think that this would lead to less shared code for data analysis. The problem is that nobody else in the group has a programming background, and they neither know nor care about good programming practices (I haven't even had any luck convincing them to use functions rather than copying and pasting code blocks within their tens of thousands of lines scripts, not to mention adding comments). That means there is almost no code sharing even amongst MATLAB users, and everyone just ends up writing their own data analysis code from scratch. So due to the culture of the lab, nobody would even have realize I used Python if I hadn't told them.

So how big of an affect your choice of language has depends to a large degree on the culture of the lab, or perhaps more properly on how familiar they are with programming.

And if you have, did you tried to get everyone to use the same language?

No. I have suggested that people learn Python in addition to MATLAB, without much success. But people who have asked me what they should learn, I have suggested learn MATLAB first, simply because they can be sure to find someone who has at least used it before. I am the only one with a real programming background, and there is a ton of legacy MATLAB code, so it just isn't feasible to switch at this point. That is why I always suggest people learn Python as a second language.

Did any situation occur where it was good that not everyone is using the same tool?

Yes, two cases come to mind.

First, is the fact that I needed to write a new importer for our proprietary data format. Due to a flaw in the format, it could cause overruns in the file, which were still readable in principle but that crashed both the native C file reader and the MATLAB-based file reader. My Python implementation, however, was more flexible, and could handle the data. Someone else in my group ended up routinely getting these overflows, so I wrote a simple wrapper script that would read the data in then convert it ta a MATLAB file, saving his project from disaster.

The second is the fact that the MATLAB-based tool for something we wanted to do is much, much, much more complicated, hard to use, and finicky than the Python equivalent. This probably saved a good month of work, and resulted in something with much better perfomance.

1

Besides working out a way to use multiple programming languages (e.g. through foreign function interfaces / compiling stuff in libraries / creating scripts which can be called by other languages) I think you could eventually agree on one programming language.

I recently participated in a hackathon. We were three programmers and had a task to solve. We "filtered" the programming languages like this:

  1. Which languages do we know which are similar appropriate for the task (in terms of ease-to-use / libraries / community stuff like tutorials / learning curve)
  2. How well do the team members know "their" language? (It is better to have one person who is an expert in a language than two who know a little bit, because that way you can ask the expert any question.)

(The hackathon was a success and I learned a new programming language and a framework in another programming language.)

0

I suggest to go with different languages: it provides cross-validation of your calculations. A software/hardware/algorithm might be wrong. The best way to check the correctness is to repeat the calculation in a completely different environment. So I think it is actually good that members of the lab use different languages.

Even if all members of the group will switch to the same language — the coding style might be different, and will not help for code transfer. It will only work if all of you code in the same langauge and also hold the same views on the coding itself. For example I prefer maximum modularity, while somebody else might prefer the fastest approach instead.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.