2

Currently, my coauthors and I use GitHub to collaborate in coding and writing but also in data exchange. We have a lot of data, often not in text format (e.g. pdf). Most of this is collected by research assistants, which which we don't share our Git repo.

More specifically, we use python, shell, R, Stata and Latex and most of it is fully integrated. That is, python and shell scripts generate the data that is used by R and Stata whose output is directly compiled in Latex.

We don't want to deviate from this high level of automatization, but our approach has two major shortcomings:

  1. Research assistants can and may not upload their data directly into our repo. Instead, they send us their data and files via another git repo. This puts additional workload on me but we want to make them use git for it's fascinating issue tracker. However, there is too much additional work for us and git is often to complicated for young research assitants (even the GUIs).
  2. Because of the data, which eventually changes, our repo is very large (>1,5GB) and internet is sometimes restricted. Getting the repo on a new computer is very time-consuming and often not working. Although the raw data changes from time to time, git, which was not made for the data exchange, keeps track of these changes. But that's useless for our purpose.

Can you suggest to me other software(s) or approaches, which combine the integration we have achieved so far where we can easily exchange data?

  • "git is often to complicated for young research assitants (even the GUIs)" - really? I find this very surprising, unless 'young' means 'actual small children'. What particular aspect of git is causing trouble? If you're doing something unusually complicated with git then could this be put in a batch file? – A E Mar 2 '15 at 16:11
  • @AE: Our research assistants don't know about batch files and I don't expect them too. That's what students don't learn. What's complicate about it? Frankly, I don't know. I just see that they hardly learn it. But discussing that is useless here. – MERose Mar 2 '15 at 16:15
  • @AE I also strongly disbelieve that somebody who is useful for research in the technical sciences can't be taught basic Git. Even more, why can't the RAs check into the global Git? Git is versioned, so if they commit things the OP don't like, he can always revert. This setup sounds to me a bit like accidental complexity more than anything. – xLeitix Mar 2 '15 at 16:16
  • @xLeitix: For one simple reason: We may not share our other data in the repo. It's partly confidential. – MERose Mar 2 '15 at 16:18
  • 5
    @xLeitix You're coming from a blind spot here... what's very simple to somebody with a computer science background can be really hard for somebody with a deep technical background in another area. I have collaborators who are great in their field, but who use Excel for all of their data analysis and find it rather challenging... – jakebeal Mar 2 '15 at 16:40
6

I've run into similar problems in collaboration with biologists, and found that a two-technology approach is best.

  • Large-scale experimental data is shared via one or more BitTorrent Sync folders. The experimentalists can just drop their files into the appropriate place in the folder, and they get synchronized with the server and everybody else's copy (like DropBox, but private, free, and unlimited size).
  • The analytical tool-chain, research products, write-ups, etc. are maintained in your preferred version system (e.g., git). This is then integrated with the experimental data just by giving a pointer to the appropriate directories.

This has the advantage of maintaining the data separation that you need, keeping the scary version control software away from the experimentalists, and also avoiding juggling massive globs of data in a version control system that was never intended to support this.

|improve this answer|||||
  • But how can we assign issue to our RAs? Does BitTorrent Sync offer that? – MERose Mar 2 '15 at 16:53
  • BitTorrent Sync just moves data around. If you want an issue tracker, you need to select it separately from one of the many out there. – jakebeal Mar 2 '15 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.