Suppose we perform experiments with input parameters (temperature, humidity, processing time...) and collect resulting data (thickness, structure, mech. properties...).

Is there a tool (or set of tools) to organize, process and export data from such experiments?

Key features are:

  • Structured files decomposition (raw text files).
  • Basic math operations.
  • Filter and sort by given parameters (show/export data from samples treated at given temperature and humidity for various times).
  • Generating tables with given parameters and list of "constants" (table of times, mech. properties and thicknesses and list containing temperature, humidity...).
  • Vector graphics output and/or output suitable for MATLAB (graph of thickness as function of time).
  • Automated (or easy-to-create) LaTeX output (report sheet).

If not, any idea, hint or recommendation how to create it is appreciated. Right now I'm thinking of a spreadsheet (Excel) as core database and MATLAB as the processor (filters, sorting, graphics).

  • Check out R and one of its function Sweave, which can integrate R analytic command into LaTeX codes. Users can run the code first in R, and then LaTeX to obtain a report with all tables, statistics, and graphics already nicely embedded. Feb 27 '14 at 17:28
  • 1
    Org mode lets you combine the data, text, and processing. The processing can be done in whatever language you want, and even in multiple languages (i.e. wrangle data with awk, process with Python, display with R, all in the same file). Everything (or everything you want) can then be exported to LaTeX, or various other formats too.
    – mankoff
    Feb 27 '14 at 21:25

I would store data in CSV (i.e. text file with a table, with values separated by commas) rather than XLS files (the first is easier to import from and export to anything). Otherwise many tools will do the job (if you are familiar with MATLAB - why not using it)?

For general data processing and manipulation, Python (with SciPy stack) is capable of everything you mentioned. In particular IPython Notebook is great for data exploration and presentation (you can use code, comments and LaTeX in such notebook - also for reports). For tabular data, use Pandas (R-like DataFrames).

For reports also you can create files in Markdown (with LaTeX), and then convert them to pdf with Pandoc - may be much easier than generation of LaTeX code. (To get you some taste what is Markdown - look at StackEdit.)

And alternative to Python is R, with knitr for report generation. If you are not sure, if to choose R or Python, then for your task R seems to be an easier and better way to start (especially with RStudio as an interface).

For a bigger list and links to tutorials, take a look at a list of software for scientists.

  • 3
    Having worked with R and Python, I'd suggest using R for the bulk of the data analysis, building an <input> -> csv shim that's as thin as possible in python. For some munging tasks, python is much better, but for data processing R is often much stronger (especially when combined with ggplot2).
    – Matthew G.
    Feb 27 '14 at 17:43
  • So the answer is NO... I thought of .csv, but with MATLAB I can't use it as table with head (meanings of columns) and MATLAB can read .xls and .xlsx files easily.
    – Crowley
    Feb 27 '14 at 18:25
  • @Crowley stackoverflow.com/questions/19613232/… Feb 27 '14 at 18:28
  • @PiotrMigdal Thanks. I'll look closer at RStudio. It seems powerful enough and I like it's licence and OS mobility.
    – Crowley
    Feb 27 '14 at 18:46

I find Google Sheets is easier and more powerful in many ways than excel. I have done a couple of projects with a sheet of raw data coming in via csv, then other sheets to process it. If you're clever, it can be done so that when the raw data are updated, everything else falls into place. Google Charts is basic but has some neat features for looking at data. The Transpose, Filter, Sort and even query (SQL) is very cool if you have lots of data.

You can collaborate in teams, commenting on interesting findings, etc. Graphs output to PNG or PDF look great in latex. Data are available in the cloud, not just on some file server in a lab. Tables are a special kind of graph that can be shared on web pages and have user-selectable options for sorting, etc.

  • 1
    Before starting with google docs make sure their terms of service (google.com/intl/en/policies/terms), particularly the section "Your Content in our Services" agrees with your needs. Mar 1 '14 at 16:54
  • 1
    I should have mentioned I use google apps for education (my uni has a domain) so those terms are likely different than the "free" services. Mar 1 '14 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.