6

Generally in the numerical section for papers I read, they include (if at all) computer specs like how much RAM, CPU speed, or type of processor. What kinds of computers specs should I include? When should I include them? Why? Some general scenarios when you don’t need to include the specs would be helpful. As I am writing an interdisciplinary paper, I would also like to know about a possible dependency on the field.

For this question I would especially like to know why. I know the basic idea is reproducible research, but are there really any cases where it is imperative to know it was an i5 Intel processor?

  • 1
    Certainly - I'm currently working on debugging something that's an OS-level issue and possibly tied to the processor. It would be helpful for someone working on reproducing your experiments to know that their AMD chip (for example) might be causing some differences. – Azor Ahai Mar 2 '18 at 19:30
  • 1
    Yes, it may matter that it was an i5; specifically, its cache size will affect lots of timings. For instance, matrix multiplication gets a lot slower when the matrices to operate on stop fitting in the cache. – Federico Poloni Mar 2 '18 at 22:33
  • @FedericoPoloni What if for my paper, I am only interested in comparing several methods and not necessarily the software-hardware interface. I feel as long as all experiments are run on the same type of computer then it shouldn't make a difference. – MathIsKey Mar 3 '18 at 20:56
  • 1
    Do you plan to report timings in your paper? If method A uses N memory entries, and method B uses 2N, then there is going to be a choice of the dimension N for which algorithm A will operate in-cache and B will no. So the timings will be skewed in favor of A, for a range of values of N that depends on the processor used. – Federico Poloni Mar 4 '18 at 7:56
  • @FedericoPoloni By same type of computer I meant exact same computer but I do get what you mean. The problem is since I don't how the guts of a computer then I don't know what others would care to know. I will edit the question to maybe ask a better question. – MathIsKey Mar 12 '18 at 20:13
11

First, some (hopefully) obvious remarks:

  • you only need to present CPU specs if you are going to publish CPU timings
  • CPU timings depend on variety of factors, including hardware specs, OS, drivers, libraries, software, configuration of this software, etc. It is unlikely that the combination of all these factors will be reproduced exactly.
  • Even a very detailed list of hardware specs that can be published in an academic paper is likely to be incomplete: for example you can say that CPU is i5, but will you mention the rev? will you mention the microcode version? will you mention how well the CPU is cooled and how recently the thermopaste was changed?
  • The same is probably true for the software: it is unlikely that one can report versions of all drivers / libraries / software involved in the process.

It becomes clear that any publishable list of specs will not be sufficient to reproduce the settings. In my opinion, the exact reproduceability is not achievable and hence is not the aim.

I always provide and use the CPU specs as a rough guide that helps me to understand the behaviour of the CPU timings reported as a function of problem parameters. Can this slowdown be due to the data falling off the CPU cache? Can it be due to insufficient memory and hence disk swapping? Does this software allow to use all CPU cores or is only one core active? Can we (roughly) compare the timings reported in this paper with the timings reported by competitors, or are the setups too dissimilar?

At least once in my own experience I encountered an example of scientific fraud, when the reported timings could not be achieved on the declared CPU due to trivial complexity estimations — it could be easily shown that even at 100% efficiency the stated CPU could not perform the calculations needed in time claimed. In this case the details of the setup were handy to make the case.

Final note: reporting CPU specs is important, but making the code available is far more important and I wish people in academia do it more often!

  • 2
    Re: the final note: "code AND data". – Matteo Mar 13 '18 at 12:17
  • 1
    I have realized that I know very little about computer specs but your final note has given me some clarity. I will report CPU specs that most everyone reports and then make sure code (and data) are available in github or something like it – MathIsKey Mar 13 '18 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.