8

Generally in the numerical section for papers I read, they include (if at all) computer specs like how much RAM, CPU speed, or type of processor. What kinds of computers specs should I include? When should I include them? Why? Some general scenarios when you don’t need to include the specs would be helpful. As I am writing an interdisciplinary paper, I would also like to know about a possible dependency on the field.

For this question I would especially like to know why. I know the basic idea is reproducible research, but are there really any cases where it is imperative to know it was an i5 Intel processor?

Improve this question
10
  • 2
    Certainly - I'm currently working on debugging something that's an OS-level issue and possibly tied to the processor. It would be helpful for someone working on reproducing your experiments to know that their AMD chip (for example) might be causing some differences.
    – Azor Ahai -him-
    Commented Mar 2, 2018 at 19:30
  • 2
    Yes, it may matter that it was an i5; specifically, its cache size will affect lots of timings. For instance, matrix multiplication gets a lot slower when the matrices to operate on stop fitting in the cache.
    – Federico Poloni
    Commented Mar 2, 2018 at 22:33
  • 3
    Do you plan to report timings in your paper? If method A uses N memory entries, and method B uses 2N, then there is going to be a choice of the dimension N for which algorithm A will operate in-cache and B will no. So the timings will be skewed in favor of A, for a range of values of N that depends on the processor used.
    – Federico Poloni
    Commented Mar 4, 2018 at 7:56
  • 1
    A tangent issue: you should also care for statistical errors in measurement: warming up, mean vs. median vs. best time vs. worst time, discarding the outliers, such things. There are special tools to do most of this for you.
    – Oleg Lobachev
    Commented Mar 13, 2018 at 13:31
  • 1
    @MathIsKey: Modern computers are complex, and the execution time for some piece of software can/will vary. One measurement alone will be useless. A computer may move from “saving power” to “running at maximum speed” to “overheating and slowing down”.
    – gnasher729
    Commented Nov 20, 2022 at 15:29

1 Answer 1

Reset to default
16

First, some (hopefully) obvious remarks:

  • you only need to present CPU specs if you are going to publish CPU timings
  • CPU timings depend on variety of factors, including hardware specs, OS, drivers, libraries, software, configuration of this software, etc. It is unlikely that the combination of all these factors will be reproduced exactly.
  • Even a very detailed list of hardware specs that can be published in an academic paper is likely to be incomplete: for example you can say that CPU is i5, but will you mention the rev? will you mention the microcode version? will you mention how well the CPU is cooled and how recently the thermopaste was changed?
  • The same is probably true for the software: it is unlikely that one can report versions of all drivers / libraries / software involved in the process.

It becomes clear that any publishable list of specs will not be sufficient to reproduce the settings. In my opinion, the exact reproduceability is not achievable and hence is not the aim.

I always provide and use the CPU specs as a rough guide that helps me to understand the behaviour of the CPU timings reported as a function of problem parameters. Can this slowdown be due to the data falling off the CPU cache? Can it be due to insufficient memory and hence disk swapping? Does this software allow to use all CPU cores or is only one core active? Can we (roughly) compare the timings reported in this paper with the timings reported by competitors, or are the setups too dissimilar?

At least once in my own experience I encountered an example of scientific fraud, when the reported timings could not be achieved on the declared CPU due to trivial complexity estimations — it could be easily shown that even at 100% efficiency the stated CPU could not perform the calculations needed in time claimed. In this case the details of the setup were handy to make the case.

Final note: reporting CPU specs is important, but making the code available is far more important and I wish people in academia did it more often!

Improve this answer
5
  • 4
    Re: the final note: "code AND data".
    – Matteo
    Commented Mar 13, 2018 at 12:17
  • 1
    I have realized that I know very little about computer specs but your final note has given me some clarity. I will report CPU specs that most everyone reports and then make sure code (and data) are available in github or something like it
    – MathIsKey
    Commented Mar 13, 2018 at 15:37
  • 1
    RAM could also be important, there is a world of difference between in-RAM computations and starting to swap out. Your penultimate paragraph is important in a broader sense, too: reproducibility is rarely of a primary concern, but knowing if the problem could even be solved at all in real-time or close using a described method could save a lot of time and money. If code runs at 10 FPS on RPi4, it may be fine for a robotics project, if these 10 FPS are achieved on 7950X instead, probably not so much.
    – Lodinn
    Commented Nov 20, 2022 at 23:32
  • @Lodinn This is right. In scientific computing we (almost) always assume there is no swapping -- everything is either in GPU/CPU cache or RAM. When data hit swap it totally kills performance.
    – Dmitry Savostyanov
    Commented Nov 21, 2022 at 10:02
  • 1
    Yep, and then there is an issue of fitting data in RAM/VRAM in the first place, as not being able to do so often requires a non-trivial amount of tinkering. These days we have tensorflow OOMing, before it could be "if you have a similar data source to ours, you better also have at least 32 Gb of RAM to run this algorithm (and be happy it is not pushing 100)". For someone looking to modernize their lab, seeing these numbers in paper after paper is also extremely helpful to calibrate their expectations. Memory could really make all the difference between running an algorithm and not.
    – Lodinn
    Commented Nov 22, 2022 at 12:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .