5

(Note: Personal Computer = Laptop)

When reading papers that involves some sort of numerical simulation of a software or custom code, many authors would remark on the specifications of the laptop for which the simulation is conducted. For instance.

enter image description here

Can someone enlighten me as to

What are some of the common specifications should you report? There seems to be no comprehensive list of specs that one must report. For example, the # of cores of your laptop, the brand/series, and RAM as shown above and ... what else, clock speed, frequency of your processor...

Can someone provide additional insights on how these specifications come into play when you are trying to reproduce or compare results? For example, if I simulated some algorithm on a dual core AMDdual opteron processor 275 with 32 GB RAM and try to compare it with the same algorithm quadcore Intel Core i7-2600 with 16 GB RAM, how do I go about commenting on how the simulation platforms affect the simulation results? i.e. does Bigger RAM necessarily means shorter simulation time? Does quadcore laptop necessary implies that the algorithm performs better compared to dual core laptops?

Improve this question
2
  • 1
    It depends largely on what you're simulating, to be honest. Machine learning and AI, obviously better specs lead to better performance, but if you were running a simulation entirely in Java, it boxes its own machine so running it on a gaming rig versus a notebook would likely lead to almost identical performance with default VM settings.
    – Compass
    Commented Feb 9, 2017 at 21:34
  • 7
    The details are not there to enable a completely scientific comparison or benchmarking across different machines, it's just to give other researchers a rough sense of the computational resources that were used/available in case they are considering trying to replicate or modify your simulation or compare it with their own code. You should report anything that seems relevant, e.g., if your code is GPU-based then include the specs of your GPU, if not then don't. If the color of the box the machine comes in matters, report it, otherwise don't. Etc.
    – Dan Romik
    Commented Feb 9, 2017 at 22:04

1 Answer 1

Reset to default
9

A bit old this question, but let's answer. I split it in three different cases:

1. You want to say that the simulation can run in a reasonable time

In this case, I usually write "The simulation run for X hours on a standard workstation.". No details needed, you just want to show that it is feasible to run the simulation and give a very approximate idea, no one cares how much RAM/which CPU you have. The goal is only to show that it is feasible on a normal computer and if it takes 2 hours or 5 hours, it is not very important.

2. You want to provide the exact time

First of all, please don't run it on your laptop. Use a dedicated machine, with the minimal amount of software running to minimize the interference. Depending on the specific domain "minimize the interference" may require simple or very complex strategies. For instance, if you need to obtain the performance of your MATLAB code, you can just close all the other applications and it will be ok. But if you need the performance of a hard real-time program, probably you need to remove the operating system and write bare metal code.

In any case, it is essential to get many samples of the execution time and provide in the article average, variance/standard deviation and number of samples.

Regarding the specification itself, write the most important information: CPU, RAM, Operating System, Compiler/Interpreter version, any other information you think is essential to replicate the experiment.

3. You want to compare your algorithm with another one

This is the most complex case. All the consideration of point 2 applies. However, it is very unlikely that the specification provided by the other article are sufficient for you to correctly replicate the experimental setup, especially in General Purpose computers (for instance, how can you get the same exact software installed? How can you run the CPU at the same exact temperature that may impact performance?).

So, usually, what you should do, it is to replicate the original article experiment in your machine (by impementing their algorithm), where you will run then your algorithm. This is the only way to get a fair comparison.

Direct answers to your questions

does Bigger RAM necessarily means shorter simulation time?

No. Not even increasing the CPU frequency necessarily means reducing the simulation time.

Does quadcore laptop necessary implies that the algorithm performs better compared to dual core laptops?

No, it depends if the algorithm (effectively) exploits parallelism.

Improve this answer
5
  • 2
    People might be interested in the RAM amount if it's close to a hard limit, e.g. the calculation completes with 16 GB but fails (or at least runs considerably slower) with 12 GB.
    – Anyon
    Commented Sep 3 at 13:52
  • 2
    The specific CPU can be important in some circumstances, such as when a hardware bug is discovered after publication that may have affected the results.
    – Matt
    Commented Sep 3 at 21:08
  • 1
    "A bit old this question" Time is relative ;)
    – Fraïssé
    Commented Sep 4 at 18:11
  • If you know the details it is better to give them as you do not know how other people might be able to use the data in the future, especially bearing in mind that (a) you are answering a very old question (b) there is no such thing as a standard workstation (c) the very word 'workstation' can imply extra capabilities above and beyond a personal computer and (d) there is probably no good reason not to give it.
    – Jool
    Commented Sep 7 at 15:02
  • @Jool You can add sure but you are providing no useful information. The execution time on workstation/laptop/mobile phones are likely in the same order of magnitude, so for the sake of the feasibility it doesn't really matter. The point is: if you cannot provide an execution time measured in an accurate way, then it is no interest on the specific machine (clearly, if you used a HPC cluster is a different story...)
    – ocirocir
    Commented Sep 8 at 18:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .