I am doing an experiment assumed to be performed in ideal environment although it is not, e.g. resource is fully allocated from a resource pool shared between different organisations. As my experiment is affected by external factors (e.g. a shared pool may not fully allocate all requested resource) the result sometime is different from what I expect. However, since my model is proven to work in an ideal environment, I am sure that if I keep repeating the experiment, I will get a result which is consistent with my calculation, e.g. when a shared pool have enough resource to allocate to my request.

Notably, when I run an experiment, I do it few times to get the average result. Moreover, I'm aware that an environment is not ideal and planning to address it in my future research.

So, is it acceptable to keep repeating the experiment until I get the (average) result which is consistent with my calculation? Furthermore, should I mention in my paper how many times I perform my experiment to get the presented result or just mention briefly about the imperfect environment and a plan to handle it in future research?

Update: my research aims to use shared or volunteer resources to perform computation. It is in the early state when I calculate the required resource for a job prior to its execution. In other words, during an execution, I assume that there are enough resources for me, which doesn't always happen. As I said, I'm planning to investigate dynamic calculation in the future.

  • You should always mention that your result is a combination of runs if that's what it is. What sort of resource is this? Can you beg some dedicated time on it to verify that contention is the real problem? – Bill Barth Jun 18 '14 at 0:28
  • 4
    You are attempting to achieve a desired result? This doesn't sound like science to me. – Brian P Jun 18 '14 at 2:53
  • 2
    I think what you are trying to ask is not "should I repeat until I get desired result" but actually "should I repeat until I get a consistent result" – ff524 Jun 18 '14 at 6:40
  • @ff524: I've updated my question based on you suggestion, thanks ;) – user12635 Jun 18 '14 at 7:15
  • 1
    I have deleted my answer as it seems not particularly applicable after the OP's last edit / clarification. I will try to provide an update later. Actually, nevermind, @ff524 already sums up the main points nicely. – xLeitix Jun 18 '14 at 10:19

When you run an experiment that is expected to have some variability (because it is not in a perfectly controlled environment) then you must run it a large number of times. Not "until you get the results that are consistent with what you expect" but "until you get results that are reasonably consistent with one another."

Then, when you describe the results, you must say something about how they were distributed (not just give the average). For example:

  • "The average execution time of Protocol A was 4.72 ms, with a variance of 0.4 ms. The average execution time of Protocol B was 5.78 ms, with a variance of 0.3 ms."
  • "The measured results were clustered in two groups, with 75% of runs falling between 1.3-1.9 ms and 25% of runs falling between 10.5-13.3 ms. We speculate that this is because of X, but we cannot measure or correct for X at this stage."

You get the idea. The point is that you present not just a single numerical value, but also some measure of how consistent the values were.

And yes, you must describe how many times you ran the experiment, whatever you know about the conditions under which the experiments ran, and how you calculated the summary statistics.

  • In addition: If you report meticulously what you are doing and have "external" information whether the requested resources were granted (i.e. you get feedback on the actually granted resources), you can even say that for further analysis you separate the analysis for those k out of the n runs that meet your assumptions from those n - k runs where assumptions were not met. – cbeleites Jun 18 '14 at 10:57

No, you shouldn't do the experiment until you get the desired result. This is poor science and borderline unethical. (I'm sure you had no ill intent and that you are trying to deal with real-world complications.)

An experiment is flawed if it is run n times but only the results from n - 1 are used and reported. (Or anything less than n). See Peter Norvig's page under the section "Warning Sign I2: Ignoring Publication Bias".

Beyond this, I believe that your experimental design is flawed. You shouldn't run an experiment assuming an ideal environment when in fact the experimental environment is not ideal. Only in theoretical models do you have the liberty to idealize your environment. In laboratory experiments, you control the environment so you can idealize as much as possible. What you have is an experiment with confounding factors, which is not uncommon in real-world settings.

The best thing to do is to estimate the effects of the confounding factors through analysis of a theoretical model or a simulation. In your case, which sounds like a queuing theory problem, a statistical simulation should be relatively easy.

With this estimate, you should be able to restate your hypotheses, essentially reducing the expected effects by the "loss" associated with "the shared pool not fully allocating resources when requested".

  • Considering that the question is actually asked immediately assuming that the OP's intentions are unethical is IMHO jumping to conclusions: without intention, the OP may be headed for wrong data analysis, but mistakes are not unethical unless there's an intention. Note that the OP is talking about averaging, which does not discard measurements. I come from an experimental field and consider it positively bad scientific advise to recommend against multiple runs (as opposed to a proper evaluation of the results of multiple runs): -1 for now, hoping for a revision that I can upvote instead. – cbeleites Jun 18 '14 at 10:49
  • It is perfectly OK to compare theoretical predictions to actual experiments in order to measure the domain of applicability of the theory, generate hypotheses about confounders, or just to measure the reliability of the predictions as they are at the moment. While the model may be expanded with known confounders, there is nothing nessecarily flawed in the DoE so far. – cbeleites Jun 18 '14 at 10:53
  • It is also OK to restrict the applicability domain to certain runs, e.g. those where feedback about actually allocated resources indicates that the requests were met as long as this restriction (clustering into separately analysed groups) follows criteria which are completely described. – cbeleites Jun 18 '14 at 11:00

As I understood, you are developing a parallel algorithm (or a computation strategy) that is supposed to work on a highly non-uniform computing environment (both in time and "space" of a CPU network). At a current stage of research, you are not yet ready to address the non-uniformness, and have to make a (significant) assumption that your computational network is in fact homogeneous.

What I would suggest - do not repeat the experiment many time, until the (non-uniform by nature) network will suddenly become homogeneous. It would not.

Instead, you could go do the computations on a proper parallel (cluster) station available in your Uni or through a shared subscription. Make sure that during the computations all used CPU's are fully allocated to your problem only, i.e. your (significant) assumption holds.

Do the experiment once, report your assumptions and results clearly, explain which adjustments for the algorithms are required to put it on a heterogeneous CPU network. Hopefully this would suffice.

  • 1
    I understand the recommendation of a more stable computing environment, but running the experiment only once seems like a really bad idea. Can you clarify? – JeffE Jun 18 '14 at 10:49
  • @JeffE Happily, but could you clarify the question first? My general point is that a deterministic computation is not affected by outside factors, and should reproduce the same results. This also includes the computation time. – Dmitry Savostyanov Jun 18 '14 at 11:03
  • 2
    @DmitrySavostyanov in ideal conditions, yes; but running several times can give you a measurement on how ideal your conditions actually are. Even if you are alone in the cluster, caches, IO scheduling, random collisions between packets... can affect performance. – Davidmh Jun 18 '14 at 11:33
  • @JeffE I would agree with this. However, I'd like to epmhasise the difference between "run the code once, and maybe couple more times for piece of mind" and "repeat the execution many times until some condition is (magically) satisfied". – Dmitry Savostyanov Jun 18 '14 at 11:39
  • 1
    "The code has the same performance every time it's run" is a hypothesis to be tested (and the test to be reported in the paper). Assuming it holds a priori is, to borrow a phrase, an undergraduate mistake. – JeffE Jun 18 '14 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.