My paper was accepted to an IEEE conference, but I am not getting the simulation results I obtained before.
In IEEE conferences will they tell you to run the simulation in front of them?
-
22"IEEE conference" tells us almost nothing. The IEEE sponsors more than 1000 conferences each year, on everything from computer science to the engineering problems in electric power transformers. Can you be more specific?– semi-extrinsicCommented Sep 25, 2015 at 7:48
-
23For the future, make sure you use the same pseudo-random number generator for all random choices and give it a specific seed. This way, you can always rerun the exact same simulation (assuming your implementation is single-threaded).– Peter BloemCommented Sep 25, 2015 at 10:05
-
8@Peter: Or just record a video of the one simulation run being reported on? Frankly, if a simulation only works for a specific set of random numbers (not a specific set of carefully chosen numbers based upon external factors!), something is wrong about the hypothesis.– O. R. MapperCommented Sep 25, 2015 at 11:12
-
5Do the new simulation result still support the conclusion in the paper? I agree with @Peter, but would add a recommendation to run tests with different seeds to check that the seed does not affect the conclusions.– Patricia ShanahanCommented Sep 25, 2015 at 11:15
-
5@O.R.Mapper Absolutely. The fixed seed lets you go back to the good results to figure out what's wrong. But you still do need to figure out why the seed affects the results.– Peter BloemCommented Sep 25, 2015 at 11:56
|
Show 2 more comments
5 Answers
Since the other answers should properly address your question, I'll raise an issue that bothered me from the go: What do you mean you can't replicate your results?
One of the cornerstones of science is the replication of results (something that is almost never done in computer science...). If you can't replicate your results, how do you expect someone else to? If nobody can replicate/verify your results, sorry, you are not doing science :(
I'm not saying that you are doing it wrong, one would need more information than that, I'm saying that the phrase
but I am not getting the simulation results I obtained before.
is very troubling and can indicate that you are, indeed, doing something wrong.
answered Sep 25, 2015 at 12:16
-
2Actually, my hard disk crashed, so the original results can't be recovered. I ran a second simulation but the results deviated from my original results in the paper.– tumpaCommented Sep 25, 2015 at 15:20
-
18Nevertheless, the article itself should be a roadmap with complete instructions on how/why you arrived at those results and how to replicate them. That's what research articles are for. If you, the original author, can't duplicate the results based on what is written on the paper, the same argument applies. Commented Sep 25, 2015 at 15:38
-
18@tumpa "hard disk crash" is equivalent to "the dog ate my homework"; that is, no one will find that an acceptable excuse. Backing up to a thumb drive is so easy these days that there's just no reason to not do so for your critical work. And everything that everyone else said about replication of results.– mswCommented Sep 25, 2015 at 16:23
-
@msw, I can't not agree especially as someone who lost the entire HDD (with both of my partitions...) a month ago, but since I backed everything crucial (code) up, it wasn't a big deal. I mean, it was annoying to reinstall everything. As far as the original question is concerned though, I would follow the "if you cannot replicate, don't publish it" mantra. Far too often have I found great results in publications that absolutely do not work as well when implemented.– MewaCommented Sep 25, 2015 at 17:14
-
5@tumpa For hard disk crashes and other acts of God, proactively create an online repo on say Github. Check in your code and result files periodically. Even if your current demo doesn't work, you'll have a much better idea of what went wrong, as you step through your comment log.– PKGCommented Sep 25, 2015 at 19:51
I have not seen any conference in which normal track papers where required to show simulations. However, you may very well have questions about your simulation and results. For that reason, I would be careful of presenting something in which you yourself can not replicate. At the very least, depending on how different your results are now than they were before, you may need to make it clear in your presentation that you are aware there may be some issues with the accuracy.
answered Sep 25, 2015 at 9:18
So far, in none of the IEEE conferences I have attended have I seen a compulsion of a demo from the presentation critics. Though some delegates do present a demo, but it would be of their own wish.
Thus, you wouldn't have to worry about giving a simulation at the conference. The average time given to you would only be from 15-25 minutes for your presentation. You should be spending enough time explaining your work.
(Nevertheless, you ought to know what is wrong with your simulation for the good of your research concerned)
Apart from what others have already mentioned here, about your inability to obtain the same results again being a bigger concern than just presenting a demo, I would also remind you that if someone from the audience asks some difficult questions, you could be caught in an awkward spot.
So, here is my advice - forget about whether you need to show a live demo or not, first figure out what is wrong with your methodology or your hypothesis. If you can justify that, then don't worry about the live demo, and if you cannot justify that, then you still have to do a lot of work.
-
1+1 for "forget about whether you need to show a live demo or not". Commented Sep 26, 2015 at 10:16
As the other answers have noted, showing a live demo is the least of your concerns. If your results are correct, and you understand the methods you've used to obtain them, then everything is fine. If not, you have bigger problems anyway.
You don't say exactly what you mean by "not getting the simulation results I obtained before", but whatever it is, it's worrying:
Your earlier results might have been a statistical fluke, or they might have been representative of only a small fraction of the parameter space.
Alternatively, if your old code consistently gave different results than you new simulation code does for identical parameter values, then either your old code or the new code (or both!) must have a bug.
Until you figure out what the cause of the discrepancy is, and verify (e.g. by comparison with earlier work and/or an independent reimplementation) that you simulations are actually producing correct results, there's a good chance that at least some of your results are bogus, and any conclusions you may have drawn based on them may be simply wrong.
The good news is that, by a lucky accident, you caught this problem now, before the conference. Even if you can't verify or correct your results in time, and end up having to withdraw your paper from the conference, that's still a lot less embarrassing than having someone else refute your results after (or, perhaps worse yet, during) the public presentation.
At this point, the two questions you should ask yourself are:
Can you honestly and confidently stand in front of your peers and superiors, and answer questions about how you got your results and why they are the way they are, with full confidence that your claims, to the extent you've made them, really hold up to scrutiny?
If not, what's the least embarrassing way to retract your claims, at least temporarily until you can reconfirm them, and how soon do you need to do that?
(Tip: It's almost certainly not waiting until just before you need to present your paper. It's most likely also not trying to bullshit your way through the presentation with dodgy data, and having someone in the audience call you on it.)
In any case, if you haven't discussed the problem with your supervisor(s), do that now. This is one of the things it's their job to be aware of and to help you deal with, and they'll (hopefully) have the experience to suggest a reasonable course of action in your specific situation.
