Are results that are derived simply by using more computational power publishable?

Question

Let's say Alice has written a paper that attacks a well-known problem in a clever new way. Her paper ends with "these methods work up to N = 10; for larger N, the calculations are prohibitively expensive".

Bob reads the paper and successfully duplicates Alice's results, but where Alice only had access to a 4-core desktop, Bob has a 4000-core supercomputer. Therefore he can calculate what Alice found prohibitively expensive, and get results up to N = 20.

Are Bob's results publishable? He didn't actually do anything special, he just had more computational power. On the other hand, the results are definitely new, and interesting (for whatever reason N = 20 is the point where interesting effects are predicted).

If they are publishable, what would Bob actually write in such a paper? Everything sans the result could effectively be communicated in one sentence ("Read Alice's paper"). If they are not publishable, how is Bob to communicate his results to the wider world?

I suspect something like this has happened in the past, in which case I'd appreciate a link to such a paper.

Answer 1

(I switched the order of a couple of your questions to give the answer a more logical flow.)

On the other hand, the results are definitely new, and interesting (for whatever reason N = 20 is the point where interesting effects are predicted).

If the results are new, and interesting, then they are by definition publishable. The key point is they add something meaningful to the sum total of human knowledge. That’s certainly a sufficient criterion for publishability (in an idealized sense it should be necessary as well, but I fear it isn’t - plenty of things get published that don’t really meet this condition).

Are Bob's results publishable? He didn't actually do anything special, he just had more computational power.

The concept of “doing something special” is both mostly meaningless and (to the small extent that it has any meaning) completely irrelevant to the question of publishability. Plenty of authors of excellent papers “didn’t do anything special” - they just had more patience and were willing to work a bit harder, or in many cases just got a little bit luckier, than their peers, and ended up discovering something significant. Publication is not a prize that’s given for “doing something special”, it’s simply the scientific community putting a stamp of approval on a discovery that it is correct, new, and worth paying attention to.

By the way, one can even say that a researcher Bob who works harder than his peer Alice is pretty much the exact scenario you are describing of someone “having more computational power”. In this case the computational power is of the human sort, but I don’t see why that should change the calculation.

Edit: another thought is that Bob actually did “do something special”. It’s not just that he had more computational power, but he had the foresight and intuition to realize that it would be worthwhile to apply that computational power to Alice’s particular problem rather than spend his time and resources doing any number of other things. So although it’s not really that important as I said above, even by your own criterion I think there is more merit to the achievement than your description makes it sound.

If they are publishable, what would Bob actually write in such a paper? Everything sans the result could effectively be communicated in one sentence ("Read Alice's paper").

Bob would simply write what he did and what the results were, and what makes them interesting. If it can really be described in one sentence I’d argue that it likely wasn’t such an interesting result after all. But on the off-chance that it was indeed interesting, then it would make for a great one-sentence paper. There isn’t any rule that papers have to be longer than one sentence.

Answer 2

Papers like this can be published, and can be extremely important

There is no inherent reason why a paper of this nature would not be publishable, and indeed, there are some examples where papers of this nature have been published in high-ranking journals and have been extremely important papers. Whether a paper of this nature is publishable largely depends on the importance of being able to compute outcomes of the problem to a higher level. In some cases this will be unimportant, and the paper would be rejected; in other cases it would be extremely important, and the paper could become an important work in the field. Note also that the mere fact that the additional computation does not involve "anything special" from a theoretical standpoint, is often quite irrelevant to the importance of the result.

The most obvious example I can think of here is Shaeffer et al (2007) Checkers is solved. This paper was published in the prestigious Science journal. It extended earlier works computing the game-theoretic solution to the game of checkers. In this paper the authors had managed to obtain sufficient computing power to (weakly) solve the game, and this was a major innovation in the game-theoretic analysis of checkers. According to the authors:

The effort to solve checkers began in 1989, and the computations needed to achieve that result have been running almost continuously since then. At the peak in 1992, more than 200 processors were devoted to the problem simultaneously. The end result is one of the longest running computations completed to date.

In this particular case, the additional computing power accruing over time made the difference between early calculations that could not solve the problem, and later calculations that constituted a solution. This particular paper only weakly solves checkers (meaning that it gives perfect play from the starting point of the game, but it does not give perfect play from every possible board state). The next expected innovation would be to strongly solve checkers (i.e., find perfect play from every possible board state), and I imagine that the team that does this will also get an excellent publication in a high-ranking journal.

Answer 3

In addition to the excellent answers that have already been given, the question indicates to me a common misunderstanding what papers actually are. They are, or at least should be, the communication of ideas, results, or some other type of knowledge that may be of interest to others. They are not badges of honour or grades or brownie points for doing X atomic units of research. In that sense, "how much work was it?", "how difficult was it?", or even "how large is the delta to existing work?" are fundamentally not the right questions to ask when evaluating whether something is publishable. If you have a stroke of genius and write down a cool idea very quickly, it's value to society is in no way diminished. Conversely, spending a lot of time on a bad idea does not make it more valuable to society.

However, as I wrote initially, it's an easy mistake to make because there is, of course, a correlation between effort/complexity and value/interestingness. Simple ideas are not often extremely valuable, because the chance that somebody else had the same idea before is high.

Apply this thinking to your question:

Are Bob's results publishable? He didn't actually do anything special, he just had more computational power. On the other hand, the results are definitely new, and interesting (for whatever reason N = 20 is the point where interesting effects are predicted).

Doing "anything special" isn't in truth what makes papers accepted. If there are interesting effects to be observed for N=20, then go for it. That said, to my ears it does sound a bit suspicious that all that was needed to resolve an open question that people actively find interesting is to throw a little more hardware at it. I would make quite sure that indeed nobody has done it and that there are indeed interesting effects to be observed (and then I would still wonder a little why nobody has done it yet, unless the paper you build upon has appeared like a week ago).

If they are publishable, what would Bob actually write in such a paper?

Summarize the original work to the extent that is needed to actually follow your paper (given the setup this may be quite a lot more detail than what you would typically dedicate to previous work - that's ok, as long as you make it clear that these are not your new results). Describe your setup and experiment. Then, if the interesting results for N=20 are what should carry this publication, describe these results and in which ways they are interesting in a lot of detail. Connect them to previous results for smaller Ns, and, if possible, go to the bottom of why the results start becoming more interesting for higher Ns.

If that does not give you enough "meat" for a normal conference or journal article, there are alternative publication venues dedicated to shorter articles, such as journal notes.

Answer 4

Dan Romik and Reinstate Monica have both given good answers. I'm going to give another answer with a slightly different emphasis with two points.

First, that there are specific journals which are often devoted to this sort of thing or close to it. One of those journals is the Journal of Computational Mathematics. That journal has a wide variety of articles, ranging from very pure number theory to mathematical physics ,but generally involving heavy duty computations.

Second, almost any paper which runs something for longer is going to be able to say something beyond just that they ran it for longer, and if I reviewed a paper, I'd like at least something beyond that. Explanations of any code tweaks they did would be one thing. Another thing would be any patterns they noticed in the data that wasn't obvious, or additional conjectures. Most of the time, if one is implementing something this way one is probably going to be thinking of the problem well enough that one will have at least something additionally novel one can say even if it is very minor.

Answer 5

Are Bob's results publishable? He didn't actually do anything special, he just had more computational power.

What you say is not correct. According to my experience things are not as simple as what you think in supercomputers. It is not just simply copying and pasting the codes. The codes should be prepared in a completely different manner and need some adjusting and tunning in some cases. You do not have that much flexibility in coding in supercomputers as you have on your laptop. In fact, improving an algorithm and making it runnable in supercomputers is itself a challenge and might be worth publishing, even if the final results are not that distinctive.

Answer 6

I think the answer is determined by WHY such a study with N > 10 is conducted. E.g.

1. Show off of computational power, for example, by calculating more decimals for Pi, beyond what is practically needed. About 39 digits are needed to calculate the circumreference of the known universe to the accuracy of a hydrogen atom. Calculating a Pi value beyond N = 40 is a sport with little scientific value per se.

2. Attempt to test a hypothesis (derived from a theory, model, or experiment) that something interesting happens when N is in a certain range. For example, model a material structure consisting of more atoms, providing a better view on how such material behaves.

Both results are publishable!

The second example is inherently very interesting for publishing. Especially, if the results are significant.

The first example could be used to report recent advances in computer performance, report a novel method for scaling up resource-intensive computations, demonstrate the use of a supercomputer on a simple example, or raise public awareness of science and research.

There is also nothing wrong to replicate an earlier study, e.g. by demonstrating how certain algorithms scale from personal computers to supercomputers.

In your paper, you can summarize the earlier study, motivate your angle, and report and explain your results.

Answer 7

A lot of the answers already given are great but something that may help Bob make a decision about whether or not to publish a paper describing his results is to classify his methodology against other fields of research. In a more soft science, what Bob did would absolutely warrant a paper because there is a lot of difference between what he did and what Alice did, even if he followed her methodology to a T. From a hard science perspective, it may be better for Bob to contact Alice and see if they could collaborate on extending her existing paper or creating a new one if they find the scope is larger.

Since computer science kind of falls into a grey area of hard vs soft science and how Bob publishes his results should be based off of his interpretation of what methodology he used. A hard science result can be easily reproduced in the sense of X result was achieved with Y equipment and took Z amount of {time, temperature, pressure, etc}. If Bob had access to Alice's source code and did not have to make any changes to run on his supercomputer, this may be a situation where he would contact Alice to share his results.

If Bob had no access to Alice's source code and rewrote from pseudocode or basic descriptions of the program from her paper, this would very much be a case where Bob should have no reservations about publishing his own paper describing his findings. There is a varying degree of how well authors describe their methodology and even if Alice was very descriptive, Bob could easily find a difference between his methodology and hers if he had to rewrite it, even if it is not immediately obvious.

Ultimately, the decision is up to Bob. A lot of answers above have mentioned there is no harm in more papers and if the result N = 20 is truly interesting, it would certainly merit its' own paper for the knowledge that it would contribute. In my personal opinion, computer science is much closer to a soft science than a hard science. To truly prove some results, it has to be reproducible by many researchers from bare-bones methodology with relatively similar results for something to be conclusively true. There are so many moving parts involved in computers that it really is more comparable to a Psychology than a Physics.

Answer 8

Its completely publishable and called 'secondary literature' (Which is different than secondary source):

It relates or discusses information originally presented elsewhere. A secondary source contrasts with a primary source, which is an original source of the information being discussed.

May fall under a review article too.

As a clarification: Secondary literature is material that interpret, evaluate, or analyze the evidence derived from primary materials and/or sources.