84

Let's say Alice has written a paper that attacks a well-known problem in a clever new way. Her paper ends with "these methods work up to N = 10; for larger N, the calculations are prohibitively expensive".

Bob reads the paper and successfully duplicates Alice's results, but where Alice only had access to a 4-core desktop, Bob has a 4000-core supercomputer. Therefore he can calculate what Alice found prohibitively expensive, and get results up to N = 20.

Are Bob's results publishable? He didn't actually do anything special, he just had more computational power. On the other hand, the results are definitely new, and interesting (for whatever reason N = 20 is the point where interesting effects are predicted).

If they are publishable, what would Bob actually write in such a paper? Everything sans the result could effectively be communicated in one sentence ("Read Alice's paper"). If they are not publishable, how is Bob to communicate his results to the wider world?

I suspect something like this has happened in the past, in which case I'd appreciate a link to such a paper.

  • 14
    I’m pretty sure that if Bob did write a paper about something like this, it’d include the standard sections (e.g. abstract, introduction, methodology, results, conclusion), and cite Alice in the introduction where he’s providing context about the research problem and why it’s important. – nick012000 Nov 13 at 3:21
  • 51
    I don't think this is exactly what you're looking for, but this paper was only possible because of increased computational power, and almost any first-year CS undergrad today could write the program to do the search. – chipbuster Nov 13 at 3:27
  • 28
    Many results at the Large Hadron Collider were achieved 'simply' because they had a bigger particle accelerator. Perhaps there is a parallel there? – Time4Tea Nov 13 at 14:29
  • 2
    @Time4Tea You juuust beat me to it! Maybe I can write a longer comment? I was trying to remember the name of the next-largest collider that the LHC surpassed. – user3067860 Nov 13 at 14:31
  • 6
    At the very list, Bob replicates Alice's results. Depending on the results, that might not be trivial and thus be valuable – Syfer Polski Nov 13 at 17:59
71

(I switched the order of a couple of your questions to give the answer a more logical flow.)

On the other hand, the results are definitely new, and interesting (for whatever reason N = 20 is the point where interesting effects are predicted).

If the results are new, and interesting, then they are by definition publishable. The key point is they add something meaningful to the sum total of human knowledge. That’s certainly a sufficient criterion for publishability (in an idealized sense it should be necessary as well, but I fear it isn’t - plenty of things get published that don’t really meet this condition).

Are Bob's results publishable? He didn't actually do anything special, he just had more computational power.

The concept of “doing something special” is both mostly meaningless and (to the small extent that it has any meaning) completely irrelevant to the question of publishability. Plenty of authors of excellent papers “didn’t do anything special” - they just had more patience and were willing to work a bit harder, or in many cases just got a little bit luckier, than their peers, and ended up discovering something significant. Publication is not a prize that’s given for “doing something special”, it’s simply the scientific community putting a stamp of approval on a discovery that it is correct, new, and worth paying attention to.

By the way, one can even say that a researcher Bob who works harder than his peer Alice is pretty much the exact scenario you are describing of someone “having more computational power”. In this case the computational power is of the human sort, but I don’t see why that should change the calculation.

Edit: another thought is that Bob actually did “do something special”. It’s not just that he had more computational power, but he had the foresight and intuition to realize that it would be worthwhile to apply that computational power to Alice’s particular problem rather than spend his time and resources doing any number of other things. So although it’s not really that important as I said above, even by your own criterion I think there is more merit to the achievement than your description makes it sound.

If they are publishable, what would Bob actually write in such a paper? Everything sans the result could effectively be communicated in one sentence ("Read Alice's paper").

Bob would simply write what he did and what the results were, and what makes them interesting. If it can really be described in one sentence I’d argue that it likely wasn’t such an interesting result after all. But on the off-chance that it was indeed interesting, then it would make for a great one-sentence paper. There isn’t any rule that papers have to be longer than one sentence.

  • I remember reading a list of the shortest published papers. (paraphrased) "Here is how to do X with Y units: <diagram>" (this was a mathematical paper for quite a simple problem) – user253751 Nov 13 at 13:03
  • 4
    +1, but I think you misinterpreted the OP a little in your final paragraph. OP knows that the results probably need more than one sentence - "Everything sans the result could effectively be communicated in one sentence". The one sentence OP refers to is the one that references the previous paper, rather than rewriting what has already been written there. – JBentley Nov 13 at 13:46
  • 5
    It seems interesting (for once!) to consider edge cases. Suppose for instance that Alice's paper contains a link to code, and Bob downloads the code and uses it on his computer without change. Then it seems that Bob has no intellectual contribution and should not write a paper. As for having the insight to devote his resources to it: well, maybe Alice's hiring committee should be authors on her paper? Or the governor of the state if she is at a state university? And so forth. – Pete L. Clark Nov 13 at 18:12
  • 6
    @PeteL.Clark I’m not really into analyzing such how-many-angels-can-dance-on-the-head-of-a-pin-type hypotheticals myself, but I‘d be happy to read an answer discussing this if you’d care to write one. – Dan Romik Nov 13 at 18:27
  • 6
    Got any examples of one sentence papers? I think "This repeats Alice's paper with N=20. " Together with all the expanded tables and charts is likely to be desk rejected. My experience is that editors and referees will make you write at least a halfway decent literature review (or cite some), explaining the previous work and why going from N=10 to N=20 allows comparison of theories, resolves a controversy, or is otherwise an interesting thing to do. – Paul Nov 13 at 21:37
27

Papers like this can be published, and can be extremely important

There is no inherent reason why a paper of this nature would not be publishable, and indeed, there are some examples where papers of this nature have been published in high-ranking journals and have been extremely important papers. Whether a paper of this nature is publishable largely depends on the importance of being able to compute outcomes of the problem to a higher level. In some cases this will be unimportant, and the paper would be rejected; in other cases it would be extremely important, and the paper could become an important work in the field. Note also that the mere fact that the additional computation does not involve "anything special" from a theoretical standpoint, is often quite irrelevant to the importance of the result.

The most obvious example I can think of here is Shaeffer et al (2007) Checkers is solved. This paper was published in the prestigious Science journal. It extended earlier works computing the game-theoretic solution to the game of checkers. In this paper the authors had managed to obtain sufficient computing power to (weakly) solve the game, and this was a major innovation in the game-theoretic analysis of checkers. According to the authors:

The effort to solve checkers began in 1989, and the computations needed to achieve that result have been running almost continuously since then. At the peak in 1992, more than 200 processors were devoted to the problem simultaneously. The end result is one of the longest running computations completed to date.

In this particular case, the additional computing power accruing over time made the difference between early calculations that could not solve the problem, and later calculations that constituted a solution. This particular paper only weakly solves checkers (meaning that it gives perfect play from the starting point of the game, but it does not give perfect play from every possible board state). The next expected innovation would be to strongly solve checkers (i.e., find perfect play from every possible board state), and I imagine that the team that does this will also get an excellent publication in a high-ranking journal.

  • I don't think the Checkers paper is an example, as they did develop new methods in the paper, the result was not obtained "simply by having more computational power". – JiK Nov 13 at 12:45
  • 5
    @JiK: From reading the paper, it does not appear to me that there is much (if anything) in methodological innovation over the previous runs. The paper cites previous literature that establishes the method, and also cites previous papers that used that method on checker boards with less pieces. So the method for the computations was developed in earlier papers, and already applied to smaller versions of the problem. The advent of greater computing power then allowed the computation to be run for a board with all the pieces, resulting in this paper. – Reinstate Monica Nov 13 at 12:54
13

In addition to the excellent answers that have already been given, the question indicates to me a common misunderstanding what papers actually are. They are, or at least should be, the communication of ideas, results, or some other type of knowledge that may be of interest to others. They are not badges of honour or grades or brownie points for doing X atomic units of research. In that sense, "how much work was it?", "how difficult was it?", or even "how large is the delta to existing work?" are fundamentally not the right questions to ask when evaluating whether something is publishable. If you have a stroke of genius and write down a cool idea very quickly, it's value to society is in no way diminished. Conversely, spending a lot of time on a bad idea does not make it more valuable to society.

However, as I wrote initially, it's an easy mistake to make because there is, of course, a correlation between effort/complexity and value/interestingness. Simple ideas are not often extremely valuable, because the chance that somebody else had the same idea before is high.

Apply this thinking to your question:

Are Bob's results publishable? He didn't actually do anything special, he just had more computational power. On the other hand, the results are definitely new, and interesting (for whatever reason N = 20 is the point where interesting effects are predicted).

Doing "anything special" isn't in truth what makes papers accepted. If there are interesting effects to be observed for N=20, then go for it. That said, to my ears it does sound a bit suspicious that all that was needed to resolve an open question that people actively find interesting is to throw a little more hardware at it. I would make quite sure that indeed nobody has done it and that there are indeed interesting effects to be observed (and then I would still wonder a little why nobody has done it yet, unless the paper you build upon has appeared like a week ago).

If they are publishable, what would Bob actually write in such a paper?

Summarize the original work to the extent that is needed to actually follow your paper (given the setup this may be quite a lot more detail than what you would typically dedicate to previous work - that's ok, as long as you make it clear that these are not your new results). Describe your setup and experiment. Then, if the interesting results for N=20 are what should carry this publication, describe these results and in which ways they are interesting in a lot of detail. Connect them to previous results for smaller Ns, and, if possible, go to the bottom of why the results start becoming more interesting for higher Ns.

If that does not give you enough "meat" for a normal conference or journal article, there are alternative publication venues dedicated to shorter articles, such as journal notes.

8

Dan Romik and Reinstate Monica have both given good answers. I'm going to give another answer with a slightly different emphasis with two points.

First, that there are specific journals which are often devoted to this sort of thing or close to it. One of those journals is the Journal of Computational Mathematics. That journal has a wide variety of articles, ranging from very pure number theory to mathematical physics ,but generally involving heavy duty computations.

Second, almost any paper which runs something for longer is going to be able to say something beyond just that they ran it for longer, and if I reviewed a paper, I'd like at least something beyond that. Explanations of any code tweaks they did would be one thing. Another thing would be any patterns they noticed in the data that wasn't obvious, or additional conjectures. Most of the time, if one is implementing something this way one is probably going to be thinking of the problem well enough that one will have at least something additionally novel one can say even if it is very minor.

6

Are Bob's results publishable? He didn't actually do anything special, he just had more computational power.

What you say is not correct. According to my experience things are not as simple as what you think in supercomputers. It is not just simply copying and pasting the codes. The codes should be prepared in a completely different manner and need some adjusting and tunning in some cases. You do not have that much flexibility in coding in supercomputers as you have on your laptop. In fact, improving an algorithm and making it runnable in supercomputers is itself a challenge and might be worth publishing, even if the final results are not that distinctive.

  • 1
    @Aaron: so what if the code can get compiled? Unless the is written to use the supercomputer's features (like very wide buses for parallel floating point calculations, dispatch systems to run thousands or millions of jobs in parallel, etc), the program is wasting the supercomputer. Making an algorithm work on a supercomputer is a nontrivial effort and often worth publishing in itself. – nomen Nov 15 at 6:23
  • 1
    @Aaron I wasn't trying to be accurate. I was just trying to share my own experience, which is true in many supercomputers. Now I don't want to talk about the obstacles related to working with supercomputers. But the obvious point is that things are not as idealized as you have stated. – KratosMath Nov 15 at 7:03
  • 2
    @Aaron I think you are technically correct that the answer doesn’t strictly answer the question, instead it challenges the premise of the question by pointing out that it is in fact rare that one can squeeze better computational performance on a problem by simply compiling and running someone else’s code on a faster computer. But keep in mind that such “frame challenge” answers are very common on this (and other) stack exchange sites and can be valuable and add useful insight despite technically “not answering the question”. In particular I think this answer is interesting and upvoted it. – Dan Romik Nov 15 at 14:25
  • @Aaron: that's what Kratos said and you "corrected". – nomen Nov 15 at 22:01
  • 1
    @Aaron: "What you say is not correct. According to my experience things are not as simple as what you think in supercomputers. It is not just simply copying and pasting the codes. The codes should be prepared in a completely different manner and need some adjusting and tunning in some cases. You do not have that much flexibility in coding in supercomputers as you have on your laptop. " Hope this helps. – nomen Nov 15 at 22:19
3

I think the answer is determined by WHY such a study with N > 10 is conducted. E.g.

  1. Show off of computational power, for example, by calculating more decimals for Pi, beyond what is practically needed. About 39 digits are needed to calculate the circumreference of the known universe to the accuracy of a hydrogen atom. Calculating a Pi value beyond N = 40 is a sport with little scientific value per se.

  2. Attempt to test a hypothesis (derived from a theory, model, or experiment) that something interesting happens when N is in a certain range. For example, model a material structure consisting of more atoms, providing a better view on how such material behaves.

Both results are publishable!

The second example is inherently very interesting for publishing. Especially, if the results are significant.

The first example could be used to report recent advances in computer performance, report a novel method for scaling up resource-intensive computations, demonstrate the use of a supercomputer on a simple example, or raise public awareness of science and research.

There is also nothing wrong to replicate an earlier study, e.g. by demonstrating how certain algorithms scale from personal computers to supercomputers.

In your paper, you can summarize the earlier study, motivate your angle, and report and explain your results.

2

A lot of the answers already given are great but something that may help Bob make a decision about whether or not to publish a paper describing his results is to classify his methodology against other fields of research. In a more soft science, what Bob did would absolutely warrant a paper because there is a lot of difference between what he did and what Alice did, even if he followed her methodology to a T. From a hard science perspective, it may be better for Bob to contact Alice and see if they could collaborate on extending her existing paper or creating a new one if they find the scope is larger.

Since computer science kind of falls into a grey area of hard vs soft science and how Bob publishes his results should be based off of his interpretation of what methodology he used. A hard science result can be easily reproduced in the sense of X result was achieved with Y equipment and took Z amount of {time, temperature, pressure, etc}. If Bob had access to Alice's source code and did not have to make any changes to run on his supercomputer, this may be a situation where he would contact Alice to share his results.

If Bob had no access to Alice's source code and rewrote from pseudocode or basic descriptions of the program from her paper, this would very much be a case where Bob should have no reservations about publishing his own paper describing his findings. There is a varying degree of how well authors describe their methodology and even if Alice was very descriptive, Bob could easily find a difference between his methodology and hers if he had to rewrite it, even if it is not immediately obvious.

Ultimately, the decision is up to Bob. A lot of answers above have mentioned there is no harm in more papers and if the result N = 20 is truly interesting, it would certainly merit its' own paper for the knowledge that it would contribute. In my personal opinion, computer science is much closer to a soft science than a hard science. To truly prove some results, it has to be reproducible by many researchers from bare-bones methodology with relatively similar results for something to be conclusively true. There are so many moving parts involved in computers that it really is more comparable to a Psychology than a Physics.

-4

Its completely publishable and called 'secondary literature' (Which is different than secondary source):

It relates or discusses information originally presented elsewhere. A secondary source contrasts with a primary source, which is an original source of the information being discussed.

May fall under a review article too.

As a clarification: Secondary literature is material that interpret, evaluate, or analyze the evidence derived from primary materials and/or sources.

  • 11
    This answer is wrong. Doing additional computations is not review; one is doing something beyond what is in the literature. Nor is doing additional computations purely discussion of pre-existing primary sources, since again one is doing something which those sources did not. – JoshuaZ Nov 13 at 12:16
  • I think you might be mistaken about what a "secondary source" is. – Sneftel Nov 14 at 8:42
  • 1
    @ Sneftel I did not spoke about a secondary source. I spoke about secondary literature. – deags Nov 14 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.