Summary
I have recently submitted a paper to a fairly prestigious journal in my subfield of mathematics, along with a co-author. I am a junior author, but my co-author is well-established. It's not the top journal in the subfield, but it is well-considered -- one wouldn't think "wow, amazing job!" [getting a paper there], but more "that's a solid, reputable journal, congrats".
About a week ago we received a rejection letter, stating as the primary reason that it was not sufficiently novel -- there have been two other papers on a similar topic (although these are far from complete answers).
My co-author was pretty angry by this -- I have very little experience, and so don't really know how these things happen. They even emailed the editor questioning it (s/he says s/he has never done this before, but felt this was extreme enough to warrant doing so); the editor said that s/he did not discuss such things, and would not go against referees. (The editor has now finished his/her term.)
The journal uses an electronic-style submission, via an online management system. As such, I assume that the new editor would be able to see that I had submitted before; as such, it would seem the done thing for me to add a cover letter, explaining that this is a resubmission.
Regarding why I might want to stay with the same journal, it stands slightly alone in terms of reputation. In the field, there are a few top journals, say with ratings 9 or 10 out of 10; then there's this one, at say 7 out of 10; then the remainder are 5 out of 10 or below. I'm not sure my paper warrants the top journals, so I would like this one if possible.
Further Context; can be skipped if desired, with explicit question below
There were two full reviewers; one was very positive, but the other negative -- albeit I'm not convinced this person read it so carefully, as s/he makes a logical error in arguing that one proof is 'immediate', and questioned one theorem, but stated it incorrectly (the correct statement answers her/his question). Additionally, there were two 'informal reviews', "without scrutiny of the technical details".
Unfortunately, the result is rather subtle. (I presented it to someone, well-established, recently who at one point said, "It looks like it would be fairly straightforward, but it is actually quite subtle.") It appears that at least one of the 'informal reviewers' (actually, the more positive of the two) didn't get the subtly, suggesting that there was only a specific difference between two other papers, but actually they are extremely different. (I'm not claiming my paper is better than those -- in fact, I submitted it to a less prestigious journal -- just different.)
The main point of rejection by the editor, though, was "originality of problem posing, ideas and methods -- in light of the existing results" (citing the two aforementioned papers). As stated above, the paper is actually really quite different from the ones in the field -- it's not just me that feels this -- but in order to really appreciate this, one needs to have a reasonable understanding of the paper, or at least the other papers.
I feel that perhaps we did not do a good enough job of explaining the subtleties in the introduction. It is one of my first papers, and I did the writing, with comments from the other author. Since I am very familiar with the paper and the subtleties, I perhaps took it for granted that a reviewer would read it fully and carefully.
Explicit Question
Does rewriting the paper somewhat and resubmitting to the same journal sound like a reasonable idea, or should I just chalk it up to "the randomisation of the review process"? The editor has now changed, and the person that handled my paper is not an associate editor.
The positive reviewer gave lots of helpful comments -- the negative reviewer only gave a few, and (other than the one where s/he hadn't read the statement carefully enough) were typographical. I would implement these changes (including the typographical ones), where appropriate. But the major change would be in explaining subtleties in the model better, and comparing with other models, explaining key difference (that are perhaps obvious to me, who has spent 2yrs on this project, but not to someone giving an 'informal review').
Bear in mind that my co-author has already contacted the editor, who said "I have no reason whatsoever to question the professional authority of the associate editor and reviewers", and would not comment on any of our concerns. (I do understand her/his point of view here, to some extent.)
As I said, I originally assume we'd just resubmit elsewhere, but since my co-author thought it worth contacting the editor, who would not consider it, I wonder whether it's worth doing this. I would probably make the changes anyway, and if we got rejected we could just submit elsewhere after...
As you can see from this question, my writing can sometimes be rather long with unnecessary details, and the key details slightly hidden -- I am trying to improve this!
Incidentally, none of the people that looked at it noticed a fairly obvious mistake, where there was supposed to be a 3/2 factor... in the one display of the main theorem!