Good question. At the risk of telling you mostly what you already know, here is my perspective:
After 12-15 years and 40-50 submitted mathematics papers (not counting multiple submissions), I have learned that one should virtually never appeal to the editor with the expectation of it changing the outcome.
In my experience "what's done is done" is the philosophy taken by editors and referees more than 90% of the time. When I have brought up flaws in the refereeing process to an editor after rejection, the most common reaction I get is indifference / mild agreement but with the view that what's done is done. Only rarely have I had an editor willing to substantively engage with issues in the referee process. In one of my earliest experiences, this resulted in an invitation to resubmit...and N months later the paper was rejected again.
At this point, when I remonstrate with the editor, I make it explicitly clear that I am not appealing the decision but just speaking up on the process as a matter of principle. In fact this kind of "principled reply" has twice resulted in an invitation to resubmit. In one case I declined that invitation (cf. the above story); the paper was published elsewhere after one more rejection. The second (rather recent) time this followed a desk rejection by the editor. I just couldn't fathom why this paper was desk rejected by this journal, and since I had had bad luck with the same paper previously, I wrote to the editor basically asking "I just don't get it; could you please explain?" Amazingly, the editor wrote back immediately saying he had completely misunderstood the paper. He invited us to resubmit the paper to the journal. The editor in chief of the journal got involved as well (see below!) in a positive way, and the paper was accepted soon after with minimal fuss. This experience made me realize that submitting a paper was like watching a baseball game: no matter how many times you've done so previously, there's a reasonable chance that you'll see something completely new!
Speaking of resubmitting following a rejection: in our field, I think that is unequivocally a terrible idea. The part of the above story that I didn't tell yet was: after the handling editor apologized profusely, he asked us to resubmit the paper via the journal's automated submission system. My coauthor and I almost did that....but we were stopped short by the dire warnings about resubmitting the same paper after rejection causing us to get automatically banned from the system! We relayed this to the handling editor, whose response was essentially "OMG" and then he contacted the editor in chief. The editor in chief looked into this and eventually asked us if we would be willing to change the title in order to resubmit. We did so, decided not to change it back at the end, and this published paper has a slightly different title than the version on the arxiv or my homepage.
Even if that doesn't happen, I think that resubmitting what is even essentially the same paper to the same journal after rejection is the swiftest way to wear out your welcome with that portion of the community. Imagine how many iterations of proofs of the Riemann hypothesis / twin primes conjecture / ... the Annals of Mathematics must get. At some point they just have to ignore them.
[However, I now remember that I got added as a coauthor to a paper after it had been rejected by a previous journal in combinatorics. My feeling was that the new version of the paper was only by historical accident a new version of the paper; in all other ways it was a different paper. So we resubmitted to that journal, and it was accepted. Then I submitted a solo followup paper to the same journal the following year. They told me that they didn't want to publish two papers in the same subject at approximately the same time (long backlog)...and they offered me the option of merging my new solo paper with the previously accepted triply authored paper!! Baseball indeed.]
Your question is about "top journals." I am starting to suspect that "top journal" means something different in mathematics than in some other academic fields, e.g. after reading questions like "I have five papers accepted in top journals but don't think I can complete my PhD." I mean, I am a successful research mathematician but have never even submitted a paper to any of the top three math journals. But I find it unlikely that e.g. the Annals is more forgiving about resubmissions / more amenable to editorial remonstrations than other math journals. I would expect it to be less so...
