Now, consider the field of Mathematics. Say a mathematician has obtained a nice result, but for whatever reason can't publish a paper on it momentarily.
This bears further explanation. Publication in mathematics does not happen so quickly, but if you have a "nice result," you should be able to submit it for publication.
Can they do anything to hold some kind of priority over anyone else that comes up with it and publishes it before them?
If two different mathematicians or groups of mathematicians obtain results at approximately the same time, they should both be able to publish. The gap between submission and acceptance of most math papers is typically 3-18 months (with anything less than six being quite speedy), and the gap between acceptance and publication is another 6-24 months or so. So if a rival paper gets published before you submitted yours, there are three possibilities: (i) the other paper was in fact first; (ii) you delayed submission for a significant length of time, or (iii) you experienced substantial delays in the publication process (including rejections of your paper).
For most mathematicians, the best way to hedge against getting scooped is to (i) write up your results in a timely manner, and (ii) post the paper to the arxiv as soon as (or even slightly before) the initial submission. In the mathematical community, an arxiv preprint will have the same effect towards establishing "priority" as a formal publication. It also has the chance to alert a rival group to the overlap of effort before the works have been published. When this happens, the two groups should examine each others papers and decide how best to proceed. In some cases this means that each paper cites the other and they both get published. In some cases the work is combined into one joint paper. And yes, sometimes one of the groups decides that what they have done is "majorized" by the other group and decides not to try to publish their work.
From the way your question is framed, I have the suspicion that you are not very familiar with mathematical publication culture and may be thinking about things in somehow the wrong way, but I can't put my finger on what it is. But in particular you write
If so, what happens in practice if someone else does indeed publish it before them?
and
That is, the practical result of this "protection" should be the ability to make the journal take your paper (which you should quickly write) instead of the one by the researcher who published before.
With regard to the first quoted passage: with very high probability, the prior publication will stand without formal comment and you will be left to try to explain/establish why your paper should still be published. I have seen "statements of priority" published about math papers, but probably fewer than ten in my entire life.
With regard to the second quoted passage: first of all, no author has "the ability to make the journal take your paper" in any circumstance whatsoever. I'm not sure I completely understand the situation you contemplate: if the journal has already published the other paper, then it is too late for them to take your paper "instead." If you're saying that they should retract the published paper and publish yours instead...well, I have never seen that happen. The only circumstance in which I think that could even be plausible is if the authors of the other paper literally stole your work and passed it off as their own.
Also...what do you mean "which you should quickly write"? If you see a published paper on work that you've completed but haven't even written up yet, then as mentioned above that other paper was probably written 1.5-3 years before yours. So how could you have priority in this situation?
