In papers themselves, there is absolutely no reason why you should ever copy any previously published text (including yours) without explicit acknowledgment, along the lines of "The background material in this section is a nearly verbatim adaptation of Section 3.2 of X". Assuming you make it perfectly clear what you have copied and from where (not hiding this information in a note later in your paper, for example) and you have permission from the copyright holder, then this is ethical, while it's not ethical without these conditions.
Of course talk abstracts are not quite the same. Let's assume we're talking about relatively ephemeral abstracts. I.e., they might appear on the web or in the conference program, but they aren't carefully archived, citable contributions to the research literature. (This distinguishes them from "extended abstracts" in CS conferences, which are actually short research papers, and there may be other intermediate cases.) These sorts of abstracts generally don't list any references within the abstract, and they aren't considered published or treated nearly as formally as published material.
In mathematics, I doubt anyone would get upset about recycling a paper abstract for this kind of talk abstract. Customs vary greatly between fields or sorts of abstracts, so you should seek advice from colleagues in your area, since "someone on the internet said it was OK" is not a compelling argument.
If you are worried about self-plagiarism, you can simply append something like "(adapted from the abstract of paper citation)" at the end of your talk abstract. However, that might stand out in its formality.
It's probably a good idea in any case to rewrite the abstract at least a little, since a talk abstract has different goals from a paper abstract.