To maximise discoverability of my manuscript, should I avoid repeating words that occur in the abstract or title in the keywords? I have a feeling that common academic search engines don't really differentiate between the title/abstract/keywords. So should I "waste" some of my precious free text keywords for central concepts that are already mentioned a few times throughout the abstract?
I see no reason that you would eschew a keyword simply because it already appears in the title or abstract. The purpose of keywords is to highlight the topics covered by the paper, for assistance in categorisation and searches. Usually there will be a cross-over between the best keywords and the title/abstract, simply because the latter are likely to mention core topics of the paper.
In regard to your consideration that you might "waste" a keyword that is already in the title or abstract, I think that is unlikely to make any non-negligible difference in the discoverability of your manuscript. Most journals allow a variable number of keywords anyway (though there is sometimes an upper limit) and some search engines might be influenced by the number of times the word appears, so repetition may make it more discoverable, not less. (Note that sometimes you are free to choose your own keywords and sometimes you are restricted to a list of available words, so that might also affect things here.)
Working as a patent examiner for nearly a year now, there probably have been multiple instances where I found out about an important patent document from the appearance of a single keyword, usually when someone was noting what terminology they were using and were listing some synonyms. I think it's good to think about which keywords are missing from the title and abstract. If there's no good way to work them into the title or abstract, it seems reasonable to put them in the keywords section. You could also put these keywords in the body somewhere. To elaborate on what I wrote earlier, I've seen synonyms in parenthetical statements in introductory sections like "X (also known as Y)". The latter would make it more clear that a keyword is a synonym and may be preferred for that reason. (To my knowledge, keywords are not given weight beyond the body of the paper in most academic search engines, so it shouldn't make a difference if the keyword appears in the body or keywords.)
Semantic searching does help get around the synonym problem, but for more esoteric topics (like those studied by academics or in the work of a patent examiner), the search engines aren't always aware of the synonyms. It's reasonable to try to help searchers in this.
(It should go without saying that this is not the opinion of the USPTO or US federal government. :-)
Many publishers will actually turn the paper keywords into clickable links on the abstract page on their website. For instance, both SIAM Epubs and IEEEXplore will turn each keyword into a link to start a new search using that keyword as the search term. Words/terms in the title and abstract won't be turned into clickable links, so including the important terms in the keywords as well will benefit the article on the publisher pages.