These days, when people judge researchers (especially hiring/grant committees, tenure boards, etc.) it is common to place much emphasis on impact. Impact is measured in a couple different ways, but all of them share the common idea that more people citing you is better.
If you only ever cite yourself, your citation count is 1. That is only one step up from the worst possible count, 0. So the debate of you citing your papers vs. the one other guy in the world who cares, is kind of dwarfed by the elephant in the room - namely that you're at the bottom. What difference does it make if you finish 99th or 100th, when only top 3 get a medal?
But given the obvious perverse incentive, usually self citations are not even counted as a full citation. Obviously you citing yourself is less interesting than a stranger citing you. In most modern citation metrics, self-citations are either scaled back with a factor, or disregarded entirely. When humans calculate an "informal impact factor" by Googling your papers, their brain will naturally apply such a factor as well. So it's like not only did you finish 99th, but you're tied for it as well.
In sum, I don't think anyone will care about whether you were cited by yourself or not, they will have already stopped caring when there is only 1 person citing you.
Incidentally, if you are the only one talking about your work, then you truly are irrelevant in the literal sense - nobody is finding you relevant to what they do. But I feel like that's putting it a bit harshly. Just because nobody has found your research relevant yet, doesn't necessarily imply that your research is inherently worthless and cannot possibly have bearing on anything else. It's just a heuristic after all. But that said, the academic community appears to be invested heavily in citation metrics for the time being, and as a result, having only self-citations will not impress many strangers.
There are some exceptional cases where I think a situation like you describe might be understandable:
- A very junior researcher for whom just publishing a paper at all is an accomplishment
- The self-cited paper is in a very prestigious journal (although such journals tend to try very hard to eliminate papers that aren't likely to get cited)
- The paper itself has extraordinarily impressive content, despite the few citations
- The person evaluating you is himself in such a "thin" field, himself has few cites that are non-self, and himself appreciates the difficulty of having a high citation count in your field
- The paper is very recent (<a few months) so it is plausible people are planning to cite it, but are just taking a while
But in my opinion, the effective solution here is not to split hairs, but instead to either broaden your work to have more relevance to people outside your narrow field, or start thinking about moving on to an allied field with broader appeal. That, or just accept that the world is not yet ready for your ideas, I suppose... Such is life.