How can I tell if I am going overboard with this? I have never heard any negative comments by anyone -- including by journal editors, whom I asked for feedback on this matter after sending my reports.
I would consider the following indicators of being too picky:
You make extensive remarks on language that the copy editor can address, i.e., subject knowledge is not required to spot and correct them. Exceptions are persistent mistakes such as a complete lack of articles or some examples to illustrate that the quality of English of the paper is not tolerable.
You persist on opinion-based matters (other than the relevance of the work), such as notational paradigms, structure of the manuscript, or level of verbosity. By opinion-based I mean that somebody else might consider your suggestion detrimental. By persist, I mean that you keep picking on such an aspect, even though the authors clearly have an opposing preference (usually expressed in a reply to the reviewers). While you should mention most such aspects the first time you are seeing the respective material, the review-process should not become a back and forth about such this.
You effectively end up doing the authors’ job and write the manuscript for them.
Apart from this, I do not think that a review can be too detailed. At the end of the day, you are giving recommendations to the editor and authors and it is their job to responsibly make use of them. However, it may be wise to take a few precautions to prevent irresponsible use:
I have never heard any negative comments by anyone -- including by journal editors, whom I asked for feedback on this matter after sending my reports.
I would not read too much into that fact. In most fields if not all fields, reviewers have become a scarce commodity and editors will avoid disgruntling them at almost any cost. If you want to make sure that your recommendations and review are not misinterpreted in terms of severity, or the categories offered for the overall recommendation are not nuanced enough, write a small note to the editor that elaborates your overall judgement and in particular how severe you consider the flaws you commented on. This also avoids that an editor makes a false blind decision due to not reading your entire review.
It may also make sense to consider that the authors may be overly obedient to your recommendations. If some of your suggestions are just this, make this very clear. For example, if you think that the work presented in the manuscript may be relevant for some application but are not sure about this (which is fine, since it’s the authors’ job to find out), make it very clear that you do not think that the authors should write this but just that they should consider this.
On another occasion I believe I submitted six "revise and resubmit" reports for the same paper, before finally recommending acceptance.
It’s hard to judge this without knowing why this happened. For example:
In the first round you requested some additional proof (or other substantial addition of material) because the paper was incomplete. The authors added this, but the new material has a gap of roughly the same severity as the original gap. This processes then repeated until all gaps were filled. Assuming correct assessments from your side, this is probably not your fault but the authors’, since they failed to properly write a paper or to do rigorous research.
If the six revisions were only about how to best phrase some sentence, you have been overdoing it.