Horror stories: You also asked the following question:
Are there any horror stories of waiting several months after a reviewer agrees to look at your paper, only to get a rejection letter?
First, time between submission and acceptance might look something like this:
- administrative screen (1 week)
- editorial screen with potential for desk reject (1 to 4 weeks)
- potential reviewers are contacted
- time for reviewers to complete assignment vary and review process is determined by slowest reviewer (can be anything, but often 1 to 4 months)
- editor/action editor has to review external reviewers (anything from days to months) and reach a first-round decision.
As an author, you don't usually find out how long the components of the review process take. Your question implies that there is one person "a reviewer" deciding to review and then rejecting. But rather, these are usually different people, and there are multiple reviewers. At most journals, the external reviewers inform and make recommendations to an editor / action editor. They do not make the decision.
So in general, there are different norms about how long a review takes. But in my field of psychology (1 month is amazing, 2 months is nice, 3 months is par, 4 months is okay, 5-6 months is slow, beyond 6 months would be concerning). Other fields and journals have different norms. The point is I would not see waiting 4 to 6 months as a horror story. It's a bit slow. But It's just how long things sometimes take, and something that might factor into where I send my work in the future. And the length of time has almost nothing to do with the outcome. So in general, rejection is unpleasant and slow review processes are also unpleasant, but they're separate issues.
If the first-round review process is taking an amount of time well beyond the norms of your discipline, then you may wish to contact the journal to see what is the hold-up. Based on the response you get, you could weigh up the pros and cons of withdrawing your manuscript. That said, given the energy that may have already been invested in reviewing manuscript, I'd see withdrawing a manuscript as very much a last resort.