Reviewing a paper takes me 4 hours upto 2 days. I only accept review invitations when I have time to read those papers. I have not yet declined any serious invitation. I try to submit my (naturally constructive) review within two weeks after being invited by an editor, no matter the time being given.
In contrast, I had to wait seven months once to receive a rejection. Other peer-reviewed journals accepted within two - three months, including revision. Therefore, I have the same question as you have. I think it strongly depends on the journal and the reviewers it relies on.
I have a theory but it is just a theory. Some journals have procedures about how often a reviewer can be approached. What if you just have to wait until a reviewer in your field becomes available?
Further I notice that reviewers easily exploit the maximum time given by an editor. An editor probably waits for all reviews to return. Then there are reviewers that need reminders or just never respond after having accepted an invitation, forcing an editor to start all over again. I even consider the possibility that a small number of reviewers intentionally slows down the process for competitive reasons.
I feel the whole review and decision process would benefit from more transparency and behavioral professionalism. Being busy is no excuse. Nobody is enforced to accept an invitation. We all, without exception, have 24 hours in a day.
You could consider sharing your review experience at SciRev.org