I've had to refuse several reviews in the past, and it was either because I didn't have the time to write a proper review, or because the topic of the paper was beyond my scope of expertise.
I also found myself in the position where I should have refused a review, because there was a conflict-of-interest that, at the time, I didn't perceive. I didn't know the authors, but I had submitted a paper on a similar topic to the same conference, and after a while, I realised that unconsciously, I was thinking that if I reject the paper (I was hesitating between reject and accept), it increases the chances of my paper to be accepted. I still managed to review the paper objectively (and for the record, I accepted the paper, and mine was accepted too). Clearly, this line of thoughts was not correct, and I'm not particularly proud of it, but once you got it, it's hard to understand how objective you will be able to be, since you could actually over compensate and accept the paper just because you're afraid of the bias.
To answer precisely your question, the situation where you don't want to review is different than the one where you can't. If you can't (because you don't understand, you don't have time, etc), then just say why, and it's fine. If you don't want to, then it's just a matter of how much you want to please the person who asked you to review the paper, compared with how much time it will take you.