Yes. It is much better to decline as soon as you realise you're not up to the review, than to write a useless review.
Normally, you first get an abstract based on which you accept or decline. If you accept, the editor may be annoyed if you decline based on seeing the manuscript, but sometimes it's inevitable; only so much can be judged from the abstract. If you do decline, however, you should let the editor know as soon as possible.
I recently read a publicly reviewed paper where both reviews clearly did not really understand the topic, but still handed in a review. To me it was quite clear that the paper needed a statistician, which neither of the reviewers were. One reviewer didn't get much further than pointing out that the introduction was too long, with not a single comment on 10 pages of detailed description of advanced statistical methods. When reading the reviews, I found it a bit embarrassing that the peer review was so clearly insufficient, and the paper virtually got published without proper peer review.
I know someone who accepted his very first paper review, only to discover when he received the manuscript that it had 130 pages. He returned it to the editor.