I expect the answer depends on your circumstances. Here are some cases that might apply. I can't specify precise cut-offs between them, but I doubt it matters too much if you're near the dividing line. (Keep in mind also that you may have specific rules to keep in mind. For example, some particularly bureaucratic universities have guidelines for exactly what you must or must not include in your CV.)
For most junior academics, I think it's helpful to list all of them, and it's almost certainly not harmful, assuming they are at least respectable, mainstream academic journals, even if they aren't prestigious. Each time you've been asked to referee a paper is a vote of confidence from an editor, and it's worth demonstrating that you are starting to become widely known. Listing seven journals, some really not prestigious, probably looks better than listing just the two or three more prestigious ones.
For people coming up for tenure at research universities, and even more so for mid-career academics, I'd recommend focusing primarily on prestigious journals. At this stage the goal isn't just to show that people have heard of you, but rather that editors of excellent journals trust your opinion about important papers. (But be sure to add some short comment to make it clear that this is a selected list.)
Once you are sufficiently senior and widely known, I don't think there's any need to bother listing journals you've refereed for, except perhaps very briefly to indicate that you still do your duty.