(Making @Kimball's points perhaps even more forcefully...) If you are interested in experimental physics, I have no real advice, but if you are interested in theoretical physics, I will assume for the sake of discussion that the situation resembles that of mathematics. In that case, in fact the question as posed involves implicit hypotheses that (I think) far over-shadow the literal question asked. That is, to get to the best grad program one can, letters of recommendation (from people in the field, who've had contact with you at the highest level you can manage) are critical. So making an excellent impression in graduate-level courses would be a great thing, apart from "taking the class" per se.
For that matter, witnessing how (presumably) seasoned experts talk about more-sophisticated (grad-level rather than undergrad, anyway) material ought to be educational and inspirational in itself, beyond the literal content (which should be accessible in written sources).
For that matter, I think it misses the mark to think of a "program" as essentially following a schedule of classes and perhaps only doing what is commanded by the coursework, homework, exams. It is permitted to read other books, papers, and to think about math (or physics...) as much as one wants. A reason to "take classes" is for easy documentation (and meeting faculty). Private study is hard to document.
Finally, the psychological element of familiarity/comfort matters hugely, in my observation. That is, the longer one has been aware of a thing, the more comfortable one is (even without "mastery"), and often the comfort/discomfort issue is a large cognitive load feature. New/scary things are harder to think about. So looking around and being exposed to things as far in advance as possible is a very good thing.
Finally, at least for "theoretical" subjects such as parts of mathematics and physics, the popular attempted distinction between "study" and "research" is crazily misleading, in my opinion. True, if "study" means some stultifying on-command busywork, that's not so good, but "research" might also mean blundering about in a fog of ignorance, so that could be bad too. The good versions of both, that is, spending one's time thinking about math/whatever, are fundamentally indistinguishable from each other.