I want to know whether I can visit lectures in German universities as a listener (without being enrolled at the uni). For example, I wish to study in Heidelberg University but want to talk to the people there, but first and foremost go to some lectures in physics or mathematics.
I am not familiar with Heidelberg in specific, but from this page (in German), it seems that you have to pay a fee to attend lectures as a guest. For instance, if you want to attend a single course that runs for 4 hours a week for one semester (4 SWS, actually 3 hours because German "lecture hours" last 45 minutes) you'll need to pay 51€.
If you are an EU citizen, you can also formally enrol as a student for 171.80€/semester. If you plan to live in Heidelberg, it might be worth enrolling just to qualify for a student season pass for public transport.
You already got two answers and several comments, but I think that a somewhat more detailed answer might be helpful.
For this answer I'm specifically referring to mathematics, since this is one of the two fields mentioned in the question and this is the field I have most experience in.
I am explicitly referring to the situation without any Covid specific restrictions in place. University policies with respect to Covid tend to be somewhat diverse and can change very quickly.
Reasons to attend a mathematics lecture without being enrolled as a student.
From your question the reasons for your plan don't become completely clear, and I can think of at least two different types of motivation to attend a maths lecture without enrolling as a student:
(1) You are actually interested in enrolling as a student, but before choosing a subject you would like to get a feeling for the "taste" of the subject by attending a few lectures on just one or two occassions.
(2) You do not intend to enroll as a student (maybe because you have other obligations, or for any other reason), but just want to learn something about the subject.
If your motivation is (1), then the most common course of action is to just attend a typical lecture for first year students (in mathematics this would typically be "Analysis" or "Linear Algebra") on one or two days. If it's not the very first lecture in the semester, you probably won't be able to understand much (because you then missed the contents of the previous lectures), but it will most likely give you a certain impression of how a math lecture at a German university works.
Please note that in such a case, it is certainly not necessary to register as a "Gasthörer" and pay money for it. The polite thing to do would probably be to talk to the lecturer just a few minutes before the lecture begins and ask if it's ok to attend today, and maybe also on another day. With very high probability the answer will be "no big deal, just take a seat" - simply because faculty have a very serious interest in prospective students getting a good impression of a math program before joining in the next semester.
It should also be noticed that such lectures in Analysis and Linear Algebra tend to be quite large (though there might be some exceptions). I have never heard of any first year math course at a German university where it is checked who precisely is in the lecture hall, so if you decide not to ask the lecturer whether you're allowed to attend and if the audience is sufficiently large, chances are that the lecturer won't even notice that you are there - so again, no big deal.
This is a somewhat different situation, and I would indeed suggest to you to register as a "Gasthörer" (and pay the required fee) if you want to attend a lecture for the whole semester (although, again, at many universities it's not particularly likely that anybody will even notice if you attend a first semester lecture in mathematics without registering).
But anyway, there are a number of things that I would like to point out since you might not be aware of them (simply because many mathematics students only learn about them after enrolling, and you are not enrolled):
If you do not have significant preliminary knowledge about mathematics from another university, the only courses that you can reasonably attend (unless you are much - really much, much, much - smarter than most other people) are the courses that are designed for students in their first semester (these are often "Analysis 1" and "Linear Algebra 1", although there might be some university specific deviations from this). Often there will also be specific math lectures designed for first year students of physics, or engineering, or computer science; attending those might also be an option, depending on your specific interests. But in any case, there is absolutely no use in attending a more advanced math course without having first learned the preliminary stuff - you wouldn't be able to understand anything.
A math course for first year students does not only consist of the lecture, but has a number of further components which are just as important: (a) There are weekly exercise sheets which you are supposed to solve and which typically get marked; (b) There are often (though not always) exercise courses where these exercises (and their solutions) are discussed; (c) there might also be further tutorials held by more advanced students, where you can ask questions or try further exercises.
While in most cases nobody will really care whether you attend a first year lecture (simply because these lectures tend to be large and your presence will not cause any costs to anybody), things might be different especially for the marking of the exercise sheets and the tutorials - because both require valuable resources at an individual level. I am not even sure whether it is possible to subscribe for these activities as a "Gasthörer" (this might actually depend on the universitiy and on the people responsible for the course).
In this context it is very important to note that doing the weekly exercises is one of the most important parts of attending such a math course. Simply attending the lecture and thinking about its contents is very unlikely to teach you much in the long run - on the contrary, experience shows that most people can only reasonably learn math if they do it theirselves, and this is the role of these exercises. So just attending a math lecture without participating in the exercises (or without getting feedback on your solutions) is unlikely to be a fruitful activity.
On a similar note it is also important to have reasonable expectations regarding the necessary time investement to get something out of such a course. A typical Linear Algebra 1 or Analysis 1 lecture will consist of two 90 minutes slots per week. But in addition to that the time for the exercise course and/or tutorial will take up one or two more 90 minutes slots. The most time consuming part, though, is typically your work on the exercise sheets. I tell my first year students that 6-8 hours work per week in order to solve the weekly exercise sheets is a completely reasonable time frame (though some are a bit slower and some are a bit faster, of course). So all of this easily adds up to much more than 10 hours per week - and this is just for one course.
So it's important not to confuse "I'll attend some math lectures" with "I'll put reasonable effort into actually understanding their content."
One small remark about physics.
I have never taught a physics course, but I took physics as a minor when I was a student - and according to my experience from back then, the effort for a typical first semester course in physics is just as high (or higher) as for a typical first semester maths course. In addition, it doesn't make much sense to attend such a course without attending, at the same time or before that, a first semester maths course for physics students. The math that you've learned in school probably won't suffice for more than a week or two in such a physics lecture (depending on the lecturer, it might not even suffice for the first 30 minutes).
If you want to attend a lecture at a German unversity, just go there. Most of the cases, there's no attendance check or anything of that kind. If you want to be nice and/or it is a very small class, talk to the lecturer either beforehand or before/after the first lecture to say that you are not a student but interested in attending. Generally, this will be just fine.
(Of course, as long as you are not enrolled as a student there will be no way to earn credits or the like.)