In my opinion, it really depends on the type of academic job that you want/are able to land. Professional life in (1) a top tier research university, is different than in (2) a medium tier Ph.D. granting department, is different than in (3) a Master's granting department with a more modest research agenda, is different than in (4) a predominantly teaching oriented, four year college.
The teaching load/research expectation continuum certainly varies across the four. On one end of the spectrum, say at (1)-(2), the teaching load will be light (as you describe) but with research expectations in terms of papers in top tier journals and landing external funding that is very high. However, even if the actual course load is less, you will spend a chunk of time working with graduate students in reading courses, research seminars, their thesis, etc. As you move from (2) to (3), the research expectations decrease as teaching loads increase. In (4), you very well may have no requirement to produce original mathematical research in the form of journal articles, but instead be expected to demonstrate "continued scholarly activity" which can take a variety of forms. On the other hand, you may be teaching 4 classes a semester.
Pay, generally---but not always and certainly not uniformly---decreases from (1) to (4). The autonomy of academic life is usually very attractive and serves to counterbalance a salary that is less than what people in some mathematical specialties could garner in industry.
In my opinion, the type of job one shoots for (and will eventually find success/satisfaction in) is a combination of one's passions (research vs. teaching vs both), innate talents (again, in both research and teaching), aspirations, competitiveness, willingness to deal with pressure, and geography, to name a few.
As a nod to pragmatism, one thing to keep in mind is that the vast majority of jobs are in (4) and (3). Jobs in (1) and (2) are highly competitive to land. I have many friends in all four categories who are happy and very few (none?) who are unhappy, although admittedly the latter category probably self-selected out of academia.
Finally, since you are a third year undergrad, you will get a MUCH better sense of how much you really like mathematics in graduate school. During that time all of this should crystallize greatly. You will also get to see the profession much more up close than you do as an undergrad. It is great that you are thinking of these things now; keep your eyes and ears open in the coming years.
This is all just my two cents. Certainly others may have very different opinions, experiences, and perspectives...