At the end of this answer I will try to give some advise on what you can do to understand a larger part of the talks at conferences and seminars (and this advice will be fairly specific to mathematics). But first, I would like to mention some things that influence how much of the talk you can reasonably be expected to understand. This is both because it is useful to have a good idea of this to see if you are doing enough to understand talks, but also because some of the advise is related to these points.
Factors that determine how much you should understand:
1. The topic of the talk.
This one should seem obvious, but the closer the topic is to your own specialty, the more you should understand.
2. The purpose of the talk.
There are various types of talks, with various purposes. The most common talk is the "by experts for experts" talk, where the purpose is to explain the speakers latest research for others who do work in the same or related areas (most seminars and conference talks are of this type). This type of talk generally sets a fairly high bar for the prerequisites needed to understand the talk, since usually the speaker will have at most an hour (often only 45 minutes), and they want to actually present their own research rather than just give an introduction to a topic a large part of the audience will already be fully familiar with.
In the other end of the spectrum are the talks specifically aimed at students, which generally do not present any new research but gives an introduction to some topic. Foe these, one should be able to understand most of the talk as long as it is not a topic too far from ones own specialty (note that these should not be confused with seminars or conference talks given by PhD students or early postdocs, which can often be even harder to follow since the new results obtained by people early in their career are often of a much more technical nature than for those with more experience).
3. How far you are in your studies.
Again a fairly obvious one, but the earlier you are in your studies (or your academic career), the less you should expect to understand of any give talk.
4. The speaker.
This has already been mentioned by others, but it bears being reiterated: There are some really awful speakers out there. If the talk is given by one of them, even those who are intimately familiar with the topic (or even the results presented) will not understand the talk.
On the other hand, there are also some amazing speakers out there who can make you understand a talk on a topic you really should not have been able to.
Figuring out where on this spectrum the speaker lies can be tough, but often one can tell by trying to get a feel for how much attention those in the room, who ought to understand the talk, are paying. If they seem to lose interest (even though the talk does not seem to be about something elementary), then probably the speaker is not doing a good job.
What can you do?
So, what can you do to understand as much as possible of a talk, relative to what you ought to understand, given the above? These will be some generic pieces of advise on how to get the most of a conference (I will mention single-talk seminars at the end).
Before the conference, make sure you get abstracts for all the talks (if possible). From these, single out a reasonable number of talks that seem the most interesting, or where you know the speaker tends to be really good. Look more closely at these abstracts, and do some reading prior to the talks, but not by necessarily looking at the relevant papers (unless the talk is on a topic very close to your own). Instead, you should look up all those terms in the abstract you are not familiar with (or which you are not familiar with in the context). This will give you a better idea of what the talk is about.
Next, see if you can find some of the main results about the objects mentioned in the abstract (often one gets a better understanding of an object if one known the "rules" it obeys rather than just knowing the definition). It can also be good to find some of the main conjectures about these objects, since this gives an idea of what sort of questions are considered the most interesting (not because it is likely that the talk will present a proof of such a conjecture, though it can happen, as I have experienced myself).
Further, one of the things that often causes a lack of understanding of a talk at a conference is simply being tired from seeing too many talks. To alleviate this as much as possible, I suggest you bring something to the talks that you can entertain yourself with in a non-obvious way once you get to a point in the talk where you have no chance of understanding more.
This might sound a bit rude to the speaker, but this is why I mentioned that it should be non-obvious. It should preferably be such that if the speaker looks at you, you will just seem to be taking notes (unless you are way at the front, you can often have your phone lying in front of you without the speaker being able to see this for example).
Along with the above it should be mentioned that it can also be quite alright to not see every single talk at a conference (though you should probably check with your adviser what the culture is at the specific conference to be sure). Which talks to skip can then be based on which abstracts seem to suggest that you will understand the least (or if you happen to know that some specific speaker always gives terrible talks, you can also skip that).
As a final note on this, I would advise that you try to see as many talks as you can. As long as the speaker is not completely awful, you will actually learn more than you notice as long as you pay attention.
In case of single-talk seminars, most of the above of course does not apply. I would say that for single-talk seminars, you can better afford to spend some more time on getting acquainted with the subject of the talk beforehand, so you should treat it like you would a conference talk that you have deemed to be of high interest to you (and don't bring anything to entertain yourself, but really try to pay attention all the way through).