Edit (July 2023):
I just read your question again, a few years later, because someone downvoted my answer ... and now I want to rewrite the whole thing. Instead, I'm going to keep the original as a quote, and put my updated view here.
It's one thing to make one remark that the lecturer might think is witty (but is also quite inappropriate), but it's something else entirely to keep at it, while watching the reaction of the recipient. Completely independent of that, it's also not very smart to keep a student in front of the class after seeing that the student is not able to solve the task in the way they're expected to. Teaching is about learning what you don't yet know, not about filtering out the people who haven't mastered everything yet.
It's also remarkable how easily people may rationalize or even defend such behaviour, particularly if it's something they themselves have to deal with. (...)
To everyone reading this: Please don't be like that. Don't be someone who puts people in such situations, and don't defend the abuse.
To everyone witnessing a scene like this: That's an object lesson in understanding that expertise in one topic does not make someone a better human (actually, it seems to make some people into self-important bullies). The fact that the lecturer was much nicer to the OP afterwards does not make their behaviour any better. If you're only nice to people who meet/exceed your expectations, you're not a nice person.
To everyone subjected to such treatment: Whatever your reaction in the moment, we all react as we do. It's an attack, and that can quickly shut down important parts of your brain. So, the best that most people can muster in the moment is to shut up, maybe even stop listening entirely for a few seconds, let the moment pass and then consider what to do next.
If you do manage to hang on to your wits: Try to leave the "victim" spot that the lecturer assigned you to.
There are some ways to do that, but they're hard to get right spontaneously. Some of those are: Asking an unexpected (ideally well-informed) question back (you're the student, and it's their job to explain!), making a return joke ("... I guess lecturing is also not for everyone, either". Haha indeed!), or maybe just stating that that was enough fun for today and sitting back down -- really, anything that demonstrates you're not just going to stand there and receive whatever they decide to hand you. This sort of thing demonstrates confidence, and most potential abusers will naturally stop at that point (or explode with rage, at which point you can sit back and enjoy the show).
A less good alternative is to directly fight back and accuse the person. That escalates the situation further, and you might not want that with a lecturer you still need to live with for half a year or more. It also carries the risk of looking like you're trying to cover incompetence with aggression -- particularly if you lose your temper halfway through your response. However: In some situations punching back is still better than taking a beating.
What to do afterwards?
I think the other answers already cover this to a large extent. Depending on how you feel about the incident, it's often good to talk to someone that you trust and/or who was there, to assure yourself that you're not making stuff up, and to find out what the procedures are for complaints about teaching staff. And then you can form an opinion on whether to file a complaint, confront the person directly, indirectly (i.e. by talking to others in the offender's environment or through your own behaviour), or to let it slide.
Letting things slide is often the "easiest" way (possibly hard on yourself but it avoids having to get up and confront anyone), but it's also the way that allows the offender to rationalize that their behaviour was totally fine because nobody complained. I hate to think that there are a lot of abusive people who continue exactly because most of the recipients of their abuse find it easiest to walk away, grin and bear it, or otherwise not force the issue. Some places have anonymous complaint procedures where the offender is not told who is complaining, in order to make that path easier, but there are of course some limitations to those... so it's not a very straightforward decision.
If, as you say, everyone was laughing, then I don't think he meant to be mean-spirited. Every time I saw someone be humiliated for serious mistakes, there was indeed dead silence, and everyone felt bad for the victim (although mostly no-one says a word...). If people are laughing, it means (in my own sphere!) that the lecturer made a joke.
8(!) years later: That's rubbish. One "joke" may be a clumsy accident, but keeping at it as you describe is not. The guy had a an attitude problem.
Now, does that mean you just got it wrong? No!
Not at all.
I know a quite a few people (especially in academia) who are just socially inept enough to think something was a good joke when they actually just really hurt somebody.
I can imagine the guy thought he was just poking you a bit. And maybe he has a somewhat-secret, somewhat-ironic dislike of non-mathematicians (who doesn't like to poke fun at "the others"?). All in good humour, of course... (he thinks).
I'd guess he probably really did not think he was doing anything wrong (obviously because nobody does something wrong on purpose), but there was a good deal of "that stupid student must be taught a lesson" in there, and that attitude can just fuck right off. Pointing out a deficiency can be useful, but embarrassing people is not. The correct behaviour in such a situation (student is at the board, trying to solve a task) is to not call students to the front unless they indicate that they can do it and to have a backup plan that avoids embarrassment if they fail.
Generally, if a student doesn't get it, that's the lecturer's fault for not having explained it in a way that works for the student.
In case that wasn't clear: I think it's his fault for not realizing what he did, not yours for feeling humiliated. Last time I asked students a question during a lecture, most of them had no idea, and I think this is the hard reality most lecturers live with. So if you were able to answer correctly, the guy has nothing on you, even if it took you a while.
What you should do? I think you should talk to some of the other students to get their view. Maybe you were easily humiliated because you're more used to success than others? Maybe they thought it was in good spirits (though maybe in bad taste)? Ask them how difficult they thought the problem actually was. Or what they thought the lecturer actually meant to say. What they say will not be "the objective truth", but maybe it'll help you understand better what the episode meant. If it turns out that actually, yes, he thought you were under-performing and wanted to make fun of you for it, by all means go ahead and report it (as others have suggested). But I'd suggest that before you do that, you should not just collect evidence from other witnesses but also try to get more of a perspective of what was going on, because that will help you understand it in your own mind -- which you have to do anyway.
Ugh. I mean: by all means go and get more perspective, and of course understanding the other is valuable -- but bear in mind that "understanding" is not the same as "excusing". It's more a tool to find a lever that you can pull. Something that gives you an idea on how to either push back or improve the situation. It's also really useful to get out of the "offender <--> victim" relation because it can give you confidence, agency, and an element of control that the offender may not have. The better you understand where the other's behaviour is coming from, the easier it becomes to stand your ground, talk back, or otherwise improve matters for yourself and hopefully others.
I'm not happy with this choice at all, because courses which I wanted are not opened this year, so I'm stuck with bunch of courses I didn't want to study at all.
Forget the lecturer, this is the problem you should probably solve first.