Here is a suggestion. It has its advantages and disadvantages and I haven't tried to implement it myself, but you asked, so here are my 2 cents.
There should be a prize/incentive for attaining a better place in the tournament compared to the other students.
Apparently you are not willing to give out extra points or anything
else that might influence the grade. Then just make a few nice drawings,
write "The first (second, etc.) place in the calculus tournament" with
the date and your signature over them, laminate, and attach some silly
little prizes (a box of pencils, a figurine, a notebook, etc.) to each.
It'll cost you 40-60 bucks and a few hours of drawing, but it'll work.
Everybody should be able to get the maximum score.
If you have 10 students, prepare 30 tasks and hand them out one by one. For every task the first one to complete it gets 10 points for the task (counted towards the grading score) and 31 minus the number of the task (from 30 to 1) to determine his rank (the first problem is worth 30, the second 29, etc.; the rank is the sum of values of solved problems). Once a student solves three problems, he is not participating any more. You run the tournament until either everyone is out, or nobody can solve the current problem, or the time is out.
There should be some kind of rule which determines whether a student is out the tournament or not.
Every student needs to participate.
Well, with this scheme at some point he'll be the only eligible competitor if everybody else is done, but if he just cannot do the task anyway, it is his problem. I think it is fair enough.
The whole tournament shouldn’t take longer than two hours; I can’t spend too much class time on that.
That depends on how many and how good the students are and on the difficulty of the problems assigned. Also, do you want just answers or explanations? You can replace the factor 3 by 2 in this scheme, but with more than 20 students it becomes somewhat problematic to implement in under 2 hours. What I expect is a fierce competition with quick answers in the beginning, followed by a slower run once the top students are out, followed by a bunch of dropouts staring mindlessly at the problem (or at the ceiling) forever (unless you impose a time limit on each problem). One partial remedy for this unevenness would be to present hard problems first and to decrease the difficulty along the way.