Does the department determine it, or does the president determine tenure after the department recommends it? Or does the process go through both ways?
And how does this vary from institution to institution?
Here's how it works at my university. There are several layers of votes; I've seen faculty turned down at almost every level. Once a negative vote is taken at any level, it is essentially impossible for the candidate to recover without an official appeal. (The appeals process is complicated.) On the other hand, I know of negative decisions that were successfully appealed at almost every level.
April: the candidate submits their CV and other supporting materials.
May: The department's promotion committee makes a go/no-go decision about whether to pursue the tenure case. A very small number of negative decisions are made here, but only in truly egregious cases.
May: The department head chooses a subcommittee of three or four faculty from the promotions and tenure committee to shepherd the case through the department.
June: The department solicits recommendation letters, some from people proposed by the candidate, but a majority from people chosen independently by the subcommittee.
August: The subcommittee writes an internal evaluation of the candidate's research/teaching/service accomplishments/potential.
September: The subcommittee presents the complete case (candidate's CV and statements, internal evaluations, and recommendation letters) to the complete departmental committee. The committee votes whether to recommend tenure. Most negative decisions are made here. Having a positive recommendation turned down at a higher level is fairly embarrassing for the department; it indicates that either the department did not do a thorough evaluation or (worse) tried to hide or excuse an obvious gap in the candidate's record. So this level tends to be the most stringent.
October: The department head writes a one-page letter summarizing the case and the department's vote, and forwards the complete package to the dean.
November-December: The college promotions committee does its own evaluation of the candidate and votes whether to recommend tenure. Evaluations are more likely to be based on measures of reputation and impact (reflected in publication record, citation patterns, funding history, and recommendation letters) than on actual quality of research. Faculty in the candidate's department are recused from any discussion. Some negative decisions are made here.
January: The dean writes a one-page letter summarizing the case and the committee's vote, and forwards the complete package to the provost.
February-April: The campus promotions committee does its own evaluation of the candidate (mostly, but not entirely, focusing on whether proper procedures have been followed) and votes whether to recommend tenure. Faculty in the candidate's college are recused from any discussion. A very small number of negative decisions are made here. Negative decisions at this level are more likely to involve problems with teaching (or raw politics) than with research, since measures of research quality and impact vary so wildly across campus.
May: The provost writes a one-page letter summarizing the case and the committee's recommendation and forwards the complete package to the chancellor and university president. The provost also informs the candidate of the committee's recommendation. Officially, this is the first news that the candidate receives about their case.
July: The chancellor and the president rubber-stamp the provost's recommendations, and then pass them up to the board of trustees, who rubber-stamp them again. (Officially, I think any of these three can overrule any recommendation for promotion, but I've never seen it happen, and it would probably cause a faculty revolt.)
Yes, the whole process really takes 15 months.
The process varies from place to place but in most places the technical determination is made by the department and to some extent the college. Higher levels of approval focus mostly on whether proper procedures have been followed.
Having said that, when things go wrong they could go wrong at any level. Usually it's politics that derails cases at the higher levels rather than straight technical merits.