First of all, as @ff524 points out there is a world of difference between a master's program and a PhD program. If you don't understand the difference, then you know too little about graduate school to be thinking about it any meaningful way.
In the US a math master's degree takes about 1-2 years and is not a prerequisite for a math PhD program. (In fact, although some students can use a strong performance in a master's degree to "launder" their academic record and gain admittance to a solid PhD program, in most cases enrolling in a master's program signals a much smaller commitment than enrolling in a PhD program.) In Canada it seems to take a bit longer: 2-3 years, and up until recently it was viewed as a prerequisite to getting a PhD. (Things seem to be changing a bit, although still probably the majority of Canadian math PhD students have done separate master's degrees, which adds to the total time to get a PhD). Getting a PhD in either Canada or the US takes 4-7 years; I believe 5 years is the most frequent number though the mean is higher. Moreover a PhD is the required certification for a future academic career in mathematics (it used to be that a master's degree let you teach at 2 year colleges, but that is rapidly ceasing to be the case in the current job market). The purpose of a master's degree in mathematics is much more fluid...or it may serve no purpose at all. You should know that the lowest common denominator for master's degrees in math is indeed very low: it is well known to US graduate programs that an undergraduate degree at a top institution with good grades is usually better than a master's degree at a mediocre institution.
It is fairly easy to "weather through" a master's degree in mathematics, even if your level of interest wanes in the middle (because there need not be much middle, in part). If you have some specific career goal for which a master's degree in mathematics would be advantageous -- e.g. high school teaching -- then not being unilaterally devoted to mathematics is not a serious obstruction. By contrast, if you are not interested in mathematics almost to the exclusion of other academic fields, then a PhD program in mathematics is probably not for you. (Certain branches of applied mathematics are exceptions to this, but the vast majority of students who do a PhD in a "mathematics department" are not that applied.)
Right now, I have the freedom to take other, non-maths courses (I'm minoring in film, for example). Will this freedom continue in grad school?
Technically yes -- graduate students are allowed to enroll for courses in any department just like undergraduates. In practice: the freedom will be severely restricted. If as a graduate student you take a course outside of your department then you should expect to be asked to explain its relevance to your mathematical course of study. Thus it is relatively common to take language courses and courses in related fields (cs, physics, statistics...). You should probably not be taking "elective" courses -- e.g. if you took a course in film, it would have to be strongly skewed to the technical aspects of image production / reconstruction / whatever or you will probably find yourself having to defend doing something that interferes with your research.
Moreover, theoretical mathematics (and some branches of applied mathematics) is one of the least interdisciplinary of academic fields. If you want to work on logic, then you may well find an expert advisor or co-advisor in the philosophy department -- but nevertheless you should expect your work to be at least 90% mathematics. There must be a few math PhD students who take more than a small handful of philosophy courses...but I have never met any.
All in all: you should not seriously consider a PhD program in mathematics unless (i) you are single-mindedly, passionately devoted to mathematics or (ii) you have a very specific career goal for which a PhD in mathematics is beneficial, and your plan has been vetted as reasonable by multiple trusted mentors. If you like but don't love mathematics as you are completing your undergraduate degree, plan on continuing to read, learn and perhaps do mathematics after you graduate, at your own pace...as a hobby.