On rather general grounds, I would recommend against viewing college as a race to be completed as quickly as possible. In all likelihood you really will not have another time in your life where you are expected and encouraged to explore the intellectual landscape in a really broad way (of course you're encouraged to explore it much more narrowly as a Ph. D. student), and if you rush through it you may look back on this with regret later on. That said, since I assume from the reference to AP tests you are in the US, given the cost of college these days I can understand viewing the expense of an extra year as an unaffordable luxury.
More practically, as you already seem to recognize to some extent, finishing quickly will significantly reduce your chances of getting into a strong graduate program. I am a faculty member involved in graduate admissions at a decidedly non-elite mathematics department, and even here we expect our applicants to have done appreciably more than the minimal amount necessary to get a degree--e.g. they should have taken multiple graduate courses (if available at their institution) and/or have done some substantive research or seriously advanced independent study. Especially since you haven't even figured out what specifically you want to do yet, I'm pretty skeptical that you will be able to build the kind of record that's going to impress graduate admissions committees if you spend a less-than-normal amount of time as an undergrad.
Of course I have no way of knowing your talent level, but whatever it may be, it seems almost tautologically true that the people who will be competing with you for admission to the kind of graduate program that you should be going to will be about as good as you, but with at least 3.5 years of undergrad experience under their belt when they apply (in the fall before they graduate). If you only have 2.5 years' experience when you apply then you have to expect that you'll lose out to those people, and end up somewhere not as good as where you should.
This suggests a sort of compromise, which happens to be what I did as an undergraduate: plan to finish after 3.5 years. Since one applies to grad school in the fall anyway, the missing senior spring semester doesn't affect the strength of your graduate applications, but you do save a semester's worth of money. (As alluded to earlier, if money is no object then your goal should be to be an undergraduate for as long as you can.) I got into an excellent graduate program doing this, and while it would have been possible for me to graduate in 3 years instead I'm sure that this would have led to me going to a much weaker graduate program, which in turn would have done permanent harm to my career.
But to answer the question in the title, unsuccessfully applying one year would not prejudice your application in future years, if your record improves in the interim. I would however recommend waiting to apply until your record is where you want it to be instead of applying early and going to a suboptimal (for you) graduate program, which would be the likely outcome.
