The answer should certainly depend on many different elements, each of which should be judged with respect to your particular situation.
First, the time issue. You should have a look at the AMS survey on journal backlogs, which also indicates the median time to get an answer (for accepted papers only, but we can assume it gives an idea of what happens to all not-rejected-upfront papers). This survey is published regularly in the Notices, and you should be able to find it on line. Then, ask yourself if waiting that time is worth a likely negative answer. Some journals have a pre-refereeing procedure, and reject a fair amount of papers very quickly. This mitigates the risk (either you get rejected quickly, or you have a much-higher-than-average probability to get your paper accepted).
Second, the impression you give to editors. I guess that an editor could remember slightly abusive submissions, and that they would not help you future cases with her; however I have no data or evidence to back up this guess.
Third, your own feelings. Receiving bad reviews is not something to underestimate: it could diminish your enthusiasm toward your own work, alter your willingness to pursue in this direction. This is probably only a small parameter, but if you have had a few difficult cases recently, you might want to take a break from rejections.
Fourth, you should ask yourself if the journal you consider to submit to is an appropriate venue if your paper do get published there. I would not fear too much people telling that your paper is clearly below the journal's level, as we often judge papers by journal's name anyway. But it might happen that a more specialized and less prestigious journal could reach the intended audience better than the most prestigious, general journal. This is not a very common situation, though.
Last, but far from least, what you expect to get from getting published in the top journal? Is it so much of a career boost compared to very-good-but-not-top journals?
Note also that some editors have the habit of suggesting another venue to papers that have good but not good enough reviews in a very selective journal. It is not easy to know this beforehand, though.