For academic mathematicians, it's not so hard to avoid predatory journals. We have a sense of the journals that we "typically" publish in, and the journals we "aspire" to publish in, and we stick mostly to these journals. For journals we don't know yet, we can ask colleagues to get their opinion. One (possibly unfortunate) goal of publishing in academia is to build a vita and maintain/advance your career, and it doesn't help as much to publish in a journal that none of your colleagues or supervisors has heard about.
For people who don't have that kind of professional experience, there are several easier questions you can ask. The better answer to each of these is "Yes". On their own, none of these questions indicates that a journal is necessarily reputable or nonreputable. But, if the journal gets a "No" answer to many of them, then I would be very hesitant to submit a paper.
Does the journal publish with no cost to the author? Reputable mathematics journals almost never charge fees for publication. Some very good journals offer an open-access model as an option, but it is almost never the default at the moment.
Is the journal indexed by MathSciNet and/or Zentralblatt MATH? These sites aim to be very comprehensive for mathematics journals. Being indexed is not really a sign of quality, but not being indexed is a red flag.
Is the journal ranked on the Australian Mathematical Society Ranking? Even C-rated journals can be OK, but if a journal is completely omitted I would take that as a reason to be cautious.
Is the journal either published by a well-known publisher, or affiliated with a university or mathematical society? Most reputable math journals fall into these categories, but not all. Some journals run by professional publishers are still not very reputable, of course.
Does the journal have a professional looking website? Grammatical errors or parts of the website that seem to be entirely missing are a cause for concern.
Does the journal have a long history of publication (say, at least 20 years)? Most predatory journals are very young; most math journals are relatively old.
The "American Journal of Pure and Applied Mathematics" has a "No" answer to all of these questions. If they really did have a 10-day turn around between receiving a paper and accepting it, as described in comments, I am even more skeptical of the quality of the journal - that is an almost absurdly fast turn around time for a mathematics journal. I would not pay them anything to publish a paper of mine, nor would I recommend it to anyone else.
Of course, you can publish in a predatory journal, just as you can publish in a for-hire press. But if you are an amateur or "outsider" looking to publish in a math journal, you are likely doing it to get a sort of "seal of approval" on your paper. Journals that mathematicians view as unreputable will not give your paper that kind of recognition among mathematicians, just as degrees from unreputable colleges are unlikely to impress others.
If your goal is just to disseminate your mathematical work, and you don't require peer review, you can often use arXiv.org instead. Depending on the area, you may need to have a professional "sponsor", but the arXiv will keep your paper available for free for the indefinite future in a way that is widely accessible to the public.
