Some open access journals, like Integers for example, don't charge the author any fee for publishing their article. Other open access journals do charge the author a fee and the author has to pay the fee so that their work can be posted online. Are there any trustworthy fee-based open access journals?
I'll try to link to pages which describe the open-access policy of the respective journals.
Both Proceedings of the AMS and Transactions of the AMS allow the author to pay a fee to make their article freely available electronically from the journal. If the author chooses not to pay the fee then the article is not freely available from the journal (but the author may post the final preprint on the arXiv or their personal website).
The (few) Elsevier journals I have looked at have a similar fee for publishing open-access. One example of which is the Journal of Mathematical Analysis and Applications. You might try their Journal Finder, which has a checkbox to only search for journals with open-access options.
While fees are unusual for Math open access journals (and I would initially suspect any journal charging them), there are trustworthy fee-charging journal, e.g. Pi and Sigma (http://journals.cambridge.org/action/displaySpecialPage?pageId=3896).
Journal of Mathematical Physics, which publishes articles by mathematicians, is a serious journal published by AIP that follows the AIP fee-based Open Access policy : http://publishing.aip.org/librarians/open-access-policy
The UK as a whole is moving towards using open access, so most of the UK maths journals, and a proportion of others, now offer open access in some form (although that can mean allowing you to put the final draft on a repository, called 'green open access'). The LMS has created the Transactions, which is fully open access ('gold').
In general, journals reviewed in MathSciNet are trustworthy. So check there for the journal you are considering.
The recent Analysis and geometry in metric spaces, while maybe not yet well-established, is as far as I can tell (after exchange with the editorial board at some occasions) trustworthy as any math journal; this is witnessed by the impressive editorial board.
Research in the Mathematical Sciences is very reputable, if new. The editor in chief is Ken Ono.