I am writing a research paper. A small but important idea for the paper was found on MathOverflow. I know the real name of the user. I tried contacting him and offered him co-authorship, but I got no reply. How do I acknowledge him now ? Do journals encourage citations under pseudonyms?
Let's say it was a big enough contribution to require a citation. You could cite the MathOverflow post, as Massimo Ortolano suggests, especially if the entire conversation was generative, and it would be worthwhile for readers to visit.
However, linking to a pseudonym is not going to be clear or able to be attributed to that scholar. While William Sealy Gosset stuck to a pseudonym, he also had a non-academic job to protect.
There are citation formats for message board postings, etc. in various citation systems, as shown here for MLA from Purdue's Online Writing Lab:
Include screen names as author names when author name is not known. If both names are known, place the author’s name in brackets.
Editor, screen name, author, or compiler name (if available). “Posting Title.” Name of Site, Version number (if available), Name of institution/organization affiliated with the site (sponsor or publisher), URL. Date of access.
Salmar1515 [Sal Hernandez]. “Re: Best Strategy: Fenced Pastures vs. Max Number of Rooms?” BoardGameGeek, 29 Sept. 2008, boardgamegeek.com/thread/343929/best-strategy-fenced-pastures-vs-max-number-rooms. Accessed 5 Apr. 2009
The best practice is to use both screen name and real name, "if known."
HOWEVER. I think the best practice, instead of violating pseudonymity, would be to send an email to the academic email address of the person you believe this is. Say that you are interested in citing this, and wonder if they would prefer this to be cited with both names, as "personal communication" with their academic name, as some closely related work they had on a blog post under their academic name, etc.
If the person declines to confirm that they are this person, you can ask in a MathOverflow chat or as a comment to that answer whether/how that user would like to be cited.
If you are truly hoping to honor and thank them--especially if it is an acknowledgement rather than a citation--then give them the courtesy of having some say in how they are recognized and whether their pseudonym becomes published knowledge.
EDIT specific to MathOverflow: MathOverflow is indeed cited fairly often, and it appears that most of the top citations are to real names, according to a Google Scholar search. One could avoid (newly) revealing the person's identity by searching to see if the person's real name is already associated with MathOverflow on Google Scholar, check the post to see if it's the same username/permanent identifier, and if so, cite in the same way the previous person did.
Discrete Lizard offers a Meta MathOverflow post dealing with a very similar issue, which suggests leaving a comment asking, and/or asking moderators to send a message to the user and ask. Finally, Peter Taylor mentions that the norm on MathOverflow is to use real names, such that many of the user names already reveal one's identity.
Don't acknowledge, cite the Mathoverflow post!
The following theorem is taken from  ...
 A. User, blah blah,
Mathoverflow has also a cite function at the bottom of the posts to retrieve a possible reference in various formats (e.g. BibTeX).
I would cite the contribution. Web citation is common. Yes, it is NOT as good as citing archived paper journal literature (web is very unarchival). But so what. You do the best you can. People cite personal communications, unpublished work, etc. You do what you can.
You could also acknowledge the person in the "Acknowledgements" section. Most people just list funding agencies here, but I ended up having a lot of great mentor type conversations with people during my Ph.D. (more so than with my advisor). So I just acknowledged them.