The distinction between a conference one attends and a conference one is invited to speak at is relevant at all levels of one's career in math.
If you don't give a talk, the distinction between a conference you attend simply by registering and a conference you attend by invitation only or only if accepted (like workshops at AIM, Banff, or Oberwolfach) usually isn't particularly important.
However, for an undergraduate to have been specifically invited to a invitation-only conference is rather unusual, and might indicate that you have a good reputation among senior mathematicians in the field, certainly a plus if applying to graduate school. If so, by far the most important way to communicate this in your file is by getting a letter of recommendation from those senior mathematicians. But it can't hurt to indicate it in your CV as well; in your "Conferences attended" section, just add a parenthetical "(invitation-only)" to the relevant entries.
But let me reiterate that what you put on your CV is 100 times less important than getting a strong recommendation letter from senior mathematicians who know you well and think highly of your potential for research.