I'm a computer science student and recently developed a new formula in the field of math. It's something important and I'm confident that the formula does what is intended to do because of my tests.
But I don't have experience in research mathematics and I don't know how to proceed now, what should I do?
Essentially echoing @Buffy's answer:
First, it's a great thing that you have found something cool, and are happy about it. That really is the immediate personal reward for "research". Sure, maybe "benefiting others", or "advancing human understanding", but those are distant and not-immediately-gratifying mechanisms.
Still, there are "professional" mathematicians and computer scientists (etc.), and collectively they/we have been thinking about things for a long time. So, first, it is all-too-possible that you've rediscovered something that's been known for some decades. I've done that myself "uncountably-many" times. There's no dishonor there... quite the contrary, it shows that your sensibilities about what to look at are aimed in a good direction.
Apart from the actual scientific content, there are pretty stringent stylistic conventions in various fields' publications, even on preprint servers. I cannot pretend to communicate these to you in an answer here. You need a more senior person to help you navigate the conventions. And, sadly, to be clear, if you transgress those conventions, people won't take you seriously, and may not even be willing to read what you've written.
Good luck, and have fun! :)
You need feedback on your idea and how to present it. Talk to a professor in your department who you trust and who might be interested in what you have done. You might want to also talk to someone in the math department who will advise you about the significance of what you have. Perhaps the CS prof can put you in touch with a math prof if you can't do that directly.
Once you have established that it is something possibly publishable, write it up. Focus in the paper on what it is and why it is important. Mention possible applications. Provide a proof if necessary or what evidence you have. Get feedback on what you write (and read a few math papers to get a feel for the "flavor" of the writing.
Then, if the feedback is good, figure out an appropriate journal (or conference in CS) and submit it there. You will get additional, but independent feedback in most cases, unless it is judged trivial. Follow the journal's process.
The process will be long and a lot of work. The original inspiration is often the easy part.
I will take a bit of an unconventional stand on this but:
You probably have not "developed a new formula in the field of math".
When stating something like this you have to consider that there is a very large volume of published information in the very old field of math. In order to establish that your formula is actually new, you'd have to have read up on at least the adjacent fields of math that are relevant to your problem.
It is a very heartbreaking thing, but when you stand between the large number of bookshelves in your institution's math department's library: Are you sure that what you came up with is new? A considerable amount of time in research goes into establishing what has been published on the matter before.
I don't write this to be mean. It is a painful experience many of us have gone through many times to come up with an interesting thought, only to find out that it was well documented in anno. ~1700. Please talk to your department and professor about it and be prepared that they will hand you a reference. Then do not give up and continue! This is part of the process.
