If your undergraduate degree is in math then I think the question of industry experience is irrelevant other than at the margins. In finance it might be a small plus if you were applying math in the job.
Most US doctoral admissions is based only on the undergraduate education and the selection is done by committees that evaluate your likelihood of success. Most early doctoral students in math in the US take quite a lot of coursework to prepare them for comprehensive/qualifying exams. If you haven't forgotten everything you learned you start at about the same place as everyone else.
One normally chooses a dissertation advisor later, after they have coursework and maybe comps behind them. You and the potential advisor will be in a much better place to evaluate one another at that point than at admission time.
One issue I see, however, is whether you can still get the important positive letters of recommendation from your previous professors who will attest to your likelihood of success. In that, it is easier to apply directly from the undergraduate program when people remember you.
This would be different other places, where individual professors select, and probably fund doctoral students; Germany IIRC.
Gaining confidence in yourself is a good thing, however. You might mention that somewhere, perhaps in your request to professors for letters.
And, if the finance was heavily mathematical and the program you want to enter is applied math, especially with a finance-adjacent option, it would be a plus. But in general, other things will count far more.