The earlier answers make good points...
And I would say, in your situation, most likely the "papers" are of little consequence in admission-or-not, and a 3.5 GPA is fine. No one cares beyond that. Letters matter a great deal when I recommend-or-not admission. What I read first, however, is the personal statement. Do be aware that in many programs there is a dominating impulse to "mechanize" admissions, which obviously has difficulty making use of either letters of recommendation or personal statements. But, still, if you explain your interests, your self-study, this can have a huge, repeat huge, impact on admission chances. I suspect you've not had much contact with math faculty in the course of your self-study, so can't get letters from math faculty. (If this is not correct, so much the better!) Letters from engineering faculty about students' mathematical talent are typically nearly worthless, since there seems to be a general tendency for "engineers" to believe that "mathematics" is "just a tool", and that they've mastered it, etc. Nevermind.
But the point is that you should not expect even glowing letters from engineering faculty to have much impact on admission to math grad school, exactly because the prejudices of many engineers are very familiar to mathematicians.
The subject-test GRE may help your chances, and you must take it, because a low-ish score is vastly better than the cluelessness indicated by having no score at all.
But for a person approach mathematics with genuine enthusiasm, but "belatedly", the personal statement, explaining unapologetically how you came to your present course, is the most important thing. Very few people choose to self-study mathematics... (although quite a few seem to believe that they have special gifts that require no study...!)
Explain yourself in your personal statement. Get letters from the people who think well of you who have the best idea of what professional mathematics entails.
Edit/addition: very literally, it is best to have letters from people who have been to math grad school themselves, preferably at better places, so have an idea of what that entails. Many or most math grad programs have "breadth" requirements apart from the eventual goal of "original research"=thesis. So it's not so much a question of your letter writers' "peer-reviewed publications" so much as their first-hand experience with math grad school. (These days some people who've done PhD's in math do end up in engineering depts, and vice-versa, but this is still unusual.) Thus, a recommendation letter should perhaps literally say something like "From my first-hand experience in math grad school at X, and observations of math grad students at Y and Z, [student] will be a success." That's the kind of thing that leaps out at me when I read these letters. (No, don't have your letter writers send CVs or publication lists.)