You certainly can refuse to be named as co-author, but at this point it's probably easier just to sign off publication, unless there are stronger reasons than you've given.
Most reputable journals have processes that require all credited authors to sign off on publication, so you can't be credited as an author if you don't want to be*. However, this may mean that the paper can't be published at all - your supervisor shouldn't be claiming sole author credit for something that is largely your work, and even a mediocre journal is likely to balk at accepting a paper if it learns that one of the authors is unwilling to be associated with that work.
Publishing this paper probably won't help your career significantly, but from the description you've given it doesn't sound likely to harm it either, especially if it's not in the field that you're aiming for. I changed career and I have several publications on my CV that have no relevance to my current career; I doubt any of my co-workers have even bothered looking them up, let alone passing judgement on whether they're a valuable contribution to the literature.
*Some loopholes may apply. I had an ex-boss who listed me and another researcher as co-authors on a poster and conference presentation, even though he knew we had significant objections to the content. While the associated journal required sign-off from all authors on published papers, this was not the case for conference material. We had to write in to request that the "error" be corrected for the published proceedings.
