Is a research position possible after spending time at a teaching university?
Yes, it's possible. There's a mild stigma to applying from a teaching university, since it advertises lack of success getting a research job in the past. Because of the high ratio of candidates to positions, academic hiring is often risk averse and the perception that nobody else wants a candidate will worry search committees. However, this can be overcome.
One thing to keep in mind is that the past difficulties could simply have been bad luck. Even very strong candidates typically do not get many job offers, and bad luck can easily change "a few" into "zero".
It's possible to strengthen one's application by writing additional papers, but this generally requires better papers or more papers per year, since search committees will normalize for time: if you've spent longer they will expect more. However, the focus is more on the future than the past. If you have a productive few years and show signs of maintaining that productivity in the future, it can make up for fallow periods in the past. If you quit publishing, then the chances of a research job will rapidly drop to zero until you resume publishing. (There are a lot of people who would like to do research but aren't prepared to actually do it, since they aren't up on the current research literature. You can't get a research job unless you demonstrate that you aren't one of these people.)
In an ideal world, search committees would also normalize for the applicants' circumstances. For example, research productivity at a teaching university would be viewed as evidence that the candidate would be even more productive at a research university. Unfortunately, in practice these effects are often underestimated or not taken into account at all.
One possibility is that there's something wrong with your husband's application. For example, maybe his research statement is not compelling, or one of his letter writers is insufficiently supportive, or maybe one of them does not know how to write an effective letter of recommendation. (You'd think that should never happen, but some well-established mathematicians simply do not know how to write effective recommendation letters. If his thesis advisor is one of them, and the other letter writers are less energetic since they assume the advisor will make a strong case, then it could be really bad for his job search.) I'd guess that 5-10% of job applicants have something seriously wrong with their application that they seem totally unaware of. This is a low fraction, so your husband probably isn't one of them, but if possible he should discuss all aspects of his application with a trusted mentor who has guided numerous students to the sort of job your husband would like. Sadly, he may not have such a mentor, but it's good to keep an eye out for one. For example, if he strikes up a conversation at a conference with a senior mathematician who seems approachable, it's worth asking for job search advice.
He thought a hiring committee would scoff at the lower output of papers done by someone in a teaching job vs. someone churning out problems in a research post-doc. He even wondered if not having "XYZ Awesome Post-Doc" on his CV would trump any research he did.
Lower productivity because of other duties is a serious factor here. The CV prestige issue is real but considerably less important: prestige might serve as a tie breaker but won't get anyone a job if their actual accomplishments aren't commensurate.
Letters of recommendation will be by far the most important factor. What your husband needs is really strong letters that address his circumstances, talk about how impressive his research is and why, and explicitly make the case that he belongs at a research university. Letter writers generally recycle letters from year to year with some updates to incorporate recent papers. If his letters are not updated to address the teaching/research university issue, then they will not help him. In particular, if they don't say in strong terms that he ought to be at a research university, then they'll be viewed as damning him with faint praise.
This is something he can discuss with each letter writer, along these lines: "As you know, I've been working at University X for the last couple of years. It's great to be in a tenure-track job, but I'd really like to work at a research university. To move to one, I'll need letters of recommendation that address this issue and make a strong case that I should be at a research university. Would you be comfortable writing such a letter for me? Of course I'll understand if you can't write one, since I know I'm asking a lot, but I'd rather ask someone else than waste everyone's time with an application that doesn't have the support it needs."
This is an awkward conversation, but it's much better to ask than to leave it to chance.
Would he be able to get back into academia if he had to leave to do actuarial work/industry/something that helps pay the bills?
It's possible he could get another teaching-oriented job, although it's by no means a sure thing. It would help a lot if he could spin the other work as informing his teaching. For example, he could teach actuarial mathematics or incorporate realistic industrial applications. In that case the non-academic experience could be an advantage; otherwise applying from outside academia would put him at a moderate disadvantage.
Unfortunately, the chances of getting a research-oriented academic job from industry are low (assuming the industrial job is not at a prestigious research lab like Bell Labs). There are people who have done it, but it's not likely or easy. Few people can maintain a high-quality research program on the side with no support while holding a full-time job, and this is necessary for returning to a research university. In particular, there's virtually no chance of returning based solely on having done research in the past, without having actively continued in the meantime.
The way it typically plays out is that you reluctantly go to industry intending to maintain your research and keep applying for academic jobs. For the first year or two, you complete and write up work you began in academia, and it feels like everything is going well, but your job applications don't do any better than they had before: you've got a more substantial research track record, but your industrial position emphasizes your inability to get a job in the past. Still, you figure that accumulating more papers will eventually tip the balance in your favor. Unfortunately, the next few years don't go as well. It's hard to find the time for research, you have few people to talk to or derive inspiration from, and progress is slow. However, you're gradually getting somewhere, so you figure it will just take a little longer. A few years after that, you start to lose your resolve to do research at all. Even if your applications were successful, would you really want to take a 50% pay cut, give up your job security, and move to another city to restart your career? And it's hard to keep your focus on an incredibly time-consuming hobby that seems like it may never lead anywhere professionally. You quit applying to anything but dream jobs you're pretty sure you won't get, and eventually you give up on them as well.
The good news is that this path doesn't generally end in depression, but rather the discovery that there are plenty of fulfilling life paths outside of research universities. It's by no means a bad outcome. However, leaving academia for industry can be really stressful if you want to return to a research university, since you have to either give up or work like hell to maintain your research program.
