It's not crazy. We've had people at my elite R1 give up tenured positions to come in as assistants or even lecturers. Usually they had good reason to leave their old institution (lack of intellectual stimulus; bullying; family; etc.).
The main thing is that you need to determine exactly what you will need for tenure at the new place. Will they start you off in year 5 as a "high assistant prof?" If so, do they expect new work to come out in that single year or can you use your old tenure file in its entirety?
An elite liberal arts college or regular R1 might hire at the high assistant (if not associate), the most elite R1s might suggest you reset your assistant prof clock to zero and decide when to come up for associate on your own volition when you feel your file is strong enough.
But this might be putting the horse before the cart. You have to apply and actually be offered a position before you should worry. It's only then - and after a few rounds of negotiation- that you can make a rational choice.
Until then, apply and dream fond dreams.