A bottle of wine is an excellent gift. You can pick up a decent wine for 30 dollars and attach a nice hand written note to it.
Of course you also tell a short story about the wine when making the gift, I myself e.g. picked up a box of very expensive wine in Italy once. I gifted a bottle on two very special occasions.
Regardless of what the more nerdy SE community says "a book, a vase, or a figurine" might not be perceived by less nerdy people very well. Action figures are usually reserved for children, and especially making a figurine of yourself (Superhero Prof. Millemila?) would entertain the entire university. That's just ridiculous.
Lasting gifts can be great, but are more expensive, I myself made good experiences with a mont blanc fountain pen, a good German or Swiss watch, a leather wallet, a handmade razor, a shaving foam brush, handmade cocktail glasses, and a fancy lighter. Staying to the classics might seem old-fashioned, but people like this really.
Giving out mediocre but lasting gifts is arguably worse than a bad one time use gift: A bad wine will lead to a bad evening at most, a unwanted book might stick around much longer in the dusty corner of the receiver's office.
I would be cautious with these expensive gifts in your setting though. A bottle of wine in the medium price range on the contrary is a common gift in business settings, and you don't need to worry about it being inappropriate at all.
Make sure to remember that the "thank you" card is probably the biggest gift you can make anyways. Especially if you express, that you are always happy to help your former student in the future. Use a nice fountain pen and thick paper. Use a suitable format, e.g. A5.