If you haven't signed anything then you don't have a contract in most places. Changing your mind within a few days will cause only minimal. if any. disruption at the other end.
Just send them something in writing along with an email declining the offer and offering apologies for your earlier statements. You don't really need to say more than that.
They might be unhappy, but would be less happy to have you join them grudgingly.
I suppose there are some places where a verbal agreement can hold legal weight, but I doubt that even then, they would think it wise to enforce it. They likely have other options.