If you haven't signed anything, or at least been given a written offer, then you may not have an enforceable contract. Changing your mind within a few days will cause only minimal. if any. disruption at the other end. 

Just send them something in writing along with an email declining the offer and offering apologies for your earlier statements. You don't really need to say more than that. 

They might be unhappy, but would be less happy to have you join them grudgingly. 

I suppose there are some places where a verbal agreement can hold legal weight, but I doubt that even then, they would think it wise to enforce it. They likely have other options.