You should be honest up front about how strong a recommendation you can write.
While of course there's nothing stopping you from refusing to write it at all, that seems to me like a somewhat unhelpful thing to do. Even if you have nothing good to say about the student, why wouldn't you give your opinion of them when asked? You don't have to, but why not? So, the question you're asking shouldn't really come up.
In practice, students probably realize that a bad recommendation won't do them much good, so if you're at a point where you hesitate to write it because of how bad your relationship was, they probably wouldn't want the recommendation from you anyway. If you tell them ahead of time, "I can write it but I won't be able to write a positive one because I don't feel comfortable vouching for you to that extent", the student will put two and two together and try to look for another recommender who will write a better letter. Perhaps they had a "backup option" in mind: Someone who hasn't worked with them as much, but might have a more favorable opinion.
Often a student will have a list of possible people to ask for a recommendation, and it is helpful to know if someone at the top of the list won't write a good one, because they can just move on to the next person. Probably the student would prefer that you told them you won't write a strong recommendation instead of agreeing out of a feeling of obligation and then writing a weak letter.
I think some people also deliberately agree, and then write a negative letter (whether through outright criticism or just damning praise) to sabotage the student's career. I personally find this dishonest and mean, but the recommendation is supposed to be a confidential appraisal of the student, and I can see how some feel it is justified (for instance, perhaps you wish to stop a very careless student from becoming a heart surgeon because it will put lives in danger). So, I suppose it's your decision whether you think the student is just not good enough to help, or so bad that you should actively impede them.