For reasons that elude me, a student I taught several semesters ago sounded me out about writing a letter of recommendation for graduate school. I won't go into it, but my recollections of this student are uniformly bad. There is no way I would want them pursuing graduate study in our department. I am torn between declining to write a letter, or taking the opportunity to quietly warn colleagues at other institutions about this person. Do I have an ethical duty to only offer to write a letter if it will reflect positively on the applicant?
Never write a bad recommendation letter. If you can't say anything nice, say nothing at all (and tell the student you can't write the letter).
As for warning other. Leaving aside the law for a moment, the only time I can see it being appropriate to warn others about a student is if you believed them to be dangerous. Examples of this might be a history of sexual harassment/assault, bullying or other abusive behaviour.
Don't do this. It's not fair to the student. It's like buying something that turns out not to be what it was advertised as. The student will be using your letter to apply to graduate school believing it is helpful when it is actively harmful. If the student is rejected everywhere, for example, he'll never be able to diagnose what's wrong or how to fix it.
Instead, talk to the student. Tell them you can't write a good recommendation letter, and if they still want it, write all the positive things you can truthfully say about the student. If you can't say anything positive at all, decline to write the letter.
In the US, at least, "warning colleagues at other institutions" would possibly be illegal. I would consider it unethical in any case. That is a secret blacklisting that should never occur.
But basically, you should never write a bad recommendation. Instead you should tell the student that you won't/can't write a letter. Let it go at that if you can, but if pressed, tell the student why.
If you are generous you can let the student try to convince you that you have an improper impression. I don't think that is necessary in most cases, however.
But, ethically, you can't write a positive letter that you don't believe in. That is a kind of fraud.
I think it is generally accepted that letters of recommendation, and other references, are cherry-picked to use only the good ones. Thus, selection panels are already aware that the absence of negative recommendations is largely due to the filtering mechanism of having the applicant choose their own referees. Panels will also usually notice if there is an obvious omission from references and letters --- e.g., direct supervisor in previous position is not listed as a reference.
In view of this practice, I would say that it is not a good idea to write a negative recommendation --- just decline to write one instead. I would be inclined to say that the student should have the same opportunity as other applicants to cherry-pick his references to get the best ones, and thereby compete on an equal basis. If he is a particularly bad student, he will probably have difficulty getting references from relevant past supervisors, and this will be obvious on his application. This filtering mechanism will still put him at a disadvantage relative to a better applicant.