For reasons that elude me, a student I taught several semesters ago sounded me out about writing a letter of recommendation for graduate school. I won't go into it, but my recollections of this student are uniformly bad. There is no way I would want them pursuing graduate study in our department. I am torn between declining to write a letter, or taking the opportunity to quietly warn colleagues at other institutions about this person. Do I have an ethical duty to only offer to write a letter if it will reflect positively on the applicant?
Never write a bad recommendation letter. If you can't say anything nice, say nothing at all (and tell the student you can't write the letter).
As for warning other. Leaving aside the law for a moment, the only time I can see it being appropriate to warn others about a student is if you believed them to be dangerous. Examples of this might be a history of sexual harassment/assault, bullying or other abusive behaviour.
Don't do this. It's not fair to the student. It's like buying something that turns out not to be what it was advertised as. The student will be using your letter to apply to graduate school believing it is helpful when it is actively harmful. If the student is rejected everywhere, for example, he'll never be able to diagnose what's wrong or how to fix it.
Instead, talk to the student. Tell them you can't write a good recommendation letter, and if they still want it, write all the positive things you can truthfully say about the student. If you can't say anything positive at all, decline to write the letter.
I think it is generally accepted that letters of recommendation, and other references, are cherry-picked to use only the good ones. Thus, selection panels are already aware that the absence of negative recommendations is largely due to the filtering mechanism of having the applicant choose their own referees. Panels will also usually notice if there is an obvious omission from references and letters --- e.g., direct supervisor in previous position is not listed as a reference.
In view of this practice, I would say that it is not a good idea to write a negative recommendation --- just decline to write one instead. I would be inclined to say that the student should have the same opportunity as other applicants to cherry-pick his references to get the best ones, and thereby compete on an equal basis. If he is a particularly bad student, he will probably have difficulty getting references from relevant past supervisors, and this will be obvious on his application. This filtering mechanism will still put him at a disadvantage relative to a better applicant.
taking the opportunity to quietly warn colleagues at other institutions about this person
There isn't such a thing as a "letter of dissuassion" or "letter of discouragement", which is what it seems you are considering writing. "Recommendation" means:
The act of telling somebody that something is good or useful or that somebody would be suitable for a particular job, etc.
Therefore, if you really think this person is not suitable for the graduate school, it means you can not recommend them, i.e. it makes no sense to write a letter of recommendation for them.
In the US, at least, "warning colleagues at other institutions" would possibly be illegal. I would consider it unethical in any case. That is a secret blacklisting that should never occur.
But basically, you should never write a bad recommendation. Instead you should tell the student that you won't/can't write a letter. Let it go at that if you can, but if pressed, tell the student why.
If you are generous you can let the student try to convince you that you have an improper impression. I don't think that is necessary in most cases, however.
But, ethically, you can't write a positive letter that you don't believe in. That is a kind of fraud.
At least as an auxiliary point: in the U.S., as opposed to many other cultures, there is the idea that earlier-life failures do not necessarily terminally limit the rest of one's life. There are no uniform exam/placement systems. There are second chances... (which is generally a good thing, although it definitely complicates everything).
So, in that style, even if a student has screwed up so far, perhaps one should not communicate that screw-up forward. Don't impede the fresh start...
In particular, I would never write a damning letter of recommendation. I might decline to write any. I would not write a dishonest one.
(And, in the U.S., comments on mental health issues, or physical health issues, are more-or-less legally prohibited, so far as I understand. Yes, in my experience, many problems are based in mental health issues... Perhaps we do not have a good system to talk about such things.)
EDIT: apart from my irrelevant grammatical error ("oppose" which should have been "opposed"), I am mildly surprised that anyone would object to "second chances". (Even the possibility of my editing for grammatical perfection could be a "second chance"?)
"U.S. kids are weak/lazy..." Yes, I've heard that many times from my faculty colleagues at my R1. But what I think it really is that U.S. kids are not as desperate as kids in some less-fortunate countries, so will not so easily bow to crazy/random (often inappropriate...) requirements.
Specifically, I would hope, ... that things are not soooo tight in the world that any slight mistake is fatal. Especially... kids? Seriously?