I seem to have reached the phase of my career where I receive regular requests to externally evaluate faculty: this means that I am evaluating the research program of tenure-track faculty (so far always at US institutions, so let's concentrate on this case) and writing a letter to be used in some ways for the pre-tenure / tenure / promotion processes. If it makes any difference, I would be more interested in answers that apply to the field of mathematics.
Well, sometimes I have reasons that I might not want to do this. Here is a list of reasons that either have occurred to me or seem plausible that they might occur with others:
I feel that I am too busy: either too busy to do a good job in the time allotted, or just so busy that it would make my life easier to decline.
I don't have much insight into the candidate's work, and I feel that many other people could do a better job.
In order to do a good job I would have to investigate certain things, e.g. whether and why a certain paper has not yet been published. It's hard to investigate things in academia completely anonymously, and whether this type of investigation would be well-received or even appropriate is not completely clear.
I feel that the candidate's work is not very strong. (Note: not very strong compared to what is a key question here, but a sticky one. I have found that institutions which are more teaching focused often ask their candidates to be evaluated by standards which sound very rigorous and exacting to me, a faculty member at a major research university.)
Especially in the last case, it's not so clear "what's in it for me" to write an evaluation that says that a candidate's work is not as good as that of many other people I know in the field. They are still working in my field, so I would rather have them there than not. I have no idea what the chance is that my letter would be taken seriously in a failure to hire/promote them. Either way, there are reasons for concern on my end.
My main question is: if I decline to write such an evaluation, is the act of declining likely to have implications for the candidate? (E.g. is it likely that the declination would appear on the candidate's dossier?) The subsidiary question is: if I have reasons like the above that would make me prefer not to write an evaluation, is it nevertheless important to write one? Is it "the right thing to do"? (Of course one wants to do a certain amount of service to the academic community. On the other hand, many/most academics get offered so many service obligations that they have to turn some down. This question should be understood as relative to doing other service tasks, not doing less overall.)