I am currently reviewing a paper for a journal and there's a section in the 'Methods' that may raise a questionable ethical concern on animal handling. The paper has a potential contribution to the field, however, there's this procedure that harms the animal in the study. I am thinking of rejecting the paper and raise a correspondence with the authors. Will the rejection decision a good decision for the paper?
You said, "MAY raise a questionable ethical concern". So, you might possibly misunderstand what they did, perhaps because they might have explained things poorly. I think that you should give the authors the benefit of doubt and let them defend themselves: recommend a major revision, acknowledging the potential value of their study, but make it clear that you insist that they document that they have followed ethical procedures in the method of concern. If they cannot provide satisfactory evidence of ethical methods, then you would be justified to reject the article for ethical problems.
And, as many other comments indicate, you should clearly explain your concerns to the editor.
The paper should indicate that the procedure was approved by the appropriate animal care committee (e.g. in the USA, an IACUC), Failure to include this information, in my opinion, is a major concern and grounds for strongly recommending rejection of the paper.
If the paper does indicate that an IACUC was involved, then it's reasonable to assume that the procedure was probably ethical, but it's fine to indicate concern and ask for an explanation for (1) why this procedure was chosen, and (2) what steps were taken to alleviate pain etc. That can still be a major concern, but on first pass I'd stop short of "strongly recommending rejection". If the authors didn't address my concerns then I would recommend rejection.
The key question is whether an animal care committee approved the procedure. If so, the likelihood is that it's considered acceptable by many people; of course, you don't have to agree and you can include that. When I was a student, one of my mentors rejected a grant application because he felt the discomfort and stress the animals would feel (even though they were IACUC-approved) wasn't justified by the potential benefits of the study. I've always considered this a model for my own reviews; animal studies come with a very high responsibility for ethical use, and it's fine to make sure your values are heard.
I love this question because it's so hard to answer. So I'll give two contrasting points.
1) Assuming the authors had approval from their respective animal care and use committee (IACUC) and followed all protocol, I do not think you should reject the paper on ethical grounds. As a reviewer at this phase, you should judge the merit of the science only. The onus resides with governing bodies (e.g., USDA).
2) Regardless of IACUC approval, if you identify potential unnecessary harm/distress otherwise avoidable given the research aim, it is your duty to reject on ethical grounds, since IACUC need not be an unopposed body.
Both points have merit, but my opinion lies closer to #1. If the authors followed protocol, your beef is not with the authors necessarily but with the governing body approving it.
As a reviewer, your role is to give recommendations to the editor, and it is generally not appropriate to have a direct communication with the authors outside of that process (unless they consent to that through the editorial process). However, I disagree with other answers here which preclude you from making your own decision on the ethics, and giving a recommendation on this basis. Here are some further thoughts:
The onus is on the authors to show their ethics: In cases where an experiment has been performed that should require ethics approval, it is appropriate for the paper (or supporting materials) to disclose the required ethics clearance details so that you can review this. If the authors have not done this, it is reasonable for you to proceed on the assumed basis that there has been no ethics clearance. You should state in your review that you are proceeding on this basis. Similarly, if the authors fail to give a clear enough explanation of the study to satisfy you, you should feel free to proceed with a negative recommendation as a result. I do not agree with the other answer here that says you should "give them the benefit of the doubt". The onus is on the authors to give all required information to satisfy the referees; it is not the job of the referees to guess what else might not have been supplied.
You need not defer to an ethics clearance: Even if an ethics clearance were to be supplied to you, I don't agree that you necessarily need to defer to this. Obviously it is something you should read and consider as part of your review, but the point of the peer-review system is to allow the referees to exercise independent judgment on the work under submission. If you form the conclusion that the experiment was unethical and the results should not be published, this is a recommendation you can legitimately make to the editor.
Don't depart from the review process: The editor will have the final decision on the matter; if your recommendation diverges from the findings of an ethics clearance, it is up to the editor to decide how to proceed. At the end of the day, the reputational risk lies on the journal, so the editor needs to take your recommendation seriously. I don't think it is appropriate for you to go further than this by contacting the authors. If you recommend against publication, and it is published anyway (either in that journal or another) then you can always write a "letter to the editor" raising this complaint.
Get more information if you need it: If you feel that there is insufficient information on the ethics for you to recommend publication, but you would like to give an opportunity for the authors to supply more information, feel free to recommend a revise-and-resubmit asking for further details on the ethics, including more details on the experiment and clearance process.