I am submitting a research paper to a journal and they asked me to sign the copyright agreement in favour of the editor before the paper is accepted. Is this right? And what happens in this case is if the paper rejected?
This is rather normal for non open-access articles in my field (Physics), however all journals I have submitted to until now had a clause that the copyright transfer is voided if the paper is not accepted or withdrawn. For example IOP’s copyright agreement contains the following sentence:
In the event that the Article is withdrawn prior to acceptance, or is rejected, this agreement shall have no effect and no party shall be bound by it.
If such conditions are included, requesting such an agreement at an early stage seems somewhat reasonable to me, since it would be a waste of time to review your paper if you did not agree with these terms.
If such conditions are not included, this is a clear warning sign that the publisher may be up to nothing good. However, even with such a clause, problems may arise, as the publisher may just instantly accept your article.
For open-access articles (pay-to-publish), a similar situation holds, however, the copyright agreement usally requires less commitment from you, as you should retain most rights on your article (since the journal does not make money with being the only one making your article available). This not applying to the copyright agreement is another warning sign of a predatory publisher.
In both cases, you should read the copyright agreement carefully and ensure the reputability of the publisher.
Unfortunately, a number of large-scale publishers expect you to sign over the copyrights before the article is accepted. For instance, the American Chemical Society, as well as the Materials Research Society will continually hound you about the transfer agreement until it's submitted, and moreover will not formally accept the article until they have the transfer agreement.
Many big publishers will often start asking pretty much immediately (I don't know if this is policy or just brain-dead software configuration). You can generally safely ignore these requests until the article is actually accepted, however, because until then the idea of copyright transfer is moot. That is certainly the policy that I follow, and it hasn't caused me any trouble yet.