I got the outcome of my paper after 2 months from IEEE Transaction. The editor said its out of scope and suggested to submit in a sister journal and he cc the outcome to the editor of that journal. There were 5 reviewers, 2 said minor revision, 1 said i dont have any question. 1 reviewer asked one question. And 1 reviewer said the idea is not very novel and gave same major comments. However, associate editor summerized that the quality of work is recognized by the reviewers but encouraged to submit in the sister journal. Here, i want to know that what could be the reason of having 5 reviewers when there are usually 2-3 reviewers and i also reviewers of this IEEE TRANSACTION? Secondly, do i need to talk to editor that it could be major revision but how they decided out of scope with reviews? Lastly, does out of scope means rejection and editor encouraged me to submit to sister journal of that transaction does it means it will go through complete review process of that journal?
One at a time:
Why are there 5 reviewers instead of the usual 2-3? There're many possibilities, the two most likely of which are 1) they routinely invite five reviewers because historically, not all of them will accept. This time they did. 2) Some of the first reviews received were poor (your description makes it seem like so, but I don't know the details) so they invited more reviewers.
Do you need to talk to the editor about how they came to an out-of-scope decision with major revision reviews? Probably not. The editor's decided that your paper is out of scope already, which is usually equivalent to a rejection.
Does the editor suggesting you submit to a sister journal mean you have to go through the review process of that journal? Yes. The sister journal will have different requirements, and the editor might not even be a member of the editorial board of the sister journal.
To make it clear, you need to actually submit to the "sister" journal. I doubt that it is automatic, since the paper is still yours. So, you need to formalize the submission.
The "major revision" suggestion might have been a consequence of the "scope" issue, and may not be required at the second journal, but there is no way to decide that here.
Once you submit, the editor will make decisions about the paper. One would be to utilize the existing reviews in whole or in part and another would be to start the review process over. It is their decision and depends on their quick look at what they have before them.
Since you have a set of review suggestions, you could incorporate the simpler ones in a new edition of your paper and send it to the new editor, delaying the major work until you know more.
Why there are five reviewers is known only to the original editor. Perhaps something in the early reviews suggested that more was needed. But perhaps it was as simple as wanting to give a new reviewer some experience and to see how the reviewer did. Only the editor can explain "why".
I agree with Allure that as the editor already mentioned "out-of-scope", save your time to immediately consider the suggestion for sister journal. I had personal experience, good comment from one of the reviewers but it ends with saying out-of-scope. We did not waste time and submit for the more suitable journal.