I am still early on in my PhD. I have not passed my comprehensives (or even began studying for them yet), but I have finished my courses. Over the past year, I began working with a group that I initially felt was a good fit for me. At the time, they were the only ones working in this area in our department. However, as time has gone on, I have realized that my professor has absolutely no idea about my field and their relation to it is completely from a different angle (hardware vs software). They are unfamiliar with the conferences in my area, acceptable standards, state-of-art work, etc. In fact, the other day I was told that everything my professor knew about my area was learned from me. Originally, I thought they were active in this field, but actually they are in a COMPLETELY different field altogether. Recently, we hired a new professor that I am certain is in my field (publishes in the conferences of my field, etc.). I have decided to switch and begin working with this new faculty member. Is this unusual? I am very scared of my current faculty member and have heard horror stories, who by the way, is a retired professor (emeritus) but still can advise me.. but at this point I don't think I want it!
I would say that you would be better off going with the new faculty member. The question now is "How do I switch?" If this emeritus guy does email (which some of them are so old they do not. Although it sounds like you maybe are in Computer Science, so hopefully the guy uses email) I would write a brief email to him explaining why you are switching. This avoids a face to face if possible. Then unilaterally switch groups. I would take care of any introductions with the new advisor in person, however.
Much of this also depends on things like if you share an office with people in the uninformed group, how large your department is (If you have 40 profs in the department, you can fade away. With 4 profs, not so much), etc.
I switched drastically in what I was studying (Algebraic group theory as a MSci student to applied statistics as a PhD student) and there were a few eyebrows raised. But, it was ultimately my career. And people in my department came to accept it. It is SO much better to have a good research group than to have a placated advisor. Do not put yourself through that.