OK, well, you have a problem. By the terms of your appointment, you are entitled to stay for the 24-month period. The faculty you're working with made a commitment to support you for 24 months, and generally speaking, it is his duty to do so. However, that's not the whole story. You also have an implicit obligation to do good research that advances his interests as well. And, depending on your goals, you probably also need his intellectual support and mentorship to achieve many of the benefits of a visiting scholar position. So, you need to work out a mutually satisfactory resolution with him. This is more a situation for interpersonal negotiation and compromise than a situation where standing on your rights and thumping the table is going to help you.
You might need to make some compromises, compared to how you thought this position was going to work out. To help with that, to begin, you should start asking yourself some hard questions and figure out what your goals are. What are your career goals after you finish this position? What would you like to achieve? Are you looking to get a good reference letter from the faculty you are currently working with? Are you looking to strengthen your c.v. with a stronger publication record? Do you think you can do good research on your own, with no mentorship, collaboration, or support from your faculty member? What kind of job are you looking for after you finish? Are you looking for a research position, or for an industry position? Can you make productive use of the next 16 months (in a way that will help your longer-term career prospects)? How strongly do you feel that you want to stay in the current position for the full 24 months if you have no support and no interest from your existing faculty member? The answers will determine how you act at this point.
Keep in mind that there might be no perfect solution to your situation, so your job at this point is to try to guide things to the least-bad outcome that is at all feasible. To do that, you absolutely must know what your priorities are and what your "nice-to-have's" are.
I can see a couple of possible options for you:
You could sit down and have a frank talk with the faculty you are working for. You could say, look, you offered me a 24-month position, you made a commitment, now it is your obligation to fulfill it. You can be polite but firm. However, this might not win you any friends, so if you were hoping for a good reference letter from this faculty member or collaboration and mentorship for continuing research, you might be out of luck, and you could be stuck in a toxic environment for the next 16 months. If you take this tack, you are basically offering the faculty member nothing positive in return, so the best plausible outcome is that the faculty member honors his commitment and ignores you for the rest of your appointment. The worst outcome is that the faculty tries to find some way to screw you.
Alternatively, you could try to understand better your faculty member's situation and then try to find a way to make yourself valuable to him. Personally, I think this is probably a much more promising direction. You can always fall back to the "you have an obligation" option above if this fails.
From your position his actions might appear arbitrary and capricious, but there is probably a logic behind them from his perspective. You could try to understand what is motivating your faculty mentor and then use that to see what you can offer him. For instance, maybe he is under tight pressure to see results, and you are not contributing in the way he had hoped. Keep in mind that you are biased; you might feel like you did an awesome job and met all of his expectations, but he might not feel that way. Or, his goals might have changed. So, you could try to understand his perspective, and then figure out how to make yourself valuable to him. Basically, look for a way to make this a win-win situation.
You say the faculty seems more interested in working with your friend right now. Well, that's a valuable clue. Maybe your friend is making contributions in some way that you aren't. Have you considered trying to find a way that you can contribute, that your faculty member would value and that would make him eager to keep working with you and make supporting you a priority for him? Have you tried asking some questions to probe about that? His needs can change over time. There's an unwritten assumption, when you join someone's group, that you will act as a team: that you will both act in each others' best interests. In particular, this is not a zero-sum situation (or it should not be); if you're doing things right, when you do great work, it should benefit both you and your faculty. You need to be flexible. If your faculty member's needs change, you might need to change your focus and your energy to support his direction. If your faculty member's needs have changed and you are not adapting to make yourself a valuable member of the team as his goals change, then that could explain his reaction.
You say your faculty has a DARPA contract, and that contract continues, but you are not a main person on that project. Well, have you considered trying to make yourself a main person? Have you considered making yourself indispensable to that project or finding a way so that you can make major contributions to the success of that project? This is not something that is appointed or handed out; what makes you a main person is not a title that someone else hands to you, but rather your own independent action. In the research world, people typically aren't going to tell you what to do. Instead, they're expecting you to identify a way you can be valuable to the project, and then go do it. You're a smart person; I'm sure you can find a way to make a contribution to that project that they will find helpful, and that will be beneficial to your career. Go do that a couple of times, and before you know it, you will be a main person on that project, not because anyone else decided you are, but because you decided to become one and you put in the energy and hard work to be.
From your question, we only get your perspective, not that of your faculty member's perspective. Personally, I wouldn't be surprised if you haven't fully appreciated your faculty member's perspective yet.
Overall, my advice is:
Enter this with a willingness to compromise and change. Adapt. Be prepared to find a compromise with your faculty member. Don't start from the position that you're entitled to a great working situation; you may be, but realistically, at this point, that kind of attitude won't help you attain one. Open your mind to compromise solutions you might not have previously considered, even if you have to accept something that's less than ideal from your perspective.
Find someone else you trust (a secondary mentor, another faculty member, something like that) and talk to them about the situation. Ask them for advice. Respect what they are telling you: be a good listener (if you find yourself arguing with them, time to back off and listen and try to understand what they are telling you). See if this can get you any additional information about what might be motivating your faculty or ways you might be able to resolve the situation. You're in an Ivy League school; odds are that someone else in your department cares about visiting scholars.
Go talk to your advisor. Ask questions. Listen. Empathize. Try to find a win-win outcome that both of you can feel like offers something positive. You can be frank and honest that your relationship seems to have gotten off on a bad foot and you'd like to work out a way to improve the relationship and find a way to be useful to him that also benefits your career. Ask him for help crafting a performance improvement plan that from his perspective would help you do a better job of meeting his expectations.
Do not start from a position that you are entitled to his enthusiasm. Do not accuse or argue. This is not a confrontation or a debate; this is a negotiation. You start by trying to gather information and understand your faculty member's perspective better. When you understand his logic and motivations, then you can try to brainstorm together ways that you can change your behavior in a way that benefits both of you (especially him). What you can offer is your willingness to devote your time and energy to work on problems that are relevant to his needs, and your flexibility and willingness to change. Maybe after having this kind of conversation, you will find some new project or new focus that both of you will be excited and enthusiastic about. Or, failing that, maybe you can find some compromise solution that is tolerable for both of you: maybe neither of you walk away exactly excited about the collaboration, but you can both live with it.
Your job at this point is to make the best of the situation. Realize that the resolution might not be perfect. Sometimes, that's life. You just have to roll with the punches and work with what you've got.
