Suppose I want to get letter of recommendation for a PhD program. Could this letter be from someone not from my institution? Could is be from someone I have collaborated with online? Or generally do schools want recommendations from professors from a student's home institution?
A letter of recommendation absolutely does not have to be from someone at your university. It should be from someone you have collaborated with. In general, a letter of recommendation will be most valuable if:
It can speak to your potential as a researcher: letters from supervisors of research projects are much more valuable than letters from people who have only had you in class
It is written by someone well known in the field that you are attempting to enter. When this person says you have potential to be a researcher, it is much more credible if they have experience working with excellent researchers in the field.
It is written by someone who knows you well. They should be able to point to specific things you have done with them, specific conversations you have had, specific ideas you have had.
You should pick your letter writers by seeing how well they optimize these three properties. It won't matter if they are at your home institution or not. The only reason not to get a letter from "someone you have collaborated with online" is if they don't satisfy these criteria. In particular, I might be concerned that they don't know you well enough if you have never met them.
That being said, it might raise a red flag if none of your letter writers are from your home institution. But if you have one of these, that should be enough.
Schools want a letter that demonstrates that you are a smart, hardworking individual, whose presence will be a net gain for the program specifically and the university generally. In most cases, the people most able to write such a letter are from the institution where you received your undergraduate (or masters, as the case may be) degree. These people have interacted with you in a classroom (and possibly research) setting, and they know you well enough to write a good letter.
To that extent, if there is another individual—with stature enough to be recognized by an academic institution (i.e., a professor or recognized researcher)—who can write such a letter, that's fine.