The expression your field might be related to my field is an expression of interest.
How many interests do you have? I hope plenty. Do you have time to pursue all of them? I am sure not.
General rule: no one has free time or free money to give out.
Namely the other person does not always invite me to work on a joint
project unifying the two fields.
Well, the few cases that they invited you to work on a joint project are the best opening to future collaborations that you can achieve. Spend some time (ideally as much as you want, practically as much as you have) to investigate if something meaningful can come out from the collaboration.
It seems to me that you are expecting that everyone has infinite time to collaborate with "just met" random people working in a maybe related field.
Play down your expectations, keep the door open to collaborations, always interpret expression of interest as a "broad" expression of interest and not as a promise of collaborations.