If I were you I'd have the same concerns. In fact, I'm not like you, and I have the same concerns.
But don't confuse the people of Israel with its government. Certainly there are bad actors there, but I also know many Israelis who are more than decent people and oppose the government actions and attitudes that you worry about.
That said, you may be wise to avoid such a trip. But don't assume, necessarily, that the professor would disagree with your true reasons. There is no way to predict, of course, since opinion about Israeli-Palestinian issues come in "all shades of grey" in Israel.
If it is a question of meeting, rather than visiting the professor's institute, you might suggest meeting elsewhere, say at an international meeting.
To flip it a bit, there are quite a lot of people in the world unwilling to visit the US because of recent trends and attitudes here - and some that are not so recent. And I won't visit Mexico, for example, though I love the place, not because the government is bad, but because of the danger of drug gangs and killings and an ineffective government.
Note that I answered the question based on its original form which was tagged israel. However, the same concerns would apply in other contexts and there might be serious issues for some people visiting some countries. But my general advice still holds. Don't conflate the government of a place with its individual citizens. Some of them would support government actions and others would not. The world currently has far too much of this. I've actually known people who won't visit Germany because of its Nazi past, though, til recently at least, it has seemed completely redeemed.