There is no definitive answer.
I'm probably just looking to see what would you do if this were your situation.
My own personal experience (coming from the other side of the question): I recently accepted a tenure track position at a university. I'm very happy with the position.
During my visa application process for the new position, the consulate gave me a letter that the university had sent them as proof of employment. The letter contained details of the hiring process. There I learned that they had offered the position to someone from Oxford before me; someone older with half the publications and one-sixth the citations I had. The letter stated that he was first choice because he had a strong research profile and he was coming from a reputable university. I was coming from a strong department in a "provincial university" in a small country. The other person turned down the position. I accepted.
I was a bit stung by the letter because I read it as an implicit rejection of my background (over which I had little control).
Someone also told me later that in the board meetings, during the hiring process, there was a professor who raised concerns about where I was coming from ... a "provincial university". Apparently the more bureaucratic members of the board were my most fervent supporters: I had a lot of highly-cited publications, I was sure to bring a lot more, who cares where I came from? I wasn't told the full details, but I inferred that some of the more senior professors seemed to be more attracted to the Oxford thing than raw research metrics.
(Of course the hiring process was much more complicated than that; but this was the gist of the letter and the bits and pieces I heard afterwards.)
I'm not sure if that anecdote is useful to you but again there is no universal answer. It depends entirely on your situation.
I would say that yes, in many situations, the university you do your PostDoc in makes a difference for your future career. I don't know the US system well, but I guess it would be even stronger the case there.
But you and your personal publication record are far more important.
My general advise would simply be to pick the university where you feel you would be most productive (in a "healthy" way).