The advice I got was not to publish in any journal that I don't regularly cite articles from. I don't think it really matters what the stats say - you presumably know the field you are in: if it's a journal that often publishes papers you find good, then it's going to be fine. (You could also withdraw the paper if it hasn't been accepted yet.)
Another wise piece of advice I was given was that it's not worth publishing things just to be published. At least in my humanities field, it's better to take the time to keep working on something until it's ready for the best possible journal rather than hurry it out. People know how long good research takes, so they won't judge you any worse for having fewer papers if all your papers are amazing - don't get tempted by quick turnaround promises to place your papers in lower tier journals but place your work in the best venues possible. That said, some top tier journals can be very efficient (it all depends on the field and the editors).
It's also surely going to depend on what other papers you have out - if you only publish in second-tier journals, that's going to give off a certain impression. If this is the only one out of a handful of papers that are in top-tier journals, noone will care. It also matters more at the beginning of your career.