A low impact may not say a lot about a journal. In my field, one of the lowest ranking journals is in fact the best journal for my work. Because of the extreme narrow scope, it has a low impact, because it'sits readership is restricted. However, like its readership, the journal's referee pool consists solely out of dedicated experts in this exact field. In turn, throwing your work in there is like throwing it into a lions den. If it survives, the paper is definitely strong and worth surviving.
In other words, when an important argument in your manuscript all hangs on that one citation from a low-ranking journal, you may want to familiarize yourself with that journal, before labeling it as a “not-so-prestigious journal”. Judge and see if it is a valid paper. Look at the author list. Authors often say more than the journal. Research groups can be prestigious too, universities can be too, single authors can be too.
In general, when it's just a side-note in your MS that needs a reference, I would definitely not leave out work based on IF only, and in fact not even bother with all this.