For example, let us consider that I write a manuscript which is to be submitted to one of the prestigious journals such as PRL, PRB, etc. In this manuscript, is it OK to cite from some journals which are generally considered as not-so-prestigious journals? If I cite more papers from low impact factor journals, does it play a negative role?
A low impact may not say a lot about a journal. In my field, one of the lowest ranking journals is in fact the best journal for my work. Because of the extreme narrow scope, it has a low impact, because its readership is restricted. However, like its readership, the journal's referee pool consists solely out of dedicated experts in this exact field. In turn, throwing your work in there is like throwing it into a lions den. If it survives, the paper is definitely strong and worth surviving.
In other words, when an important argument in your manuscript all hangs on that one citation from a low-ranking journal, you may want to familiarize yourself with that journal, before labeling it as a “not-so-prestigious journal”. Judge and see if it is a valid paper. Look at the author list. Authors often say more than the journal. Research groups can be prestigious too, universities can be too, single authors can be too.
In general, when it's just a side-note in your MS that needs a reference, I would definitely not leave out work based on IF only, and in fact not even bother with all this.
The impact factor (IF) is a metric of citations. If enough manuscripts from a low-IF journal get cited more often, the journal's IF will improve.
In general, you should consider any manuscript independent of the journal it got published in. However, there is one caveat: If you suspect that a journal doesn't follow a good review process (such as predatory journals), you should examine the manuscript with extra caution. Furthermore, there is usually some correlation between the quality of a manuscript and the work described therein and the quality of the journal it got published in.
As a reviewer, I have no problem with a few citations from low-IF journals. However, I will be very critical if there are no citations from mid- to high-IF journals because that is an indication that your work might not be very relevant or innovative.
Your own work is not based on the impact factor of your cited work. The quality of your argumentation and thesis you discuss in a paper mainly builds on a coherent discussion of the state of research, independent from where it is published. Only referencing high-impact factor journals indicates that you only did a shallow literature search in the more well-known journals, which is not very professional in my opinion.
Academic research is based on citing other relevant work that you used or inspired your work. Omitting those sources is a practise that should be very strongly discouraged, and you could be accused of plagiarism. There may be reasons why that work was published in a less prestigious journal, despite its relevance (or not), for example having been written by a student in a less prestigious institution who is not yet known, or who is publishing work that may be controversial (but not necessarily invalid). As the authour of your article, you should view yourself as the "expert", and be able to ascertain the validity of the papers you are referencing, regardless of where they were published in. If you are using some of their results, or referring to views contained therein, you have an obligation to referencing them.
You need to be confident in your work, which should be evaluated only by its content, and not by the other articles it is referencing.
Impact factor (IF) is a quite debated metric for journal quality. But in no way it addresses the quality of a single paper in that journal, and especially the quality regarding your needs. So you can cite this paper, as long as it effectively contribute to the work your are performing.
Beware though, since some journals tend to increase their IF, some editors may favor papers that cite a lot (or a sufficient quantity) of papers from the same journal, or even from the same society or publisher group. It is unfair un general, yet this exists.
The number of citations received by the paper can sometimes help balancing the IF of the journal it was published in.
Finally, aside from the IF, which is not relative to the field (some fields tend to have weaker IF than others), the journal search at Scimago provides a ranking of journals in different fields.