This might be field dependent, and it is good to follow your advisor's advice as they know the field better than you (or I) do. But I find it a bit strange.
The one thing that would seem to support the statement is that, having a firm date for publication, there are probably fewer review-rewrite cycles for special issues so that weak or marginal papers don't get improved much. But, they are also much more likely to get summary rejection early.
If a paper is good, then the venue in which it is published doesn't change the quality of the paper. It is what it is.
Moreover, in some fields, special issues have special editors that are top-level academics in the theme of the issue. They are in a good position to know what is important/interesting/novel in that field.
Frankly, I'd prefer a special issue for my own work since it is more likely to be seen by those who care the most, since a special issue is normally theme restricted.
But, your advisor is probably referring to a "perception" problem that they (and maybe others) have with special issues. If we "think" the papers are lesser quality then we won't consider them important, independent of the actual quality.