I don't think you can or should "drop" the paper. To paraphrase the jewelry slogan, a publication is forever. You can't take it back just because you don't like the direction the journal has gone, or how people's opinions of the journal have shifted. Also, from a legal perspective, it's likely you signed a contract giving the journal the perpetual (and possibly exclusive) right to distribute your paper, and you might also have transferred copyright; so you can't legally prevent them from keeping your paper.
And indexing services are telling the truth when they list your paper as published there; you can't prevent them from sharing that fact.
One could argue a moral exception for a "predatory" journal which actively misled you about its standards and practices, in which case one could say your consent to publish was fraudulently obtained. Even in that case, though, I don't see what you can practically do, since the journal is unlikely to agree to drop your paper.
How it will affect your career is unknowable - it depends on how important this paper is to your career, the opinions of the people evaluating you, why the journal was blacklisted, etc.
I think all you can do is resolve to be more careful when choosing publication venues in the future.