If the manuscript was processed through some online system, then in fact "withdrawing the paper" should be an option available to you at any time. If you access the system, click on that, wait a suitable amount of time, the paper should indeed be withdrawn.
If you had direct correspondence with an editor, you should email the last editor you corresponded with and say "I would like to withdraw paper XXXX [giving enough information to clearly identify it] from journal YYYY [ditto]." You might want to give a brief explanation of the reasons for a withdrawal and/or discussion of your future plans with this paper, but that is probably mostly a face-saving device. In most cases, it should not be strictly necessary.
If you both dealt with an editor and dealt with an online system, then it would be safest to do both of the above.
By the way, this is not at all an unusual action to take. You are not going to get an argument or have to defend yourself. Just do it.