Is it possible that you misjudge why your advisor wants to see the other prof. on the paper? You say that the reason is that he was "declared as my second adviser on the project" ... but then you go on to explain that he actually was part of the discussions leading to the paper, so it does not, or at least not entirely, sound like a courtesy co-authorship to me.
Of course, you (and maybe even your advisor) do not value the input he had, but unilaterally throwing him off the paper for this reason is a dangerous road to go down. If he had any part in actually doing the research or writing the paper, I would at least talk to the guy first and ask him whether he thinks that he contributed enough to become an author.
However, prepare that there is a non-zero chance that the answer to this question will be "sure, I advised you regularly on the project since it's start, why wouldn't I be a co-author??" (and mean it!). Decide in advance how you will react to this. Note that, if the prof. feels this way, there is likely no way to remove him from the paper without him being annoyed or worse. Your task then becomes to decide whether you want to add him to the paper purely to keep relations intact (or not).