I contacted a professor for PhD study, and discussed the project. I would like to make a formal application: should I first ask would it be competitive?
You can ask us but this is a very difficult question to answer for faculty members.
On the one hand, we don't want to say "yes" and make you feel as though you've been guaranteed a spot when we have no idea of how competitive the pool is this year (see multiple questions/rants on academia.se that reflect just this kind of misinterpretation) -- nor do we want to say "no" and shut you down prematurely.
So you end up with a qualified "maybe" which doesn't help you (the student) much.
90% of the time you will get the professor to tell you that he/she doesn't really make the decisions (a committee does) and that the degree of competitiveness is always dependent on the particular applicant pool for that particular year. This is all true; it is also a gentle way to tell you that if you want to know, just apply.
Now in your case you've "discussed the project." Does that mean that he/she has already expressed that he/she would be willing to take you in to work on a particular project? If that's the case, I would assume he/she has already checked out your credentials?