In France, the most common starting date is September 1st by far. I do not think there is any legal problem with starting on another date. However, 1. you will find that the bureaucracy is very rigid, and it is possible that this adds some complications; 2. it's possible (and in fact almost certain) that the grant you get want requires you to start on September 1st.
But I'm pretty sure I know people who started their PhD on another date. The university allows PhD students to register until mid-December for a given year, usually. I think it is more common if your PhD is funded by a private company.
Also note that there is a weird interaction between the academic year (September to August) and the civilian year (January to December). Usually if you start your PhD in September, then you get 3 years (36 months), but you are allowed to extend this (unpaid!) to the end of the civilian year, totaling an extra four months. If you start your PhD on another date, I do not know how that works.
As for applying in late June (and note that today is technically June, but next Monday is July already...), it is pretty much a guarantee of failure, unless you are very lucky. My impression is that application season is end of May/middle of June, and results are basically right now (beginning of July). When it comes to such decisions, deadlines are strictly enforced usually.
But even before applying you need to find an advisor and start thinking on a research proposal, etc. I think the usual recommendation is to start looking in January when you want to start in September. If you can do your master's degree thesis with a prospective advisor it's a big plus, at least in math.
Maybe it is possible that you know someone with a big grant (e.g. an ERC grant) who can basically decide to hire a PhD student on the spot with no oversight, but other than that, it's too late for this year's batch of PhD grants.
PS: there is no such thing as a "summer semester" in France. There are two semesters (thus the name...), the first from September to December, and the second from mid-January to mid-April. The rest is filled with exams and holidays. But as a PhD "student" you are actually an employee, and you do not take courses. Semesters are irrelevant unless you teach.