This is a challenge that I know we face from our end as a graduate program: the supervisor will be extremely important for the student, but not until they actually start their dissertation process.
So yes, regular meetings are key. You should talk about:
- your courses: progress making, anything interesting that you're working on as part of the coursework;
- any TA/RA duties: your supervisor might be able to find funding for you (field and university dependent), but can often serve as a sounding board around issues like balancing teaching and coursework, etc.
- comprehensive exams: if applicable, your supervisor will want to make sure you suceed in comprehensive exams.
- yourself: this depends on the supervisor, but we often want to know who are students are. How are you doing? Especially during COVID, your supervisor will probably want to know how you're holding up. You can make a bit of small talk and get to know each other.
This is important for building that relationship, and also for your supervisor to get to know you. What's working well for you? What successes have you found in coursework? It may come time for scholarship rankings and opportunities, and you want a supervisor who can make the case for you if they happen to be at those tables or have other opportunities.
All supervisors are different: some will want regular meetings, some less so, but you'll need to discover the shape of what will hopefully be a long and productive relationship to come.