I assume you are creating your
CV for some
application (job or graduate studies). I understand that having three to four good referees is good enough. You need not provide more than that. I am particularly speaking about the academia in US and I believe it would be different elsewhere (e.g. UK).
Of course you may put different referees according to the requirement. I mean you may need to create different
CVs for different occasions. Some referees would be better able to acknowledge your work for a given situation. For example you may need to take a
recommendation letter from a (for example) Mathematics teacher when applying for
graduate studies in Math.
In fact I have also seen that many people do not put the references. In an application process, they are required to separately provide the names and contacts of the referees.