I assume you are creating your CV for some application (job or graduate studies). I understand that having three to four good referees is good enough. You need not provide more than that. I am particularly speaking about the academia in US and I believe it would be different elsewhere (e.g. UK).
Of course you may put different referees according to the requirement. I mean you may need to create different CVs for different occasions. Some referees would be better able to acknowledge your work for a given situation. For example you may need to take a recommendation letter from a (for example) Mathematics teacher when applying for graduate studies in Math.
In fact I have also seen that many people do not put the references. In an application process, they are required to separately provide the names and contacts of the referees.