If you have well regarded publications and some demonstration of teaching experience at the collegiate level, your age may not be of significant concern to hiring committees. If I was a 45 year old on the hiring committee, is there really that much of a difference to me between a 23 year old and a 27 year old? (Okay, yes, there could be a difference. But how much? Some of that difference would amount to life experiences like graduating college, experiences you already will have attained, just at an earlier date).
Long story short, hiring committees are used to hiring new colleagues that are "young" and fresh. Being 23, 24 years old is likely not a point of concern.
I will add this: Anecdotally, many of the people I have run into in academia that are very young for their academic age (e.g. 19 year olds getting a PhD) have not gained the soft skills necessary for actually being a college professor. You can have a PhD and still have no clue how to deal with the guy who's 5 years your senior being a discipline issue in class. I know that when I was a graduate student in my early 20s and teaching classes, some students took advantage of my being very young. This is not to say this is your case. As long as you can demonstrate that you can be a mature instructor of a college class (letters of recommend and teaching ratings can speak to that), you will be fine. (And frankly, at 23, 24 years of age, you should be fine). Being a mature teacher is something that every hiring committee will look for anyway.