I just discovered some mistake after the online publication? There are two mistakes- instead of figure S4A (supplementary figure), it says Figure S3A at two places. It's very embarrassing, should I write to the editor about this?
This sort of typographical error should be handled with a "Correction". You'd say something like "In the article such-and-such (citation) the text on line x, page y should read "Figure 4A" instead of "Figure 3A". The online text has been corrected."
The details of how that's done depends on the journal, so you should discuss with the editor.
It's quite common to have to issue corrections, it's not a big deal, and aside from being a minor embarrassment an occasional Correction doesn't reflect on you in any way. I suppose if it were a regular thing it might get noticed, but this should not be a big deal.
You should inform your coauthors, use this opportunity to search for other misprints and finally write to the editor.