I suspect Sean Carroll is emphasizing this more than most people would, thanks to his tenure denial. It's not always important - plenty of serious bloggers have received tenure, and I know of one mathematician who published a novel before successfully getting tenure at a serious research university. However, Carroll's right about the risks: it will hurt your chances if people spend too much time comparing what you did accomplish to what they imagine you might have accomplished if you had focused more. If your hobbies could be viewed as taking time away from research, then it gives anyone who doesn't want you to get tenure an excellent opportunity to try to derail your case.
So basically, if your tenure case could be viewed as marginal or you might have enemies in the department (or among your letter writers), then you should worry about this. Otherwise, I wouldn't let it dictate your life, but I guess it depends on your tolerance for risk. One common-sense approach is not to go around emphasizing to colleagues how much time you are spending on hobbies.