The important thing to recognise here is that they're comparing you to the norm (i.e. themselves). Asking "why do you do so much, relatively?" is equivalent to asking "why do you do so much more than me", and to "why do I do so little, relatively?". Consider your answers in this light.
You may have felt that saying "I work hard" is a modest answer to the question, but it really isn't. Almost anyone can "work hard", it's perceived almost as a moral choice. Perhaps it shouldn't be, but hard work is a virtue and you're saying you possess much more of it than they do. The janitor works hard, and it doesn't seem to have done his publication record any good, so your answer lacks insight. Which is not unexpected in a three-word answer ;-)
Imagine if someone asked Terence Tao "how come you've published so much, achieved so highly, and won the Fields medal?", and he said "the only reason is that I work harder than you do". No, it's not only because of that, it's also (indeed as a prerequisite) because he's way better than me at what he does. That was true before he put in the hours and the effort. There are people who've put in just as much hard work as he has and achieved less (albeit, I'm not one of them). So for him to claim that the only difference between himself and me is that I work less hard would be an insult. I can accept that he works harder, but I cannot accept that the only reason I'm not a Fields medallist is lack of effort on my part, and it's rude to tell me that it is. If he's going to pick one reason then I would far rather he picked "because I find mathematics much, much easier than you do". But, for what it's worth, here's what Terence Tao actually says instead of saying "I work hard".
Now, your colleagues might not be ready to admit that you're smarter than them (indeed, you might not be, there might be some other reasons), so you're right to be cautious about claims to your own ability. But avoiding a claim to your own ability by making a claim to your own virtue isn't helping matters.
If your colleagues genuinely want an answer to their question, because they're half-wondering whether you have some silver bullet that they could benefit from, then you need to look more closely at yourself and find more detailed answers. As others have said, it's just not plausible that you do 4-13 times as many hours a week of work than any of them does (although, with different teaching commitments and working hours and focus levels it's certainly plausible that you do 4+ times as many hours a week of proper research work than some of them). Find what else.
This isn't necessarily easy, because you don't know why they only produce 1-3 papers a year, but try to figure that out. These people will be open to the possibility that you're better than they are, but they can't believe it's just that you do 4+ times as much work, and if there's anything else you do that they could in theory do too then they want to know about it. For example, if you've found some way to avoid "overheads" that cut into everyone else's research time, then perhaps you think to yourself, "I'm just concentrating more on my work then them, i.e. working harder". But you can do better than that, you can help them remove some of the barriers to them working harder on their research in the hours available.
If, after a full investigation, you literally cannot find any reason other than hard work, then you might just have to say, "I've thought seriously about this, the typical postdoc in this department after all overheads and distractions and everything, only actually manages to sit down to about 20 hours of concentrated work on their research a week. I'm incredibly unusual, I seem to have more stamina than most and more ways of avoiding the rest of the world, and I manage to sit down to 80 hours of uninterrupted research work a week. That's why I do 4 times as much research as the average". That's a much more satisfactory answer, for them, than "I work harder".
If your colleagues are just looking to diminish your achievements due to their own insecurities, then as you've identified they want to come away with some conclusion like, "his papers aren't high quality", "his co-authors do all the work", "he's struck a rich, easy vein of work and is rattling through it spinning off a paper for every result". You have no obligation to pander to this nonsense, but for your own sake don't compound it by implying that they could do the same as you just by working harder. You could perhaps prepare some genuinely modest answer, in the hope of achieving the best case outcome, which is that they come away thinking, "well, he's a nice chap, and he made me feel good about myself, lucky for him he's getting so many papers out of his work". I can't give you one, though, because it has to relate to your actual work processes as compared with theirs, and it has to convince them that they aren't doing anything wrong, so that they can feel secure in their work. It could perhaps start from, "what can I say, editors seem to really like what I'm doing, I think perhaps it's because X, and I can polish off the papers really quickly because Y". When mollifying the insecure, make yourself sound blessed, don't make yourself sound like you somehow earned and deserve 4-13 times the public recognition they're getting.
Hiring panels want you to be excellent and work for them, so your approach to those can be completely different. It's not a necessary politeness to be modest at interview. If they ask a question like this then their fear is that you've produced 10-13 mediocre papers per year but nothing great, in which case they wouldn't want to hire you. They want you to allay this fear. You need to justify to them the value of your work, so that they're confident that you've produced the 1-3 solid papers they expect, and more. You need to justify it in more detail than "well, a top tier journal accepted it", or "I have a high citation count", which means talking qualitatively about your impact on your field and on the work of others. You also may need to convince them that you somewhat understand your own process and success, because they're running a department of interacting individuals, not a paper-factory. If they're going to hire you for tenure-track then they want to know that you'll benefit the department, as well as generating research. They won't turn you down for producing too much high-quality work, but they might turn you down if they think the good work you're doing would be more than cancelled out by the negative effect on department morale of you going around telling everyone the secret to success is to put the effort in.