The concept of "doing enough work" has to be measured against some standard. Are you aiming to win the Turing Prize? Then you probably aren't doing enough work. Are you doing enough for your advisor to say that you are? Then by your own words you are.
So realistically, you have to determine at what standard to compare yourself to. There are two broad categories in which to measure: how much you need to do to advance professionally at some level (e.g. obtain a post-doc, get an assistant professorship, etc.), and how much you need to do in order to satisfy your self-esteem.
The first is primarily concerned with the materiality of living. Find a job with health benefits and be able to pay a mortgage. Whether or not you are focused on a tenure track job, you should really have a backup plan semi-worked out for if you leave academia. This can take some pressure off. For a job in academia, get an idea of what recent PhDs did to graduate and linearly interpolate for where you are at.
The second is trickier. Steeped in academia is the currency of prestige and recognition. One of its appeals is to be a revered expert in your field. Trading complements like "brilliant" or "genius" are something peculiar to academia. I would suggest you evaluate how important recognition is to you and how much of your anxiety on the amount of work you do is tied to envy of the success of others.
Again, to emphasize, both of these are value judgments that only you can make. Personally, I found I was too taken in by the prestige culture in academia which contributed to my health issues that required me to leave.
Finally, and this is something I cannot contribute to that much, is the question of efficiency. If you could be doing more work without changing much more than your approach, then in that sense you aren't doing enough. I.e., if you spend too much time on irrelevant details.
Most of the answers here have addressed this last issue. As such, my interest was primarily in calling to your attention my first two points.