At face value, it seems clear that the issue is that you're trying to mix measurements of two very different kinds of performance. I think the REAL issue is deeper than this, but let's start with the two things you're mixing up:
- You want to measure outcomes. Did the group end up with "correct" results from the lab?
- You want to measure a specific (learning) process. Did the group learn from mistakes and questions?
The first measurement is usually the default in classroom settings because it's the most obvious. The second measurement has value, though: You're trying to prepare students for life, not just hand them a set of answers. Measuring their ability to learn, and encouraging them to be learners, is important.
That said, measuring outcomes has the advantage that it's not dependent on the process. It's easy to see if the lab result is correct or not. I don't think that's the focus of your question, so let's concentrate on the second concept - measuring the learning itself.
That brings us to the core issue with your proposed process: Measuring learning from mistakes and questions is very, very different from measuring learning in general. You hinted at this when you said,
For the majority of our students, there will be more than enough stuff in each lab to write about. However, this isn't universally true and some groups will do the lab just fine without asking any questions.
The issue here is that you've singled out a single type of learning to measure. The students who learned by failure and questioning are correctly being evaluated by your grading approach. This is good - you're encouraging that type of student to hone their skills. So let's look at the second group - the students who "did the lab just fine without asking any questions." The important thing to remember is that they weren't born knowing how to do the lab - they learned it too, just in a different way than the question-askers and mistake-makers. The reason why your new approach lacks fairness (as you identified yourself) is that it's only attributing value to one type of learning.
So - is it a bad idea to grade your students based on the questions they ask and the mistakes they make? If you're trying to grade them on their learning process, which is what I think you're trying to do, you need to make sure you're accounting for all types of learning not just the one style. This will be really challenging - people learn in multiple ways and transition between learning styles in a fluid manner, and many learning styles don't leave behind a "paper trail" you can measure after the fact. So let's get to your specific question:
How can I reward the students who learned from their mistakes without penalizing those who don't make any?
I may be frame-challenging your question, but if you allow me that liberty, I would rewrite it as:
How can I encourage students to be aware of, and work on improving, their learning skills, versus just rewarding them for getting the right answer to a specific assignment?
I think the answer to this will be much broader than a single stack exchange question, but some ideas to think about:
- Include a discussion/presentation during/after the lab where a selected group walks the other groups through how they approached the lab, with a focus on how they prepared or learned the material beforehand and how they dealt with issues/challenges.
- When a group asks questions during the lab, let the other lab groups answer it (instead of you). When an answer is given, have the answering group provide explanation of where they got the answer from.
- Include an entire lab session, or elements of a lab, that are not part of the graded solution and are deliberately "unanswerable" in that they cover material you haven't taught or are otherwise deliberately difficult. Have the students come up with their own way to resolve this. This could even be a lab where the process intentionally causes failure, which would "force" the students to use your "learning notebook" as a tool to document and explain the failure and their approach to solving it. Purposefully cause them to go off-script and respond to the failure with creativity.
Basically, you need something that accounts for multiple learning styles, and something that encourages accountability instead of just allowing students to "cheat the system" by pencil-whipping busywork in order to get a grade. Notice I left your word choice of "reward" out of my re-worded version of your question - I think awareness and demonstration are probably more important than a reward - at the least because the reward just causes students to focus on the end, rather than the means, which totally dodges your goal of getting them to focus on learning skills versus just a final result.
Edited to add:
Let me further add: I don't think the "learning notebook" is a bad idea. It actually sounds like a great way to encourage people to recognize failure and accept it as part of a (positive) learning process, versus being afraid of failure. It would be great to include along with my three suggestions above. That said, I do think it becomes cumbersome to try to include it as a graded item, and more importantly, regardless of whether you grade it or not, focusing on it (as a graded item) misses the larger opportunity to consider and encourage other learning styles.