advantages and disadvantages of white-box testing of students assignments?

Question

This is a fork of my question homework-testing-method-for-schools but i'm taking it in another direction.

I was asking about a homework testing method that'll encourage students to write better code and reduce the TA's proofreading task to a minimum.

I was thinking of using unit testing, (or anything else you can think of), in order to turn students in the right direction (design wise), and hopefully giving them good thinking and coding habits (we're talking first programming course in collage).

so my new question is:

1. If you think there's something wrong and not educational about unit testing students code please explain why do you think so..

2. If you think this is a good (or just an o.k) idea please tell me if you have any ideas of how to design the tests.

thank you for your answers.

Answer 1

A lot of what I said as an answer to this question is probably also applicable here, so I will not repeat it.

If you think there's something wrong and not educational about unit testing students code please explain why do you think so..

I don't think there is anything particularly wrong about unit testing student code, but it will likely not do away with manual grading or direct lab sessions entirely. All things considered, the way how TAs traditionally check programming assignments (build, run, check behavior against assignment description) is pretty close to what a unit testing framework does anyway.

If you think this is a good (or just an o.k) idea please tell me if you have any ideas of how to design the tests.

The major challenge I see is that you want your tests not only to serve as a framework to check whether students implemented the right functionality, but also to check whether they did it in the right way. I think unit tests and generally automated grading will not be able to help you here, aside from limited hard-coded rule checking.

A lot of the problem is already encoded in your use of terminology. You say you want "unit tests that do white box testing". This isn't really feasible, IMHO. Unit tests test interfaces. So, to cut it short, I do not think that unit tests will help you in this regard.

Answer 2

As a tester I would suggest giving a couple lessons on testing itself, its techniques and TEST DESIGN. The latter is useful to write proper set of tests. Anyway, teaching them good habits of writing tests for their code is a huge plus. As an advanced task for one of the latest your lessons I would suggest you to switch the roles with your students: tell them you are going to write a class with given interface and defined behaviour, and their task then would be to write unit tests that would 1. cover the described behavior and 2. find some bugs (probably, left in the code by your intention). That would teach them writing test based on the desired behavior, not the implementation.

Answer 3

I have been involved with two classes that used automated testing in different ways, and in both cases is was an excellent idea that was generally well received by students.

One was a class on compilers, which naturally encouraged the use of regression-testing because most assignments were to write a piece of software with specific functionality. In this class, every assignment had two sets of detailed regression tests: one was provided with the assignment to be used during development. The other was kept secret and revealed only when the assignment was returned. Grades were a mixture of the two sets of results: the first batch was "easy points" that everybody was expected to get, the second set was the real differentiator demonstrating that a student's solution was sufficiently general and deep.

The other class was a large (300+ student) artificial intelligence class, where automated testing of homework was introduced in order to lighten the grading load and allow the TAs to spend time on small-group tutorials instead. In this class, all of the homework was done through the automated system, including both coding and non-coding questions (e.g., numerical calculations, multiple choice), and students could submit their answers, check if they were correct, and resubmit again and again until their got it correct. As such, homework was viewed as "required practice" and everybody was expected to eventually get all of the questions correct, though we didn't care when. The grades for the class then came almost entirely from quizzes, exams, and projects, with the homework percentage used as a multiplier on the total (well, technically it was multiplication of a complex formula that essentially amounted to: "If you blatantly ignore the homework, we'll drop you a letter grade").

So in sum: automated testing can be an excellent solution and I think more classes should adopt it. It requires a bit more up-front investment, but can pay off both in terms of time and in terms of pedagogical value. How exactly you design it and integrate it depends on the goals, as illustrated by these two examples, and I'm sure there are many more.