Dysfunction in education
In measurement, dysfunction is what happens when, by measuring a certain aspect, you degrade the overall performance of your process. While this kind of problem is not likely to occur when you perform measurements by your own will, dysfunction has been observed in many occasions when measurements are linked to evaluation. When salary is bound to a performance indicator, employees might be tempted to trick the measurement to get better ranking. Even out of the business/engineering context, think about all that blog spam to get better ranking on Google. That’s dysfunction.
When you think about it, education has a lot of dysfunction problems. Exams are just a way to measure the comprehension, and the measurement result is bound to evaluation and ranking. The problem with this dysfunction is not about the students cheating, it’s with the exams themselves. For disciplines where truth exists, such as mathematics, evaluations make sense. Either you have the right answer or you don’t. The problem is that there are very few disciplines where the evaluation can be 100% objective. Yet, all of them use a similar measurement system to assign grades.
Software engineering is all about subjective answers. Of course, I’m not considering an assignment where a piece of code has to be written. Two types of evaluations can be made.
- Open questions where any answer can be right depending on the context it’s taken.
- Narrow questions where only those who spent time memorizing lists of elements can succeed.
Both cases have their own dysfunctions. In the first one, the results will be entirely subjective. The one correcting is simply not allowed to give 100% to everyone. The lucky ones will either be those with an opinion matching the correctors opinion, or those including as many concepts as possible into the text. Getting the same opinion as the teacher is fairly easy. All you have to do is attend to the classes and repeat everything said when the evaluation moment arrives. Including as many concepts as possible generates a solution full of buzz words and no real structure. Sadly, those really thinking about an acceptable solution rarely fit in the lucky group.
I once saw a teacher who attempted to evaluate open questions based on if it made any sense. It was a complete disaster. When a subjective criteria is evaluated, the are no evaluation grids, no way to prove it was fair. As a result, lower grades complain until they are actually part of the higher ones, or until everyone has the exact same grade (you can actually hear those same people complaining later about the grades being too close).
On the other hand, narrow questions can’t evaluate comprehension at all. It does not even require attending to a course. All you need to do is take a full day to memorize everything highlighted in the course slides.
These are examples of dysfunction. I guess anyone who studied long enough can recognize some of these student types. The worst part is that if you want to compete against them, you actually have to play with those rules. In the end, what do you get out of it? Engineers that can’t think by themselves, copy solutions straight from books. I once heard a complaint about modern electrical engineers (or what is mechanical?) not being able to solve problems anymore. All they would do is perform calculations. Looking at the state of dysfunction in education, this is not surprising.
There is no way I can think of evaluations can be removed from education. Without evaluations, you would get a diploma after sitting in a classroom for 4 years, with no guarantee at all you learnt anything (which is not really different from the current situation, but that’s not the point). I think the solution resides in applying the engineering concepts in evaluation as well. Open questions are the only way to verify global understanding of the concepts. Knowing the details is not as important as knowing where to find them. To make open questions successful, a few steps have to be taken.
- Inspect evaluation material before handing it out to students.
- Test the evaluation on a sample group of students.
- Use redundant correction on critical evaluations.
- Request independent validation of the results before releasing the scores.
Important artifacts part of a software projects should be inspected to make sure they are correct. In the same way, evaluation artifacts should be inspected to ensure that no ambiguities remain, or as few as possible, remain in the final products. I’ve seen situations where the primary discussion leaving an exam was “What was question #3 all about?”. You could ask as many people as you wanted, you would get a different interpretation. This is actually when the question appears to be clear enough so that no one complains about it during the exam. I got to cross a few unreadable questions as well, containing typos, duplicate words, missing punctuation, …
The best usability test is always to get a real user to sit and try it out. Why not do it for evaluations as well? It could allow to remove the questions that are too easy, those that are completely impossible to answer and get a well balanced exam that can be challenging.
When the nature of the exam is subjective, having a redundant correction can help raising the confidence level of the final score. For the students, it’s an indication that the result is not based on a single person’s opinion. For the correctors, it’s a protection against those that would complain to get their grades up. In a least-critical evaluation, having an independent auditor to double check 10-20% of the copies could be enough.
Of course, it takes time, and most professors probably don’t like the idea of stepping away from their research projects to worry about the 1st cycle students, but if they ever want the result of their studies to be applied, they will need to make the students believe in their discipline. I don’t know of many students who applied many of the “state of the art” concepts exposed in the classrooms during their internships (except maybe those related to design). Providing quality evaluation that force to understand, and promotes understanding, instead of memorizing might do a difference in the behaviour of software engineers in the industry.