“Why Female Professors Get Lower Ratings” (NPR) contains a true gem regarding an experiment conducted in France:
Overall, there was no correlation between students rating their instructors more highly and those students actually learning more.
[Note that it doesn’t seem as though the researchers, university teachers themselves, considered the hypothesis that university teachers have no consistent effect on student learning outcomes, which would certainly explain the above result.]
To answer your question with a question. What are students evaluating teachers for?
If you can’t answer the second question then the first is meaningless. The problem isn’t student evaluations, the problem is that students evaluate teachers based on their own (unknown) reasons. Did they find the teacher personable? were they hard to understand? did they get an unexpectedly good or bad grade (or expect to). I never rated teachers based on any objective teaching performance evaluation I had no knowledge of. Somehow, that seems to be the way administrators want to use them.
They are a useful qualitative evaluation from an important perspective. Certain trends can be usefully gleaned from them. Is a teacher consistently horrible, My guess is that bad teachers would shine through on both the test taking and the evaluations, but maybe not. It is also important to understand that the teacher is not the only input to the final exam scores, again a useful trend. Like any vector based datapoint, the point itself is oftentimes useless, it is only useful across time and space. Teacher evaluations should be treated like amazon reviews, sometimes very useful in avoiding a really bad product, but in and of themselves incapable of comparing two competitive products.
Certainly when I read amazon reviews, the quality of the review itself matters a great deal. I’ve read too many, “this product sucks, I needed a 1/4″ bolt, this is a 3/4 bolt” Then I scroll up to make sure I’m looking at the correct page, and it definitely is a review of a 3/4″ bolt on a 3/4″ bolt product page.
I learned the material regardless of the quality of my teacher. I had some teachers so awful that I would say that made the material more difficult to learn than simply reading the book, looking at you freshman calculus professor. In that case I had to work my tail off to learn the material, and go to other professors office hours to figure a couple of things out. I don’t think it changed my final exam score, only the amount of work I had to do to get that score.
Teachers who made it easy to learn, got higher rankings from me. I learned the material either way. That is my grading criteria, I answered that question in detail on most of my evaluations. Plus the ten or so forced ranking questions. I think I was OK at answering that question, how the administrators used my answers I can’t say. Is determining how easy a teacher makes it to learn, the ultimate question of quality in teaching? I have no idea. I certainly never stated that on an evaluation I simply described the qualities the teacher demonstrated that made it easy to learn.
I had one professor twice. a week before the final exam, he would post 8 questions and two of them would be on the final. He openly stated that you could simply type up the answer beforehand and drop it off, but he wouldn’t state which two would be on the exam. i.e. type up all 8, four times the work, or show up at the exam and only answer the two. Once I did the former, once the latter, similar grade each time. In the first case my final exam schedule would’ve pressed me for time, so it was easier to type the 8 and get it out of the way, in the latter I had more evenly spaced finals and opted to not do the extra work. My grade was similar in each class, the amount of work I had to do varied.