Analysis: Evaluations and their discontents
Photo: Afton Halloran
Lecturer Leopold Galicki thinks that course evaluations do not give clear enough representation of course quality
In the last 20 years, evaluation culture has become part of many institutional areas of life, especially in the education sector. Evaluation committees were well-known long before the 1990s. In the private sector, consumer surveys on products and services have existed for decades.
What is new is that, within the institutional world, there has been a greater involvement of consumers of welfare state services as evaluators. Of course, universities have also been influenced by this evaluation trend.
It would be interesting to look at this evaluation trend in the welfare state based on a larger study of many institutions. But in this context I confine myself to share my experience from one limited area: The student course evaluation.
Course evaluation not enough
The student course evaluation is a formalized element in a course at the Department of Sociology at the University of Copenhagen, where I teach.
In general, I find it beneficial to students that they, in the final stage of the course, get distributed questionnaires and have 15-20 minutes to reflect on, and address, a number of concerns regarding the quality of education.
The teacher gets an opportunity to confirm, or rule out, the eligibility of the course, as seen from the students' perspective. As a teacher, you get feedback on a number of specific indicators on the quality of the course, like for example the use of learning tools such as whiteboard and power point, guest lectures, and on student involvement.
At the same time, students can relate to the academic level, and the teaching and learning. The evaluation questionnaire contains open questions on learning methods and on other negative and positive aspects of the course. As far as English-spoken courses, the teacher's and students ' English proficiency may be evaluated too.
The questionnaire's anonymity ensures that students can expand their criticism without fear that it will negatively influence their grades, or give them an image as a student who complains.
However, it is precisely this anonymity, that I am concerned with.
Anonymity leads to wild variation
Apart from positive aspects of anonymity, there is a disturbing part of it.
There is always a small segment of course participants, maybe around 5-10 per cent, or one or two out of every 20 students, whose evaluation is what I would call 'totally negative'. In the questionnaire this can be seen when course participants respond with a 'no' to the question whether the aim of the course has been fulfilled.
There can be several reasons for a strong critique: Psychological and socio-psychological factors can influence the evaluation - I am thinking here of the mental state of the individual at the time of the evaluation and the student's welfare within the social group, including the teacher, which are all constitutive factors of the course.
The anonymity of the questionnaire can serve as an inviting platform to unfold the student's frustration.
Students always have more or less different backgrounds to evaluate a particular course, This can lead to wildly varying evaluations. A folk saying has it, that no matter what, there is always someone who is dissatisfied. So what is the point of writing about it?
Inconsistent evaluation
The reason I write this article, is because the one or two strongly dissatisfied students in a group of 20, will usually reveal an inconsistency in their evaluation.
Not so much due to the respondents' mental state or their unfortunate welfare in the group. The persons demonstrate their inability to relate to, and logically evaluate the study/work situation in which they participate.
From the answers, it is as if there are black holes in the student's ability to observe, consider, and absorb course content, and the teaching methods used. The result is that the student's answers are directly contrary to the facts.
Let us take an example from one of my own courses: On the question of the involvement of the text readings in the teaching situation. The question was: Has it been too much, appropriate or too little?
The respondent responds: Too little. But in the specific course this response is impossible. The whole lecture series was based on PowerPoints containing representations and interpretations of passages from the texts this lecture is referring to.
There are other examples of a student who only read 20-30 percent of the syllabus, but evaluated most indicators on course quality negatively.
If a strong negative evaluation is only expressed by ticking off a closed question, it is difficult to see the lack of consistency. But the complete lack of sense in the evaluation is revealed when a student responds to open questions.
Often, the answers to the open questions directly contradict the answers given as ticks in the boxes in the closed questions.
Integrity - in the educational context and in general
But if it is all about one or two participants among 20, is it not just much ado about nothing?
It may be about 5-10 percent of the participants, but it is a pattern. It is worthwhile to articulate these marginal groups.
This is not just to avoid possible irrational elements, which might influence the evaluation of a given situation or a process. It is also about recognising that irrationality and emotion can be found in a university context.
With this realisation, we may be able to prevent this irrationality from affecting learning and teaching processes. Remember: These 5 to 10 percent, who exhibit the lack of skills to consistently evaluate and who unfold emotionally irrational behaviour, are aspiring to be the academic power in our society.
The pattern they demonstrate in their assessments should therefore not be made light of.
Anonymity is, for better or for worse, an important prerequisite in many evaluation contexts, and also when it comes to the assessment of the quality of a university course. But to relate constructively to the learning process means to practice criticism, which is not only important in scientific contexts, but in a democratic society in general. The autonomy of future academics depends precisely on the ability to demonstrate a critical approach in public contexts.
Evaluation culture provides an opportunity where students can demonstrate independent critical thinking about specific, but fundamental, things.
My experience is, that there is a small, but always existing group using the shield of anonymity to exercise an inconsistent criticism which falls apart after a more detailed analysis.
The relatively few, but resistant cases of criticism lacking rationality should, I believe, be highlighted and discussed after the evaluation situation.
Anonymity should not be violated. But the vast majority of constructive evaluators and the small minority of inconsistent evaluators can benefit from the exposure to irrational assessments and get something of thought-provoking substance. They can get to see the importance of consistency and integrity in a critical approach to all the various institutional contexts.
Stay in the know about news and events happening in Copenhagen by signing up for the University Post’s weekly newsletter here.
10 comments
It is downright perfidious to posit unprovable remarks as to the internal processes of another person, just because they say something you do not like to hear or may not understand where is comming from.
I mean, lemme get this straight: If I were to roundly critisize your course, your response is to immediately attack my mental well being, my work ethics, and my logic skills?!
It is your kind of lecturer that gives higher education a bad rep. Aloof and superior and also not really worth the effort to take down a notch or two.
Sad, really.
Michael Barner-Rasmussen,
din kommentar er blot endnu et eksempel på, at der findes mennesker der har akademiske ambitioner, som alligevel ikke er i stand til at diskutere et problem på et akademisk niveau, men blot kaste mudder i hovedet på andre mennesker, på grænsen til det injuriende.
Ifølge dig er jeg er perfid, fordi jeg tager nogle eksempler på kritik i en universitær undervisningsmæssig sammenhæng op til en diskussion, og jeg mener helt og holdent at kunne argumentere at det handler om en ulogisk, simpelthen hul i hovedet kritik!
Til trods for, at det både i mit debatoplæg samt mine efterfølgende kommentarer til læsernes reaktioner, står klart, at anonymiteten på ingen måde krænkes ved at omtale eksemplerne på en uacceptabel kritik, så fremstiller du mig som én, der kunne finde på ”to attack your mental well being, your work ethic and your logic skills”
Mens jeg på ingen måde har villet eller kunnet skade en eneste af mine studerendes personlige integritet, for, som sagt, henholdt jeg mig til anonymitetens ukrænkelighed, så vil jeg i relation til din kommentar, der er forsynet med dit navn, svare: din kritik er et slag under bæltestedet og røber, at du ikke evner at læse andres kritik og forholde dig til den på et sagligt niveau. Det er himmelråbende uakademisk.
I much agree to the two comments below. Personally I think it occurred 2 or 3 times where my evaluation was very negative compared to a generally positive evaluation from the rest of the class. I would not say, as suggested, that this was due to my mental state, I was actually well awake, not stressed and not under the influence of any narcotics. The reason for my negative evaluation was that, using my different background and point of reference, I though the course was sub-standard. At the same time the rest of the class apparently had lower expectations, or where not used to a better standard.
A real good teacher would have a feel for the individual needs of each student; after all, education is not mass production.
Then the two arguments surprise me as they do not support the point the author likes to make.
Example 1:
''Let us take an example from one of my own courses: On the question of the involvement of the text readings in the teaching situation. The question was: Has it been too much, appropriate or too little?
The respondent responds: Too little. But in the specific course this response is impossible. The whole lecture series was based on PowerPoints containing representations and interpretations of passages from the texts this lecture is referring to.''
---
Could this not indicate that the student is dissatisfied with the fact that the whole course is based on PowerPoint? No reading can definitely be perceived as ‘too little reading’.
Example 2
''There are other examples of a student who only read 20-30 percent of the syllabus, but evaluated most indicators on course quality negatively.''
---
Could this not be a perfect cause-effect relationship? Because of anonymity the person evaluating honestly admits that he did not read much. The reason for this most probably is that the course was uninteresting and perhaps the literature was not adequate.
Anyhow, good to read about the teachers perspective on course evaluation.
Hi GIJS.S.
OF COURSE, there can be certain grounds for a strong critique and therefore the criticisms in no way is a starting point of my article. The problem is therefore not such a situation which you describe where the student gives an indication that a course has a sub-standard. Furthemore it is true, as you point out, that students may have quite different backgrounds in relation to the given course content and form. This may influence the evaluation.
What is problematic is the evaluation inconsistencies. It is a screaming inconsistency when the student’s strong negative course evaluation is based on the criticism of the lack of involvement of text readings in the teaching situation. In the course concerned each lecture is based on the quotes from the syllabus texts. At the beginning of the course the student receives the reading structure which is explained by the teacher. The students are invited and encouraged to an active reading, i.e. they are welcome to present and review the syllabus texts, raise questions during or after the lecture, and they are told that they are very welcome to write emails to the teacher where they can raise issues and ask to clarify certain texts and the consistency between the texts. When one does not avail him-/herself of these opportunities, but at the end of the course anonymously emphasizes that the course has had too little text readings in the teaching situation it is not simply an expression of inconsistency, but also exposure of one's laziness, which ceteris paribus is unacceptable.
Another thing: from what you wrote I got the impression that you believe that you only have to do with text readings in teaching situation where there texts are written on paper. However Power Point may be particularly used to text readings in teaching situations. Instead of saying, now we turn to page xx, the text is on the canvas.
As for your objection to my second example of inconsistency, I agree with you that it is about a perfect cause - effect relationship. The reason for the negative evaluation involves the following circumstances: 1) the student has only read 20 or 30 percent of the curriculum, however 2) at the beginning of the course the readings structure of the curriculum has been presented and explained, which could inspire to a constructive critical engagement with text readings and the course content and form and thus tempt to read much more then 20 – 30 procent of the curriculum during the course, 3) during the whole course the student has not availed him-/herself of a manifest opportunity for verbally or in writing to ask and get into a dialogue with the teacher and other students.
Despite these circumstances the student evaluates the course negatively and as not interesting.
GIJS.S.: yes, this is a perfect causal effect relationship - between laziness and negative evaluation.
This is one of the weakest op-eds I have ever read! It does not examine any facts to back up any of the claims it is making and it seems motivated purely by the negative evaluations that this professor has received for his course(s). Perhaps, rather than dismissing all negative feedback out of hand, he should examine the possibility that his students did not find his teaching as wonderful as he thought it was.
If the concern is that negative (perhaps unjustly so) evaluations will affect the professor's career, there are simple statistical means to take them into account, by removing outliers.
The op-ed does not offer any actual solutions anyway, simply raging against anonymity but then stating that it should be preserved. Not everything that sounds good over a couple of beers with friends makes for a good article.
How can you conclude from what I've written that I turn against anonymity in the course evaluation? How can you conclude that I am against strong critique of a course?
On the other hand: do you think perhaps that one should not bring the anonymous evaluations to light and be critical of them? Don’t you think the critique in general, but especially within the academic world, has to be constructive?
"However, it is precisely this anonymity, that I am concerned with."
"Anonymity leads to wild variation"
It is these statements of yours that lead to my conclusions.
Feel free to address my main points, which are the lack of support data (what you have is a "sample size of one", aka "testimonial") and the lack of any possible solutions offered.
Being concerned with something (here anonymity) in no way means that you are against it wanting to abolish it. I state that: “Anonymity is, for better or for worse, an important prerequisite in many evaluation contexts, and also when it comes to the assessment of the quality of a university course.” Let me here add that in general anonymity is a crucial and unavoidable moment in the sociological method as well as the sociological ethic. Therefore it is an absurd conclusion that I should be against anonymity.
My concern with anonymity is based on exactly my experience with the course evaluation; there are relatively very few cases which demonstrate that the anonymity can be used as the shield for exercising inconsistent critique, because the critic/evaluator is shielded from facing response.
Therefore I also write, that “apart from positive aspects of anonymity, there is a disturbing part of it”.
Regarding your two main objections, that is lack of support data and lack of possible solutions: My observation is based on more than 200 questionnaires as well as a talk with a person who has a long experience with evaluation questionnaires, however not within the university milieu. Don’t you think it is enough to get a meaning about a certain aspect of course evaluation, and then share this meaning to a relevant public?
The solution to the disturbing part of anonymity is shortly mentioned in the article; with the greatest respect of anonymity the cases of an inconsistent critique included in the course evaluation questionnaires may be discussed within the group from which the critic comes. Let me repeat: “Anonymity should not be violated. But the vast majority of constructive evaluators and the small minority of inconsistent evaluators can benefit from the exposure to irrational assessments as to be exposed to some thought-provoking substance. They can get to see the importance of consistency and integrity in a critical approach to all the various institutional contexts.”
"If you don't like our Sociology courses then it must be because you're mental. We're perfect and what sort of irrational person could criticise perfection".
This seems to be what he really wants to say. My evaluation of the Sociology courses at KU was mostly negative, but not because I want to spite anyone, and certainly not because I think I could do better myself. In fact, it was because the courses were badly structured, the teaching was of poor quality for a university with such a good reputation.
Anonymity is a good thing. Do I want to look my lecturer in the face and openly criticise him, his course, his faculty and his university? No, I do not. Chiefly because I don't want to appear to be rude, but also because I don't want to offend him. Anonymous evaluations provide a way for students to communicate their true feelings to the powers that be, without worrying about reprisals. If Mr Galicki has a solution, that is just as cheap, quick and easy as a paper-survey then he should have used this article to spell it out. He has not done that, and therefore is perhaps guilty of exactly what he used his article to deride. I.e. "inconsistent" and destructive criticism.
Like in case of No name’s and GIJS.S.’ comments you wrongly and absurdly conclude that I’m against critique and anonymity. You may, of course, interpret my op ed in whatever way you wish: However as I don’t want to ignore your comment, I have to repeat in relation to your comment too: my op ed is due to the concern about a particular aspect of anonymity. On the one hand the anonymity is a crucial principle in the case of the university course evaluation to which I have experienced, and in general anonymity is often necessary in sociological research as well as in many other situations when private sphere and public sphere meet each other. On the other hand anonymity provides an unlimited possibility for critique full of nonsense. In my world view inconsistency and nonsense is a natural part of individual and collective human life, and only in utopia everything is rational. However within the academia inconsistency and nonsense are not natural. Nonetheless they can take place, and does it happen, the solution is an exposure, discussion and realization of the specific conditions which have contributed to illogicness.
Regarding your remark on mental state of evaluator: of course, you don’t need to be mental in order to deliver a critique which people concerned perceive as inconsistent and nonsense.
Regarding your evaluation of the sociology courses at Copenhagen University-
Among many different reasons for your critical stance on the sociology teaching quality at Copenhagen University could be the fact that you are not strongly motivated to study sociology. (let me stress, that I am not saying that it is the case that you lack motivation). I can see from some internet forums where the American students discuss the Sociology, that some of them are very unsatisfied with the Sociology courses at the American universities. I don’t use the statements below to generalize about the attitude of the American students to the Sociology courses, as I don’t regard your critical stance on the Sociology courses at Copenhagen University as an expression of the general opinion. I mean the students who strongly criticize the Sociology courses, whatever it is in USA or in Denmark, do not understand the essence of sociology as a social scientific discipline. It can create a motivation problem regarding study effort. Furthermore some students mean that sociology is politically oriented towards leftwing ideas and values. This can, predjudically, give a picture of sociology as politically biased social science and contribute to lack of motivation to study.
A reservation: I’m just voicing my thoughts. It is in no way a sociological analysis. However it can serve as preliminary thoughts for a sociological research.
Now, read beneath three critical statements from three students who take a stance on sociology at the American universities. The fourth comment is positive.
1)”Sociology is cake and a disgrace to American academia. Do not major in sociology if you have any self respect and ever hope to get a date with a hot woman. No girl respects a guy who majors in sociology.
Not so fast. Since you won't need to study, sociology is the ideal major for someone who just wants to party. You will get lots of dates with hot women.
I didn't say there were no good reasons to major in sociology. If you are bound and determined to party for four or five years with little or no inconvenieces like studying then sociology is for you.”
2)”The sociology department at most universities is a vehicle by which students who should be in trade school (or flipping burgers when you look at the effort they put into their studies) are kept in the financial aid pipeline thus ensuring a constant flow of federal dollars into the voracious maw of higher education.
Sociology and its retarded cousin, psychology, are probably responsible for most of the bloated growth of American universities. If these programs didn't exist they would have to be invented otherwise there would be no place for people who just want the college lifestyle without the work to go.”
3)”Sociology is essentially political indoctrination. They indoctrinate you with feminism, politically correct, feel-good socialism and marxism, and attempt to disguise under the guise of some "assignments" and "essays" which they give no one above 85% to make it seem like it's hard and academically challenging.”
And the fourth one which I, with my reason and heart, strongly go along with.
4)”I am a sociology major. Do I think that it is the easiest? No. Do I think that it is the hardest? No. Sociology is a very reading and writing intensive major at the majority of universities. I happen to love people and learning things about them and the situations that they are in. In ways, it is more difficult than a science such as biology because there is not one specific answer to a problem. It takes much critical thinking and understanding what goes on behind the scenes. Sociology is like biology, etc. in ways of the research aspect. It's like combining both a liberal arts and a science degree in my mind.”
Subscribe to newsletter
The University Post
- most read
- least read
- newest
Guide to Denmark's summer festivals 2012
Summer is right around the corner and so are the many festivals that sweeten this time of year. The University Post brings you 10 festivals worth looking forward to while you finish off the semester’s last exams
-
Graphic: Where are the jobs in Europe? -
Gallery: Life revue '12 -
Gallery: Physics Revue '12 -
Gallery: Canada students’ protest -
Gallery: USG Dance show -
Gallery: Party Watch, CSS Year Party '12 -
Gallery: War of the Wardrobes, Black Diamond, Copenhagen -
Gallery: Spiders at the Zoological Museum -
Gallery: Stars with Brains 2012 challenge -
Gallery: Royal opening of 'Day of Research' 2012 -
Gallery: War of the Wardrobes, Luxembourg Gardens, Paris -
Gallery: Taking hair sample from Egtved girl -
Gallery: Fieldwork in Disko Bay, Greenland -
Gallery: Research project is last hope for Danish ash trees -
Gallery: Follow Marte's free work-out -
Graphic: When do European students leave home? -
Graphic: Dissatisfied European students -
Gallery: War of the Wardrobes, Karolinska, Stockholm -
War of the Wardrobes: Copenhagen art event -
Gallery: Pictures from University Post Party at Studenterhuset
Photo Competition: Show us your room
Send us a photo of your room and win tickets to the NorthSide Festival
Kontakt redaktionen
Write us an e-mail: uni-avis@adm.ku.dk
Nørregade 10
1165 Copenhagen K
Denmark
Tel. +45 35 32 28 98
Copyright 2009 © Universitetsavisen.ku.dk