Tonight, we'll be discussing assessment in ESL/EFL classes. For this blog assignment, think carefully about a test or assessment that you have taken at some point that you were unhappy with for reasons other than your own lack of preparation. What was the issue with this test/assessment? Was it an issue of reliability or validity? What kind, and in what way? Thinking back, what would have been needed in order for the test to give a higher level of reliability or validity?
Please answer with 2 paragraphs, one which describes the problems with the test, and another that suggests ways to fix those problems.
Responses should be submitted by classtime on May 4.
Wednesday, April 27, 2011
Subscribe to:
Post Comments (Atom)
Throughout my EFL student life, instruction was solely form-focused rather than communicative and thus, assessment was solely carried out in terms of traditional written tests. For example, fill in the blanks or circle the correct answer (multiple choice questions). As I moved to college, I started taking courses that in essence were communicative and focused on listening and speaking skills rather than reading and writing. However, students speaking and listening skills were tested through paper and pencil exams. In such cases, the issue with these exams is a validity matter, because I do not think that the appropriate way for testing listening and speaking is through written exams!! Clearly, these tests were designed to test anything else but listening and speaking.
ReplyDeleteInstructors of listening and speaking classes should be able to develop assessment tools that while valid are also reliable. To design a valid assessment tool that tests speaking and listening is to actually test students’ ability to produce spontaneous speech and test their listening through actually listening to a stretch of speech whether recorded or recited by the teacher and test students listening by asking comprehension questions. It can be quite challenging to develop reliable listening and speaking assessment tools due to the wide spectrum of variables that can go wrong starting with administrative issues all the way to teachers’ biases and subjective assessment issues. Nonetheless, I believe that having a well designed rubric that teachers can follow will tremendously help to eliminate lots of reliability issues that can arouse during speaking and listening assessments.
In my high school one of my teacher gave an assessment test and it consisted of an assay. The issue of this assessment did not reflected the students ability and there overall learning of the material, Furthermore, I believe that the grading system of the teacher consisted of the accurate rubric dimensions because it did not reflected the students learning outcomes and levels of learning. Also I think that the teacher compared the essays with the rest of the class. I also believe that the test was based on Validity because it only measure what the teacher believed it was suppose to measure which was grammar and not content in the essay.
ReplyDeleteI believe that if the teacher would of change some of here grating strategies and allude the students to have a lilt freedom on what to right instead of giving a list, the grading can still be based in grammar but also it can be fun for the students and the teacher or she could give a concepts in within students will have a knowledge-base. If the professor trade to know their students than when it came down to the grading she would now the students’ strengths and weaknesses in comprehensive manner and will give a little more credit of those who have work harder. Also, if the teacher provides opportunity for students to synthesis own work it will also help the students understand their own weakness and strength.
One assessment tool I have administered has been the commercially developed type, placement tests which generally come with the texts being used for the class. What I call the straight forward, quick assessment diagnostic test, designed to place learners in appropriate instructional levels. Serviceable, and frequently used because they have construct validity and scoring reliability; but not particularly effective in determining what the learner really knows. This type of test is very problematic when dealing with learners at the lowest levels of literacy and language skills because they do not capture the functional skills necessary for second language learning. These tests certainly do not address what the learners need to know or want to know.
ReplyDeleteTheir intrinsic value lies in the “hard data” they provide for researchers and educators who want to quantify learning outcomes. The problem with them is not so much the test itself as it is the appropriateness of the test for measuring exactly what you want to measure. Correcting that problem lies in being realistic about what you want to know. For example, using a standardized “literacy” test (assessing reading and writing) to measure speaking and listening ability is not the way to go. Another fix to this problem is to use them in combination with classroom-based assessment tools such as teacher-developed tests, learner self-assessment tests, surveys, questionnaires, oral interviews, et al. In the end, both of these instruments are the key to assessing learner progress and program success.
The tests that stick in my mind have been the large-scale “aptitude” tests that we all have had to take at some point in the past: the GRE and so forth. I first took the GRE as an undergraduate in Fall 1981; more recently, I took it again in 2009. What a difference a couple of decades make. 1981 was paper booklets and scantron-like forms and so on. 2009 was the testing center, all online, and rather different components. (We had something like logic problems in 1981, which are gone nowadays.) The issues are many, as we began to discuss in class: whether students have taken a test-prep course, the atmosphere in the testing center, and the student’s comfort (or discomfort) in the high-pressure environment in which these tests are administered.
ReplyDeleteThe powerful ETS organization controls these tests and refines and revises them constantly, and they tell the world that they work on making the tests both as reliable and valid as possible. The academic community (universities) all over the world accept the test results and use the results both for the intended use (to add a “leveling” factor to student grades across universities) as well as for non-sanctioned purposes (using the results as cutoffs for admission to degree programs, for example).
I question that these tests have a great deal of validity for what they test. The tests can be reliable for those students that perform well on them, but I question that poor results that plague certain students are reliable. The tests now include a writing component, and the scoring is supposed to be normalized and the assessors well trained. So I can see that writing and reading and quantitative skills are somewhat reliably tested. For us as language teachers, I would like to point out that the basic academic-success skills of listening and speaking are not tested at all.
I think that quite a few changes would be needed to achieve higher reliability. Perhaps the most basic would be to increase the amount of testing that students are allowed to undergo. Entire university careers are determined by one four-hour test that is essentially a single-opportunity assessment. If the student does poorly, another (expensive!) attempt at the test is needed. As well, adding listening components to the test would require a revamp of the current model, but would help assess an important skill that is crucial for academic success, which these tests purport to evaluate. Finally, adding a speaking component would help with that skill, too, which is also not assessed at all.
As far as validity, the verbal component seems to test reading comprehension with some validity, but reading of many types other than what the tests present is necessary for academic success. As well, we often hear that the best way to raise a verbal score is to study the vocabulary that the tests assess, and this fact seems to make that result pretty invalid, especially when you consider the specific words that occur in the “likely to be on the test” lists.
I was finally able to get a copy of the assessment tool used by the community center where I teach. I have really been curious because my observation of the class is that it is very multi-level. The class I teach is the “advanced” class. With some, I have been concerned about general literacy issues – they cannot form a sentence. With others, they seem to be truly “advanced,” lacking only some of the subtleties for that level.
ReplyDeleteThe test must be a proprietary product of the community center since no attribution to the source was evident. The test is all multiple choice and fill in the blanks. Items range from very simple picture identification (a picture of a person in bed sleeping and the question is, “They’re ________ a) cleaning, b) sleeping, c) swimming,” to more subtle grammar issues, “Ana wants _______ the game. A) win, b) wins, c) to win.”
Of course, there is no testing for speaking or general writing ability. My questions about this approach are many. Should we try to determine the students general ability to organize their thoughts into a cohesive sentence (or paragraph) before placing them in an advanced class? Should we at a community center worry if they can write effectively? And, of course, why don’t we test their ability to speak?
I’m trying to weigh the social issues surrounding this as well, considering the mission of the community center. This has given me a lot to think about.
When I took the TOEFL test last year, I found something that could be changed for even more reliable test. To briefly introduce the procedure, in the internet-based TOEFL test, it first starts with the reading section and then continues to the listening, speaking and finally, the writing section. As each person’s pace differs from that of one another, you could be in listening section while your neighbor is still solving reading problems. AND this caused a problem! When I was in the middle of the listening section, the girl next to me moved on to the speaking section and started to answer quite loudly! Because I was listening with the headset on, the noise didn’t completely block the sound. However, it surely was annoying and made me nervous. Furthermore, I noticed that some of students didn’t click the ‘proceed’ button after their listening section and just be seated there listening to what their neighbors are talking to answer the speaking questions. Then, they might have guessed the questions and prepared what to say!
ReplyDeleteIt is beneficial for test takers that they can flexibly manage their time. They could quickly finish what they are confident about and invest more time for confusing problems. However, it also causes side effects; it may distract people because they all could be in different sections, and if some of the test takers can infer what problem they are going to have from what others’ saying, it is not fair. In both cases, it decreases reliability of the test. To resolve this problem, I think there should be enough space between two test takers. The room I took TOEFL last year was small. It had about 12 seats but there was no enough room among seats, making it easier to be distracted with the voice of neighbors. Another suggestion is to start the next section all together. Someone who is done early may take a rest outside or stay in the room, but he or she should wait till the time limit to go on to the next part of the exam. This will somehow reduce the disadvantages.
Looking back, I remember some assessments and exams back in my University in Russia that were unrealistic and ridiculous as far as the amount of material that was required of students to know. For instance, I remember a test at the end of EFL Country Studying course where the whole semester we were writing down lectures on History of Great Britain, and then we had to memorize them all and pass the closed-book test on very specific details and dates. The whole course was very strange, as the teacher did not explain anything, did not go over vocabulary or anything like that, she was just dictating from a history book, there was no guarantee that we even wrote the things down correctly. There was no practice or review, there was just a test at the end. Some students passed just by guessing, the majority failed.
ReplyDeleteSo, in this case, the test could have been valid, as it was actually testing the knowledge of History and traditions of the country, language of which we were studying, but on the other hand, it was not reliable, as the students really did not know the answers and they were just trying to guess them, as the material was not properly taught, it was overwhelming and even though the teacher might have covered everything that was on the test, but the way she did it was absolutely impossible for the students to learn or understand.
The teacher could have turned this EFL Country Studying class into an exciting class of discovery of historical and cultural things about the country with such rich and amazing history like Great Britain. The teacher should have had a review, she also should have been more specific about what the test could include and, of course, she should have had mid-point assessments to see where the students were in terms of understanding the material. She should not have waited until the end of the semester to find out that most of the class failed the course. Moreover, the teacher should not have been so intimidating that the students were afraid to say anything.
So, I shared this scary story, which is absolutely true even though a little embarrassing, to make a point that before conducting the final assessment it's a good idea to have some assessments along the way to make sure that the students are learning and the teacher is not just talking to herself or himself.
Having taken a lot of Spanish classes throughout my life, I've seen one kind of assessment keep popping back up, Verb conjugation. In Spanish it is important to know how to conjugate the verb correctly and how to attach pronouns to the verb. However the way many classes test this information is not reliable. On these test they ask you to: simple conjugation changes from imperative to the target tense, conjugation changes in a paragraph with at most two possible verb tenses, and conjugation change from one tense to another. This test has some validity that the student may know the tense conjugation, but it does not show if they know how to use it correctly.
ReplyDeleteTo fix this problem, questions on the the test can be designed so that the student needs to respond in a certain tense. This would show that the student knows how to both conjugate the verb, but also know when and how to use verb tenses. Also the parts of the test that ask for verb conjugation in the paragraph should always have more than one tense and should be designed to try to trick the student into using the wrong tense so there would be more active thinking involved in the responses rather than just trying to decide the gender and number of subjects.
I remember a test I took as an undergrad in a human cognition class that was reliable but not valid. Each class was given a multiple choice scantron test that made it extremely reliable. Unfortunately it had validity issues because the professor emphasized the importance of reading our textbooks and familiarizing ourselves with that author's work, but then problems arose on the test. She would put questions on the test that were discussed in the textbook but with answer choices that included what the book said and what she had said about that topic in class, and keep in mind that what she said in class contradicted the textbook. However, when she would mention these things in class she would not mention that it contradicted the textbook nor would she dwell on her opinion of the topic, so I was always at a loss as to whether this time around she wanted the book's answer, or her answer, which varied as well.
ReplyDeleteA way to fix this would have been to either omit those questions, or include only one possible answer choice for those questions, either her opinion or the author's opinion, but not both. Otherwise it is hard to tell whether she is testing us to see if we attended lectures, or whether we read the textbook. Or better yet, she could have made those questions free response essays to see if we had done both and understood the concept well enough to discuss that differing opinions exist on that matter.
The midterm exam for English Literature background class was an open book test. The questions were about descriptions on the Greek and Roman Myths used in English Literature. We were gathered in a classroom and we took the exam. Probably the professor thought that in order to quote relative myth in our answer, open book test was more appropriate. However, some of the students prepared for their answers from the previous year’s exam questions. Some of them even made their answers already and copied them from their cheat sheets. I did not think about previous year’s exam questions. He did not offer to give us any copy of it. He did not refer to that kind of questions in the class at all. Some of my classmates had their own social connections and they got the previous year’s exam questions. They nailed it. I was very upset. It was not fair to the students who did not have a chance to see the previous exam questions and it was hard to get one: it was only circulated among the a few students.
ReplyDeleteRecycling previous year’s exam is unfair. If the professor gave it to us before the midterm, then it would be fair. If he could not have made up a new exam for the class, he could have given an assignment or a project to replace the midterm. As a teacher I am tempted to recycle some of the questions from the previous classes, however I guess I should not even think about it.
In many ways, I agree with Martin's evaluation of ETS standardized tests. However, I would like to narrow my focus specifically to the relatively new essay-writing portion of the SAT exam. I'm just an undergrad junior, so during high school I was in one of the first sets of students to experience this additional test section. I actually believe that this portion of the test was pretty reliable, given tthat writing is very difficult to evaluate fairly. In many high school English classes, students were (and are) given the SAT writing rubric in advance so that they know what the graders are looking for. Though this test section was/is graded by humans, it seems that test graders are uniformly trained and that essays were submitted to multiple graders to ensure scoring was as fair as possible. I think the real issue at stake for the SAT writing section was/is validity. It tests for solely form, not content – as long as a student's essay is spelling/grammatically correct and well-organized into a five-paragraph structure, that student is nearly guaranteed a perfect score. In an attempt to be “unbiased,” perhaps, this test does not evaluate the truth/strength of the content of a student's argument and its supporting examples. Graders cannot dock points for “facts” that are incorrect if they are incorporated into the 5-paragraph or so structure. In fact, my high school English teacher encouraged me to lie on this portion of my SAT – to make up “facts” or personal examples to fit the prompt, because this could not effect my score!
ReplyDeleteFor the writing portion of the SAT to be valid and reliable, I would recommend that also measure some sort of content rather than solely form, in order to best evaluate the strength of students' arguments. For example, it could test a student's knowledge of social studies (a subject which is absent from the SAT anyway) by offering prompts that force students to argue and use critical thinking about historical events that they should've studied in high school.