Are Qualitative Assessment and Student Self-Assessment Useful in Reading Instruction?

Teacher question

Our reading program has us evaluating students on several strategies and reading skills (e.g., orienting, predicting, monitoring, story elements, identifying point of view, word solving, retelling, inferring character traits, determining theme). It provides grade level rubrics so that we can tell the difference between whether students are doing 4^th grade or 5^th grade work. We are also encouraged to have the students themselves self-assess their progress on these elements. The idea is that we are to use these evaluations to help students see where they are. Is this kind of thing useful or is it a waste of time? 

Shanahan’s response

There are really two questions here — one dealing with whether this kind of qualitative evaluation of students’ reading ability provides useful information that would facilitate teaching and learning and one concerning whether we should involve kids in self-evaluation of their own reading ability.

What would it take for such a teacher evaluation tool to be useful?

First, it’s important that the skills and abilities evaluated must be central to reading growth. I have no doubt that someone can evaluate how well a student makes predictions, but I‘m  dubious that improving prediction will result in higher reading achievement. Likewise, I’d expect a real payoff from getting kids to pause appropriately during oral reading — makes sense to evaluate that — but getting the student to alter his or her voice when reading aloud probably wouldn’t pay off, so I wouldn’t take the time to assess that.

I think you could profitably jettison many of those rubrics without any loss.

Frankly, I wouldn’t spend a lot of time trying to evaluate specific comprehension behaviors. Instead, I’d evaluate kids’ comprehension by having them read texts and write summaries, engage in retellings, or answer questions.

Second, the evaluation must describe not just the reading behavior, but also the context in which that behavior must be demonstrated. The items that you sent me don’t do that. What level should the text be? How clear should the theme be? Are the students supposed to read the text on their own and write a theme statement or are they going to discuss it with the group? And so on. Let’s face it: If the text is easy enough, most of your kids will meet many of those goals. If it’s a hard text, then not so much.

I’d recommend standardizing how you will make those judgments.

Also, it’s important that these evaluations be made about specific reading events. You can’t assess generally. Often teachers will flip through these kinds of assessments at the end of a report card marking or to prepare for parent conferences. The problem with that approach is “haloes and horns.” We all tend to expect coherence. We make an overall judgment: “Jamal is not a very good student/reader.” Then, when asked about his vocabulary, fluency, comprehension strategies, and so on, we try to make those judgments consistent with our overall view. In other words, kids have either haloes or horns. We are not good at developing separate — and perhaps contradictory — judgments on long lists of related skills and abilities. But, if asked to evaluate something specific that we just observed, that we can do reasonably well — at least with some training and practice.

That last point isn’t unimportant. I’d feel better if the publisher could provide evidence that teachers have successfully and accurately made these judgments — and that doing so improved their teaching and student learning. Barring that, there is at the very least a need for some kind of professional development, aimed at guiding teachers to assess students’ reading. Again, my hunch is that is not what is being done — which to me means these assessments are probably not very useful.

But what about student self-evaluation?

I’m not big on the idea of kids grading themselves or trying to determine if they’ve learned a strategy adequately. Those kinds of evaluations are better relegated to the teachers.

But involving students in self-assessment should have more of an instructive purpose than an evaluative one. Peter Afflerbach, a professor at the University of Maryland, says that he groups “self-assessment with closely related metacognition, self-awareness, comprehension monitoring and even executive function.” All of which, of course, are related to reading comprehension.

Reading instruction should help kids to develop metacognition when they’re reading (Lin & Zabrucky, 1998). For example, your seven-year-old, Olivia, is reading her new library book. She reads, “I love my supper.” She pauses, looks at the picture with a puzzled expression, and re-reads that last line: “I love my surrr—prise… I love my surprise.”

Or let’s say your boss asks you to read a document. She tells you that she wants to talk to you about it this afternoon. She isn’t specific about what she wants to know. You read the text and think you understand it generally but recognize that there are a lot of specifics that need to be reviewed before the meeting.

Those two examples show metacognition at work. These readers are thinking about their thinking. They are paying attention to their reading and making the adjustments necessary to be successful.

Developing those kinds of abilities for reading is an important instructional goal.

Involving students in self-evaluation can be an important part of instruction towards that goal.

There is a lot of research showing the importance of metacognition to reading (Johansson, 2013), and several studies show that we can teach students to monitor comprehension, fix up misunderstandings, and select appropriate strategies (National Institute of Child Health and Human Development, 2000).

However, I know of no studies that have evaluated the self-evaluation part of their instructional routines. It is easy to think that would be a useful step, but at this point I’m not convinced the evidence is adequate.

Again, from Professor Afflerbach: “In terms of practice, I think there’s a demonstrable gap between the promise of late 70’s and early 80’s metacognition research and realization of that promise in reading curricula.”

I think he’s right about that.

Research reveals the challenges and complexity of self-evaluation (Dunlosky & Lipko, 2007; Glenberg, Wilkinson, & Epstein, 1982; Österholm, 2015; Pressley & Ghatala, 1990), but overall it shows that it can contribute to learning (Andrade, 2023). Most readers aren’t especially good at determining how well they have comprehended a text. And, the research hasn’t been especially articulate about how to successfully teach kids to evaluate themselves — at least in ways that make them better readers.

Nevertheless, it makes sense to me to involve kids in evaluating how well they are reading text passages, and if they recognize where their comprehension is falling short, to consider what strategies might address the problem.

How to best do this?

Remember there isn’t a lot of research direction here. One thing that I would do, however, would be to have students read texts at a range of difficulty levels. It is a lot easier to self-evaluate if you can experience a range of degrees of comprehension.

Also, studies show that readers do better with self-evaluation when they are actively reading; for instance, self-assessment improves when readers read and summarize rather than when they only read (Maki, Foley, Kajer, Thompson, & Willert, 1990).

The scheme you showed me isn’t very good in my opinion, but its heart is in the right place. I’d suggest that you trim it down, standardize it, and convince your district to invest in professional development aimed at enabling you and your colleagues to evaluate successfully.

But remember, the purpose of student self-evaluation is less about assessment and more about teaching. Getting kids to evaluate how well they understand paragraphs or sections of a text — which ones they are certain they understand, which ones are confusing them — can be a good starting place for starting those instructional conversations.

References

Andrade, H. L. (2023). A critical review of research on student self-assessment. Frontiers in Education. doi: 10.3389/feduc.2019.00087

Dunlosky, J., & Lipko, A. R. (2007). Metacomprehension: A brief history and how to improve its accuracy. Current Directions in Psychological Science, 16(4), 228–232. https://doi.org/10.1111/j.1467-8721.2007.00509.x (opens in a new window)

Glenberg, A. M., Wilkinson, A. C. & Epstein, W. (1982). The illusion of knowing: Failure in the self-assessment of comprehension. Memory & Cognition, 10(6), 597-602.

Johansson, S. (2013). The relationship between students self-assessed reading skills and other measures of achievement. Large Scale Assessemnts in Education, 1(3).

Lin, L., & Zabrucky, K. M. (1998). Calibration of comprehension: Research and implications for education and instruction. Contemporary Educational Psychology, 23(4), 345-391.

Maki, R. H., Foley, J. M., Kajer, W. K., Thompson, R. C., & Willert, M. G. (1990). Increased processing enhances calibration of comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(4), 609–616. https://doi.org/10.1037/0278-7393.16.4.609 (opens in a new window).

National Institute of Child Health and Human Development. (2000). Report of the National Reading Panel: Teaching children to read: Reports of the subgroups (00-4754). Washington, DC: U.S. Government Printing Office.

Österholm, M. (2015). What is the basis for self-assessment of comprehension when reading mathematical expository texts? Reading Psychology, 36(8), 673–699. https://doi.org/10.1080/02702711.2014.949018

Pressley, M., & Ghatala, E. (1990). Self-regulated learning: Monitoring learning from text. Educational Psychologist, 25, 19-33.

Listen to this post on Shanahan’s podcast:

https://player.cohostpodcasting.com/7763ba9d-7746-4eeb-83a5-38e6bfd1c124/495b8963-4f00-41df-adae-5781df3322bf