RIP to Advanced Phonemic Awareness

Selected comments

Comment from Jessie

Thank you for this thorough explanation of the research that led to Kilpatrick’s hypothesis. As a follow up question, would there be any benefit of doing addition, deletion, or substitution tasks to directly support phonics instruction? (So not just skill drill on these practices for the sake of the practice itself, but rather as an entry point for print learning.) For example, would it help students to do a few warm up addition tasks orally first in preparation for a lesson on blends (so they hear that pattern of moving from a CVC word to a CCVC or CVCC word)? Similarly, would it be helpful to warm up with a few medial substitution tasks as students are first learning the VCe pattern (so they can hear the distinction between the closed syllable short vowel sound and the long vowel sound in the VCe pattern)? Thanks for sharing your expertise! 

Reply from Tim Shanahan

Jessie —

Nobody really knows. It is clear that analysis (segmenting) and synthesis (blending) have functional roles to play in reading and spelling. The idea that engaging in skills like deletion and manipulation is meant to help those kids who have phonological deficits to develop them – in other words, practicing with something harder that might stimulate the kinds of skills the other kids are picking up. I guess that means such additional practice could be helpful to a small percentage of your students – and then, again, since it hasn’t been tested, it could just be a waste of time (and a waste that would reduce the amount of decoding instruction you could be providing). If I were you, I would wait until the idea has been evaluated OR test it out in your own class to find out if you can see any difference in student progress (in other words, if you try it, try it with a grain of salt).

Tim

Comment from David A. Kilpatrick

Thanks Tim for your blog. I think it went a long way in terms of clarifying this issue. I really appreciate that.

I would, however, like to address something. You mention in one of your responses the idea about putting “belief above data.” To be clear, that’s not what is going on here. Quite the contrary, actually. My view on this issue emerged by complete surprise. I wasn’t looking for it but noticed it when sifting through the data in the intervention research literature (i.e., struggling readers) and considering standard score gains on nationally normed word identification tests reported in those studies. The data from the intervention studies induced the “belief.” Is it a hard scientific finding? No. Is it a demonstrably clear pattern in the scientific literature? Yes. It’s more than just a nice idea.

You and I both agree that a direct study of this phenomenon is needed, in which two groups of struggling readers get identical phonics instruction but the only difference between groups is the nature of the PA instruction. Where we seem to disagree is in what manner the existing intervention research should guide our efforts with struggling readers.

Studies that use phonics with no additional oral phonemic awareness typically show a benefit compared to a control group (usually of a whole language intervention). However, improvements on nationally normed tests of word identification in those studies are consistently modest, usually 3 or 4 standard score point gains. Even though national norms are far from a perfect point of reference, they let us know if a student is actually “gaining” on his or her peers.

Then, when we look at phonics intervention studies that add oral PA (or PA with non-lettered tokens) but only train to the point where students can segment and blend, the results range between 6 and 9 standard score points on normed word identification tests. But in studies that used phonics instruction and more rigorous phonemic awareness intervention (orally and with non-lettered tokens), usually involving deleting and substituting phonemes, standard score gains hover around 14-17 points on normed word identification tests. Some of these studies had control groups, some did not. But control group or not, we have to ask why one type of instruction consistently yielded substantially higher standard scores on the same type of word identification measure. We simply don’t see such strong gains using the less rigorous PA interventions.

These findings cut ACROSS studies, rather than represent the results from WITHIN a given study. So where we differ has to do with whether we put any stock in the fairly consistent pattern of standard score point outcomes on word identification tests.

I’m not suggesting that this pattern in the intervention research is a hard finding like the results of a random-control-trial (RCT) experiment. But given that teachers are looking for best practice, I’m suggesting it might be a good idea to emulate the studies with the strongest word identification outcomes rather than emulate the studies that consistently display lesser results. That seems reasonable to do while we await further research that may settle the issue.

I’m talking about struggling readers, not Tier 1 instruction.

Anyway, I wanted to be clear that we are not talking about beliefs or hunches or ideas untethered to data, but rather something that emerges from the data. We differ only on where to set the “best practice” bar. True, there is no RCT, but there is a clear pattern of empirical findings. I think emulating the studies with the strongest outcomes until further notice is a reasonable way to conceptualize best practice.

Again, I appreciate you clarifying the relationship between phonemic proficiency and orthographic mapping and explaining to folks that the term “advanced phonemic awareness” is a confusing phrase in need retirement.

Reply from Tim Shanahan

David —

I’m concerned about the issue that Nathan raises below. He isn’t saying that you went out looking for this pattern, but he is saying that he is skeptical about whether that pattern exists at all. Tiffany raises a real similar question. My original concern is that the idea of “advanced PA” and the associated activities that were presumed to teach it hadn’t yet been evaluated in classrooms and that I didn’t think that was good for kids. You appeared to share some of those concerns – believing people had conflated instructional activities with the cognitive changes you were trying to elicit and by generalizing something intended only for struggling readers to the whole population.

But these folks are questioning the pattern itself it seems to me. I very much like the suggestion that your next step should be to try to nail down the construct by conducting an appropriate meta-analysis to see if there is such a pattern, how reliable it is, how strong it is, etc. The nice thing about that suggestion is that it wouldn’t require a long wait for schools to be willing to be studied again due to COVID. Those data exist, you’e reviewed those thoroughly to your satisfaction and now you could put the idea up to scientific scrutiny at this point. I didn’t think of that myself but given how egregious some of the misinterpretations have been so far, I think it would either reassure some of those who have bought into those ideas (thinking them to have been nailed down) or it might do away with the issue altogether if Nate is correct about the pattern itself. Cool and helpful idea, I think.

Tim

Comment from Nathan Clemens

Thanks Tim, I agree with a number of points here particularly about the lack of evidence. A significant problem that must be acknowledged is that David has repeatedly framed his recommendations as being evidence-based. For example, from Kilpatrick & O’Brien (2019): “Yet when students receive more challenging phonemic awareness training, particularly using phoneme manipulation activities (phoneme deletion and substitution of phonemes within various positions within words), a greater degree of phonemic proficiency develops. This presumably allows students to more easily remember the words they read, resulting in the largest standard score point gains found in the intervention literature (e.g., Alexander et al., 1991; McGuinness et al., 1996; Simos et al., 2002; Torgesen et al., 1999, 2001, 2010; Truch, 1994)” (p. 203).

Just a cursory look at these citations reveals that none offer a basis for causal inference, nor were the studies even designed to test the hypothesis in the first place. Misrepresentations of the evidence base like this have been relayed to thousands of educators.

Comment from Tiffany K Peltier

Dave and Tim, thanks for your responses.

Dave, I agree that a systematic review of the literature would be a valuable thing to do before we can say this pattern you are proposing is actually a part of the science of reading.

I’ve seen states mandate some of these hypotheses you have made about phonemic proficiency, “advanced PA,” oral PA instruction, the PAST assessment etc., as part of *law* under the term “Science of Reading” (such as Oklahoma and Arkansas) which seems very problematic. I hope states read this and begin to shift their practices back to what we know from peer-reviewed studies.

I agree with Dr. Brady that some of the recommendations to teachers over the past couple of years have been problematic too, not only because a systematic review hasn’t been done, but also because the recommendations from the patterns you posit are contradictory to the research we have directly comparing instruction and intervention for students in PreK, K, or at-risk for reading difficulties. It would be great to be able to see all of the studies together to be analyzed in a systematic way.

Great conversation. Thanks for the blog!

Reply from David A. Kilpatrick

Tiffany and Tim,

Yes, of course a meta-analysis would be most useful here, which is why I’ve been working on one.

Back in 2013, I started doing a review of intervention studies for students with word-level reading difficulties. I was following the lead of Joseph Torgesen whose 2005 research synthesis that appeared as a chapter in Hulme & Snowling’s “Science of Reading” volume was, I believe, the only synthesis/review of word-reading intervention based upon standard score gains rather than effect sizes. It was in the course of working on that paper that I noticed the pattern I described above. I have assumed the reason this pattern had not previously been noticed was because all other reviews used effect sizes. Effect sizes do not represent a stable point of reference across studies and stronger or weaker performances by experimental groups can be minimized or exaggerated depending on the performance of the control group. Although the idea of using standard scores as the outcome variable has its own issues, it does not share this major drawback displayed by effect sizes. Given that no one looked at the research from this vantage point before (other than Torgesen), it should be at least somewhat unsurprising that something was noticed that previously had not been noticed. Torgesen did not notice this pattern because he did consider differences in the nature of the interventions but rather focused on testing out the metric of number of standard score points per hour of intervention across studies. While I was working on this project, I was invited to write a book that took up the better part of two years. The review did not get completed, but I summarized what I had found in that 2015 book. Earlier this year I resumed working on it. Given concerns voiced here, I’m happy to bump up that project on my priority list.

Comment from Lauren Thompson

Thank you for the clarifying post. I’m so glad that you got Dr. Ehri and Dr. Kilpatrick directly into the conversation. I have accepted that Kilpatrick’s theory is still a theory, but an interesting and productive theory. It’s got me going back to the NRP report and trying to parse exactly what the conclusions were about phonological and phonemic awareness instruction. Perhaps you/they can help.

1. The NRP report states in a number of places that studies showed that phonemic awareness activities that involved letters at some point, in some way, led to faster acquisition of reading skills. Most of the studies involved young children, who were perhaps still learning letter-sound correspondences. Did using letters with PA activities improve students’ sound-symbol knowledge, and thus improve there reading development, or did the letters clarify what phonemes are and help students to grasp the concept of phonemes? Both? Perhaps students with good sound-symbol knowledge don’t need to use letters to become better at phonemic awareness. Many of Kilpatrick’s One Minute Activities, which my students enjoy, are very difficult to do if letters are involved, because of the different spellings of rimes. Does the NRP report advise against doing phonemic awareness activities with sounds only with students who already know their ABCs?

2. The NRP report also repeats that doing a few phonemic awareness activities was better than doing many. What’s unclear is whether this means working on only a few types of activities in each session, or teaching only a few activities through the whole program. Kilpatrick’s program calls for teachers to focus on only two levels of activities in one session, one at the near-proficient level and another at a near-accurate level. Another popular phonological awareness program calls for teachers to present up to 10 different activities in one session (without letters, by the way). Do the findings of the NRP report support either approach? Or advise against them? The NRP mentions two activities in particular, segmenting and blending. Are these the only activities that are recommended? Or is this recommendation only for beginning students in K and 1?

Perhaps these could be two ideas for future conversations.

Lauren

Reply from Tim Shanahan

Lauren —

The National Reading Panel found that phonemic awareness instruction in Grades K-1 and with older remedial readers with deficits in phonemic awareness benefited from explicit PA instruction. The instructional benefits were biggest for the normal range of K-1 students, they were somewhat smaller for at risk K-1 students, and with the least payoff for those older kids.The studies provided between 1-99 hours of PA instruction in these studies, with the best learning response from those who received between 14-18 hours of it. That would be comparable to 15 mins per day for a semester. The panel did not conclude that to be the right amount as it was recognized that some kids would not reach those blending and segmenting benchmarks and should receive more (and for kids who could do those things proficiently, the time could be less). The studies did not favor any specific lessons or lessons focused on specific PA skills — though getting kids to the level at which they could blend and segment was the target (perhaps some of those more complicated activities could stimulate kids to segment better, etc.). However, it was found that the instruction should stay simple (focusing on only 1-2 skills at a time to avoid confusion). PA worked best when it was combined with letter instruction, so no, it would make no sense to either wait for ABC knowledge to be there or to hold off on ABCs. Many people think of PA as being a kind of readiness step for explicit decoding, but that moving back and forth between PA and decoding appears to be better (I think the same can be said about the research conducted with preschoolers and kindergartners reviewed by NELP). I know the idea of including letters or in combining PA and phonics in those ways upsets some folks, but in the 51 studies that we reviewed that was best for children’s learning.

Research since NRP and NELP hasn’t really challenged those findings and until it does, I think those features of PA instruction are the best bet. David’s theories certainly suggest some other possibilities and when those are evaluated, it is possible that we may need to adjust our thinking.

Thanks.

Tim