Which Should We Use, Nonsense Word Tests or Word ID Tests?

Teacher question

I am an Assistant School Superintendent. We are moving toward explicit phonics instruction this year and are debating between using the nonsense words assessment or the decodable words assessment. Do you have thoughts about this? I have consulted with several people who I respect, and opinions are varied and passionate.

Shanahan’s response

I feel your pain.

Recently, a colleague asked me to make a similar recommendation to help figure out something about a grandchild’s reading. I suggested the use of DIBELS Nonsense Word test, given the specific purpose and its easy availability.

You’d have thought I’d recommended drowning kittens or banning the Barbie movie!

People do get passionate about the strangest things.

I try to save my passion for non-empirical questions (Go Cubs, go!). If we have data that will allow us to make a sound determination, I’d turn the heat down and try to follow the numbers. Remember this is about trying to do what’s best for kids. It is not an opportunity to vent your spleen or espouse your philosophy.

There are two different kinds of tests used to determine student progress in decoding. Both kinds have a proven ability to evaluate how well students are learning their phonics and both can predict later success with oral/text reading fluency and reading comprehension.

Word identification tests have been around for a long time — more than 100 years. Nonsense word or pseudoword tests are a newer development.

Researchers were concerned about the validity of word identification tests for determining the effectiveness of decoding instruction. Word identification tests often focus on irregular spellings (e.g., the, of, done), the kinds of words that are inconsistent with the spelling patterns usually stressed in phonics. Such tests couldn’t tell much about the effectiveness of phonics instruction. Even word tests with more common spellings were suspect. With such tests it was impossible to know if a student decoded a word or just remembered it from previous exposures.

The solution to the problem was the creation of nonsense word or pseudoword tests. Because the researcher (and, later, the test designer) constructs the words by mimicking English spelling patterns, there are no exceptional spellings, one-offs, accidents of morphological history, and the like. Whether teachers are leading the kids to memorize Dolch or Fry list words or are just providing them with repeated exposure to certain words through phonics instruction, it was certain that the students wouldn’t have previously seen before letter combinations like dop, lan, or sepe.

The idea was that a nonsense word measure would provide a purer look at how well students can decode, and their performance on such a test should reveal their decoding progress.

As is often the case, scientists often may identify a real problem, but solving it may not be so easy.  

At first blush, the nonsense test appeared to do a terrific job of assessing decoding ability, perhaps, more valid than the traditional word identification test.

Over time, their faults became evident.

Often, if teachers know that their students are to be evaluated with nonsense words, they start teaching them to the students. This teaching is a waste of time for producing readers and renders useless the intended improvement in test design. Researchers and school district administrators must be vigilant in discouraging teachers from fraudulently enhancing their students’ test performance. (I don’t think most teachers are intentionally trying to defraud — they just want to make sure their kids do well on the test, and teaching the specific test items seems logically to be the most direct route to that outcome.) well meaning but unfortunate

A more important issue has to do with the nature of decoding. There is more to decoding than pronouncing letter patterns. Pseudoword tests provide a useful assessment of that part of the process, but not of the rest.

As Richard L. Venezky so aptly described the process:

A third function of phonics is to generate a pronunciation for a word …. This function is problematic, in that the imperfections in English orthography make such generation uncertain. If a word is totally unknown, the reader has little basis for deciding whether any particular pronunciation is correct or not (Venezky, 1999, p. 202)

Phonics is a tool for helping readers to decode the words in a text. But that is a necessarily imperfect process due to the complexity of the English spelling system. Some “experts” throw up their hands, ready to surrender; for them, phonics would be useless because of the complexity of our spelling system. But as Venezky points out, readers don’t need to arrive at exact pronunciations. Reasonable approximations are good enough, and then the readers make adjustments and consider alternatives based on their knowledge of the English language.

Nonsense tests, by their very design, can tell us whether students have managed to master particular spelling patterns, but they prevent students from any kind of self-evaluation and adjustment of pronunciation, key aspects of decoding. As such, these tests may do a good job of evaluating student learning from a decoding program, but they are unlikely to do equally well in predicting later reading achievement, as measured by oral reading tests, or reading comprehension tests.

What do the research studies have to say about the usefulness of these measures?

For the most part, word identification tests and nonsense word reading tests tend to be interchangeable early on. There are copious amounts of validation data showing the value of both (e.g., Fien, Baker, Smolkowski, Kame’enui, & Beck, 2008; Vanderwood, Linklater, & Healy, 2008). They both work reasonably well (i.e., there are high correlations between these measures and other reading tests).

However, in direct comparisons in which students are taking both tests so that they can be evaluated head-to-head, the word identification tests tend to do a bit better. For example, in one well-done study it was found that word ID tests provided a “clearer index of reading growth” (Clemens, Shapiro, Wu, Taylor, & Caskie, 2014). Early in first grade, the tests were indistinguishable, but by second semester the word identification tests inched ahead.

Similarly, in a very large study of first graders (n = 3,506, from 50 schools), it was reported that the Nonsense word Fluency tests did the best job of predicting end of year reading fluency and comprehension for most kids (Fien, Park, Baker, Smith, Stoolmiller, & Kame’enui, 2010). There are other studies of this with similar results (e.g, Fuchs, Fuchs, & Compton, 2004). However, this was not true for the higher achieving students. As kids’ reading advanced, leaving out those word identification skills that Venezky noted becomes a real problem.

By third grade the correlations between NSW and word ID separate to a greater degree with the real word performance becoming the best predictor of ORF for most kids (Doty, Hixson, Decker, Reynolds, & Drevon, 2015).

Finally, a recent meta-analysis of data show that across many studies, word ID tends to have the best relationship with various reading outcomes (January & Klingbeil, 2020).

None of these differences just noted are especially large, though they are often statistically significant. Nevertheless, some authorities suggest including both in early reading inventories, and that makes a certain kind of sense since they tap a slightly different array of skills.

I certainly have no problem with ongoing monitoring of decoding skills with nonsense words, alongside a word reading check to determine how well kids can read those most frequent words.  

If you are only going to give one, and your specific interest is monitoring phonics progress in grade K-2, I’d go for a real word reading test — especially second semester of grade 1 or later and with my highest achieving schools. Those tests should do a slightly better job of revealing student progress towards success in reading. Just make sure, given your purpose, that the word ID test that you choose includes many words with regular spelling patterns.

But remember the differences here aren’t large. In a different situation (e.g., I’m a school psychologist and a student has been referred to me due to a concern about his/her phonics ability), I would likely give you a different answer. You really can’t go too far wrong in this case.

References

Clemens, N. H., Shapiro, E. S., Wu, J., Taylor, A. B., & Caskie, G. L. (2014). Monitoring early first-grade reading progress: A comparison of two measures. Journal of Learning Disabilities, 47(3), 254-270. doi.org/10.1177/0022219412454455

Doty, S. J., Hixson, M. D., Decker, D. M., Reynolds, J. L., & Drevon, D. D. (2015). Reliability and validity of advanced phonics measures. Journal of Psychoeducational Assessment, 33(6), 503-521. doi.org/10.1177/0734282914567870

Fien, H., Baker, S. K., Smolkowski, K., Smith, J. L. M., Kame’enui, E. J., & Beck, C. T. (2008). Using nonsense word fluency to predict reading proficiency in kindergarten through second grade for English learners and native English speakers. School Psychology Review, 37(3), 391-408.

Fien, H., Park, Y., Baker, S. K., Smith, J. L. M., Stoolmiller, M., & Kame’enui, E. J. (2010). An examination of the relation of nonsense word fluency initial status and gains to reading outcomes for beginning readers. School Psychology Review, 39(4), 631-653.

Fuchs, L. S., Fuchs, D., & Compton, D. L. (2004). Monitoring early reading development in first grade: Word identification fluency versus nonsense word fluency. Exceptional Children, 71(1), 7-21. doi.org/10.1177/001440290407100101

January, S. A., & Klingbeil, D. A. (2020). Universal screening in grades K-2: A systematic review and meta-analysis of early reading curriculum-based measures. Journal of School Psychology, 82, 103-122. doi.org/10.1016/j.jsp.2020.08.007

Vanderwood, M. L., Linklater, D., & Healy, K. (2008). Predictive accuracy of nonsense word fluency for English language learners. School Psychology Review, 37(1), 5-17.

Venezky, R. L. (1999). The American way of spelling. New York: Guilford Press.