Perpetual decline

I have come to notice that the Black-White  gap is in a state of perpetual decline. Every couple of years a report comes out showing that the mean score difference has narrowed and, yet, every time I compute the standardized difference, the difference comes out to about 1 +/- 0.2 SD. As example of this score narrowing, Steve Sailer recently pointed to the newly released 2011 PIRLS reading results. The 2011 PIRLS reading gap is only 53 points — on a metric that has international standard deviations of 100. This is down from 57 points in 2006 and 63 in 2001!

Yet when I compare the standardized gap between 2001 and 2006 I find exactly no decline, because while the gaps declined from 63 points to 57 points, the pooled standard deviations also declined from 74 to 68 (i.e., the variance was decreased). Unfortunately, I have been unable to find standard deviations for 2011 — so I can’t determine if there was a narrowing in the standardized difference or  a narrowing in variance. The pooled SD would have had to decline to 63 in 2011 from 68 in 2006 for the standardized difference to remain 0.84 given the score difference (of 53 points). A safe bet would be the the 2011 pooled B/W standard deviations were similar to those in 2001 and 2006 (mean 71) and so that the standardized difference was 0.74 SD; if so, the gap would have declined 12% in the last 10 years.

PISA 2001, 2006, 2011

This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Perpetual decline

  1. JL says:

    The overall SD in the PIRLS 2011 US sample is 73. In 2006, the overall SD was 74 while in 2001 it was 83. The B-W SDs were 7 percent lower than the overall SDs in 2001 and 2006. This would suggest that the 2011 B-W SD is 68 and the B-W gap 0.78 SD, a reduction of 7 percent since 2011. However, if there are more Hispanics and Asians in the sample than before you’d expect the difference between the overall and B-W SDs to be larger than before.

    The PIRLS is probably less g-loaded than the NAEP tests. This is what it says in the US report:

    In examining passage length and difficulty, PIRLS 2011 passages were shorter on average than the NAEP 2011 passages. Readability analyses indicate that, on average, the PIRLS 2011 passages were about one grade level lower than the NAEP 2011 passages.

    Item-by-item content showed some differences between the assessments. About half of the PIRLS 2011 items were mapped to the NAEP “locate and recall” cognitive target. Most of the remaining PIRLS 2011 items were mapped to the NAEP “integrate and interpret” cognitive target. Very few items were mapped to the NAEP “critique and evaluate” cognitive target. By contrast, NAEP 2011 had more items to assess the “integrate and interpret” as well as the “critique and evaluate” cognitive targets than did PIRLS 2011.

    In summary, there are distinctive differences between PIRLS 2011 and NAEP 2011. Overall, these differences suggest that the NAEP 2011 reading assessment may be more cognitively challenging than PIRLS 2011 for U.S. 4th-grade students.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s