District Responds to Report on Middle School Test Scores

A report on test results at District 200’s four middle schools recently released by Educate 200, a small group of residents who oppose replacing Hubble Middle School with a new school on another site, misrepresents the facts and uses an unconventional approach to data analysis.

The state tests analyzed in the group’s report are designed to measure a student’s knowledge of expected learning outcomes in each of the tested areas.  Results are reported in percentages of students who exceed, meet, or do not meet the standards identified for their grade in school.  Each student’s score reflects his or her knowledge of the standards. The score does not indicate the student’s standing among the entire tested group.

Educate 200 reports that Hubble students rank number one among the four middle schools in total growth, overall achievement growth, and ISAT growth over a two-year period.  To make this statement, the group compared scores of the elementary schools that feed into the middle schools to the middle school scores.  This is not a valid comparison for the following reasons:

• The student demographics of the elementary schools do not mirror those of the middle schools they feed into.
• Their analysis is not a cohort study (a defined population group followed prospectively), but one of students at different ages and grade levels.  The state, in fact, has not provided the data base to allow districts to conduct cohort studies.
• The analysis duplicates the weight of ISAT scores, while also including the scores on different tests given to special education students and students who speak little or no English.
• Two years of data does not constitute trend data, especially when cut scores and rubrics for some state tests changed during that time.

According to data provided by the state, Hubble’s overall performance on state tests was third highest among the four middle schools in 2004 and second in 2005.  On only the ISAT, Hubble ranked third highest among the four middle schools in 2004 and 2005.  The ranking for Hubble or any of the other three middle schools does not mean one school is better or worse than any other.  The demographics of the schools are different and the students tested each year are different. 

The Board of Education is planning to build a new school to replace Hubble to give students a facility that is designed to support the middle school program and to give them a building that is comparable to the other three middle schools.  Building a new school also saves money because it is less expensive to build new than to renovate to the same standard as other District 200 schools and it is less expensive to operate. 

Following is a more detailed response to Educate 200’s study of middle school test scores:

Recently, a report entitled Hubble Huskies:  Leading the Pack in Learning was posted by the EDUCATE200Foundation. The report is designed to highlight the achievements of the students attending Hubble Middle School during two school years, 2003-2004 and 2004-2005.  The Executive Summary of the report states:  “District 200 student achievement test results show that Hubble Middle School ranks number one among the four District 200 middle schools in Total Growth, Overall Achievement Growth, and ISAT Growth achieved by its students, based on current (2004-2005) test scores.”  We recognize and embrace the accomplishments of our students attending Hubble Middle School; however, the achievement information relative to the academic achievements of all District 200 middle school students requires correction and clarification.

While the report indicates that the published state test scores support the statement that Hubble Middle School ranks number one among District 200’s four middles school, the methodology applied to the data to make this determination threatens the integrity of the statement. The report provides a middle school basic achievement rank table, described as “raw” test scores ranking the performance of District 200’s four middle schools.  The report proceeds to provide an additional analysis of the state’s data by comparing the feeder elementary schools’ performance data to performance data at each of the four middle schools.  The report indicates that the thirteen elementary schools were grouped by the middle schools attended by the elementary level graduates.  Scores were reported for ISAT and overall achievement, over a two-year period. 

A basic premise of the report is that the demographics of feeder elementary schools mirror the demographics of the middle schools serving the students of the elementary schools; in fact this is provided as a justification for the use of elementary data as a comparative basis for middle school achievement levels.  This premise is not true for District 200 due to the District’s commitment to center certain student educational programs.  The centering approach is used for our direct-serve English Language Learner Program and selected special education programs at the middle school level.  As a result, the demographic profile of feeder elementary schools does not universally predict the demographic profile of the middle schools in District 200 and in fact, is very different in some cases.  A second premise of the report is that current state data provided for elementary schools (which includes data for students assessed in third, fourth and fifth grades) can be used in this manner to assess achievement and achievement growth when compared to current state data for students in middle schools (which includes data for students in seventh and eighth grades).  The state has never recommended or supported the use of state data in this manner.  State tests are standards-based criterion-referenced assessments designed to assess student achievement of specific subject content.  This is not a cohort study; to date the state has not provided the necessary data base for districts to conduct cohort studies.  Additionally, this report uses the overall performance data as well as overall ISAT performance data as the basis for the calculations.  Overall performance data represented in this report includes all state assessments, IAA (Illinois Alternate Assessment for designated special education students), IMAGE (Illinois Measure of Annual Growth in English for designated English Language Learners) and ISAT.  The overall ISAT performance data includes all ISAT scores for administered state assessments.  Therefore, the ISAT scores are counted in both the overall performance data and the ISAT performance data, duplicating the weighting of ISAT data.  The integrity of the data is compromised when used in this manner.  Finally, the report references that test scores, when compared across two years and two tests per school, minimize the effect of outliers and are a reasonable proxy for academic achievement.  As indicated, these scores represent more than two assessments.  Also, two years of data cannot be considered trend data especially in light of the fact that the cut scores and rubrics for some of the state assessments changed during the testing window referenced in this report.

Indicated below, is the overall performance data for each of the four middle schools for school years 2003-2004 and 2004-2005 as presented by the State.  We have not modified or changed the statistical representation of the facts.  We have included the overall performance data which includes all state assessments as well as the overall ISAT performance data.

As indicated by the data provided directly from the state, without any manipulation of the information, 2004 data indicate that 75.3% to 89.1 % of District 200 middle school students met or exceeded state standards on all of the state assessments and 2005 data indicate that 86.3% to 90.7% of the students met or exceeded standards on all of the state assessments.  Overall ISAT performance during 2004 indicates that 82.2% to 89.3% of District 200 middle school students met or exceeded state standards and in 2005 86.3% to 91.1% of the students met or exceeded state standards.

The percentage of students meeting or exceeding state standards between 2004 and 2005 has decreased as much as 1.4% as defined by overall performance on all state tests and increased by as much as 11.3%.  The percentage of students meeting or exceeding state standards between 2004 and 2005 has decreased by as much as 1.4% as defined by overall performance on ISAT and increased by as much as 7.7%. 
 
Points for Consideration

• The intent of their “analysis” was to evidence that when given the feeder elementary schools’ performance, Hubble’s performance indicates greater levels of growth than the other middle schools
• Their premises included that comparing the middle schools in isolation is not fair since the demographic profiles are different – comparing their scores to feeder elementary schools is a way to evidence growth
• They ignored that we center ELL and certain special ed programs – demographic profiles at middle schools do not mirror elementary schools
• They averaged percentages of overall performance on all state tests and overall performance on ISAT tests to determine percentages – state has never recommended using the data this way or to use “level” data to predict student performance at a different level
• Middle school data, presented using state data, indicates that in fact, Monroe Middle School has enjoyed the greatest gains during the last two years.
• Three of our four middle schools have evidenced gains as measured by the overall performance data and ISAT performance data.
• The process used to determine their “growth” numbers is as follows:
•  Each feeder elementary school had their 2004 and 2005 overall performance data and ISAT performance data “averaged”.  These two averages were “averaged”.
• The feeder elementary schools combined averages were averaged again to determine a summary average.
• The middle school 2004 and 2005 overall performance and ISAT performance data were averaged in the same manner as the elementary schools. 
• The middle “average” was compared to the elementary feeder school “average” generating the “growth” numbers.
• Here is an actual mathematical example:

The data for this table is taken directly from Appendix B of their report.  * Only 50% of the students from Lowell attend Hubble.