Our ratings at The School Review are comprised of three components: Academics, Learning Environment, and Safety. While we recognize any metric is inherently subjective—depending on the various indicators chosen and the process by which these variables are combined—our rankings and grades allow for the comparison of neighborhood schools across a variety of characteristics. Ultimately, we believe our site provides equitable and transparent information that can help students and parents find the best school to fit their desired needs.
Academics. Since the passage of the No Child Left Behind Act (NCLB) in 2001, states have administered various standardized assessments to qualify for federal funding. These assessments are intended to establish minimum expectations for what students should know at the end of grade levels and subject areas. There are, however, noteworthy differences in how states choose to provide their assessments and report results.
First, states vary considerably in the quantity and academic subjects in which they administer standardized tests. As one example, California assesses students in grades 3-8 in only math and English language arts (ELA)/literacy using the California Assessment of Student Performance and Progress (CAASPP). By contrast, Virginia’s Standards of Learning (SOL) assessments include math and reading tests in grades 3-8 as well as tests in Virginia studies, civics and economics, science, and writing. Similarly, at the high school level, California only assesses students in math and ELA/literacy in grade 11. Conversely, there are 11 subject-specific SOL tests administered to Virginia high school students, including in world history, algebra, and biology.
There are also important differences in how states choose to report their standardized tests results. States make available a school’s pass rate on each assessment with varying levels of detail. Most states report assessment results in terms of the percentage of students falling within each achievement classification (e.g., “needs improvement”, “basic”, “proficient”, “advanced”). Though typically four, the number of achievement classifications can range from three to five assessment levels. Other states report only the percentage of students who passed the test (i.e., classified as “proficient”, “met expectations”, “met standards”, “on track”, “at grade level”). For nearly all states1, we base our analysis on the percentage of students who have passed the test, and collapse achievement classification categories where applicable.
There are several other ways that states differ in how they chose to report their standardized test results. Many states redact test results when the number of students completing the test is below a predetermined sample threshold (e.g., less than 20 students). Some states report test participation rates; other states do not. Some states report results by grade level; other states report only aggregate school results. And while most states report the total number of students completing a test at each school, other states report only proficiency percentages. We make every effort to present results as accurately as possible. This includes weighting results (when possible) across grade levels and tests by the number of students completing the assessment, to ensure each test result is counted equally.
We also confirm test completion is uniform across schools in a state before including the results of a specific test in our analysis. For instance, the Regents Geometry and Physics assessments were only completed by a subset of schools in the New York state in 2016-2017. Consequently, the distribution of these tests results may not be representative of all high schools in New York. Higher or lower achieving schools may be more or less likely to administer either of these exams thereby creating a selection bias. Thus, we do not include these assessments, or the results of any assessments not completed by all (or nearly all) elementary, middle, or high schools in a given state.
Finally, to reduce performance fluctuation from year-to-year, we include up to three years of state test score results. Greater weight is assigned to the test scores from more recent school years. For states having three years of test score data, the weights from most recent to least recent administration are as follows: 44%, 33%, 22%. For states making available only two years of test score data the more recent year is assigned 60% of the weight and the older year is assigned 40%.
In addition to state administered assessments, we include indicators of college preparedness as part of the Academics component for middle schools and high schools. Specifically, the Academics rating for middle schools includes the percent of students who passed Algebra I in 8th grade. Research shows algebra completion serves as a “gateway” to more advanced math courses and high school graduation (Atanda, 1999; Silver, Saunders, & Zarate, 2008). At the high school level, the Academics rating includes the percent of students enrolled in at least one Advanced Placement (AP) or International Baccalaureate (IB) course as well as the percent of students taking the SAT or ACT. These indicators of college preparedness and likelihood of future higher-education enrollment are from the 2015-2016 Civil Rights Data Collection (CRDC).
Learning Environment. We evaluate a school’s learning environment using five variables: the ratio of staff personnel to students at a school, the percent of teachers who are novice2, the percent of teachers who are uncertified, the percent of teachers who are chronically absent3, and the percent of students who are chronically absent4. These variables are computed using 2017-2018 data from the National Center for Education Statistics (NCES) and 2015-2016 data from the Civil Rights Data Collection (CRDC).
Safety. We measure the safety of an elementary school using three variables: the percent of students at the school receiving one or more in-school suspensions, the percent of students receiving one or more out-of-school suspensions, and the reported incidents of physical altercations at a school per 100 students. Each of these variables is computed using data from the 2015-2016 Civil Rights Data Collection (CRDC).
Once collected and computed, we standardize each test result or variable across the distribution of schools of the same type or level in the state5. In other words, elementary schools are compared only to other elementary schools, middle schools are compared to only other middle schools, and high schools are compared to only other high schools6 . We use the designations provided by NCES in 2017-2018 to determine school type.
Standardizing test results or variables across only the distribution of schools of the same type is needed given that variables can differ substantially across level. For example, the average percent of students chronically absent is 13% in elementary schools, 15% in middle schools, and 26% in high schools. Moreover, comparing only schools of the same type is necessary when an identical standardized assessment is administered in multiple grades. For instance, we compare a given middle school’s test result on Algebra I only with other middle schools and a given high school’s test result on Algebra I only with other high schools. This ensures we do not compare the pass rates on the same assessment of two distinct populations (i.e., eighth graders versus ninth graders).
To create the variable or test result classification labels, we then categorize standardized scores into quintiles. For a given variable, elementary, middle, and high schools in the bottom 20% of their respective school type distribution in their state receive a “Poor” label, schools in the second quintile receive a “Fair” label, schools in the third quintile receive a “Good” label, schools in the fourth quintile receive a “Very Good” label, and schools in the highest quintile receive an “Excellent” label. This categorization process is repeated for all variables.7
After each variable is standardized by school type and assigned a quintile label, we average across variables of the same domain to compute the three component measures Academics, Learning Environment, and Safety. To better understand this aggregation process, the results of Hutchison Farm Elementary School in Loudoun County, Virginia are presented as an example.
First, to create the Academics component measure for Hutchison Farm Elementary School, we average across the standardized assessments. There were eight Standards of Learning (SOL) tests administered in Virginia elementary school in each of the three most recent school years for which data is available: Grade 3 Reading, Grade 4 Reading, Grade 5 Reading, Grade 3 Math, Grade 4 Math, Grade 5 Math, Virginia Studies, and Science. Hutchison Farm School’s most recent average standardized assessment scores are 0.38, 0.51, and 0.27. This means that, in each of the last three years, students at this school scored a little less than half a standard deviation higher on the SOL tests than the average elementary school in the state. After aggregating annual test results and then weighting by year (44% for 2016-2017, 33% for 2015-2016, and 22% for 2014-2015) we find Hutchison Farm Elementary School’s average assessment score ranks in the 66th percentile of elementary schools in Virginia8.
Next, in a similar aggregation process, we average across the standardized results of the five Learning Environment variables. Hutchison Farm Elementary School has a ratio of staff personnel to students of 17 to 1, four percent of its teachers are novice, none of its teachers are uncertified, nine percent of its teachers are chronically absent, and six percent of its students are chronically absent. After standardizing each of these values and taking an average score we find Hutchison Farm Elementary School’s Learning Environment ranks in the 94th percentile nationally of elementary schools.
Finally, we average the standardized results of the three Safety variables. Hutchison Farm Elementary School had eight reported incidents of physical altercations per 100 students, one percent of students received one or more in-school suspensions, and one percent of students received one or more out-of-school suspensions. Once we standardize each of these values and compute an average score we find Hutchison Farm Elementary School’s Safety ranks in the 65th percentile nationally of elementary schools9.
Elementary, middle, and high schools are assigned normative grades on each of the three component performance measures (Academics, Learning Environment, Safety) depending on where in each distribution the school’s standardized scores fall10. Specifically, letter grades are assigned to schools according to each decile. Schools in the top decile on a component measure (top 10%) receive a grade of A+, schools in the second decile (81-90%) receive a grade of A, schools in the third decile receive a grade of A-, schools in the fourth decile receive a grade of B+, schools in the fifth decile receive a grade of B, schools in the sixth decile receive a grade of B-, schools in the seventh decile receive a grade of C+, schools in the eighth decile receive a grade of C, schools in the nineth decile receive a grade of C-, and schools in the bottom decile (bottom 10%) receive a grade of D. Because Hutchison Farm Elementary School ranks in the 65th percentile in Academics, 94th percentile in Learning Environment, and 56th percentile in Safety it receives grades of B+, A+, and B, respectively.
Table 1: School Grade Categories for Academics, Learning Environment, Safety, and the Overall Measure
|A+||96-100||Top 25% are in the A range|
|B+||66-75||The next 40% of schools are in the B range.|
|C+||26-35||The next 30% of schools are in the C range.|
|D||0-5||The bottom 5% of schools are in the D range.|
Lastly, we compute each school’s Overall score by averaging the scores of their three component measures using the following proportional weights: Academics (60%), Learning Environment (25%), and Safety (15%) (see Figure 1). Using the grade categories shown in Table 1, we then assign a normative Overall grade to each elementary, middle, and high school.11 Grades are based on where in their respective school type distributions the Overall score falls. We also impose eligibility constraints for the highest and lowest rated schools.12 Combining the component scores associated with Hutchison Farm Elementary School’s Academics grade of B+, Learning Environment grade of A+, and Safety grade of B yields an Overall component score. This score ranks Hutchison Farm Elementary School in the 77th percentile nationally, which corresponds with the Overall grade of A-.
Atanda, R. (1999). Gatekeeper courses. National Center for Education Statistics, 1(1), 33.
Silver, D., Saunders, M., & Zarate, E. (2008). What factors predict high school graduation in the Los Angeles Unified School District. Policy Brief, 14.
Soloman, S. R., & Sawilowsky, S. S. (2009). Impact of rank-based normalizing transformations on the accuracy of test scores. Journal of Modern Applied Statistical Methods, 8(2), 9.
Texas only reports the percent of students classified as “approaching grade level or above”. ↩
Novice is defined as a teacher in their first or second year. ↩
Chronically absent in the context of teachers is defined as a teacher missing more than 10 days during the school year. ↩
Chronically absent in the context of students is defined as a student missing at least 15 days during the school year. ↩
Before standardization, mathematical transformations (e.g., square root, natural log) were applied to several heavily skewed variables to create more normal distributions across schools. In addition, all Safety and Learning Environment variables were reverse-coded given that greater values for these variables indicates lower school quality. ↩
We also standardize tests results across grade where applicable. ↩
The standardized scores for the variable percent of teachers who are uncertified are not categorized into quintiles as majority of U.S. schools report having no uncertified teachers. Instead, the quintile labels for this variable are as follows: If 5% or less of teachers are uncertified, a school receives an “Excellent” label, if 6-15% of teachers are uncertified, a school receives a “Very Good” label, if 16-30% of teachers are uncertified a school receives a “Good” label, if 31-50% of teachers are uncertified a school receives a “Fair” label, and if more than half of teachers are uncertified a school receives a “Poor” label. ↩
For middle and high schools, we also include indicators of college preparedness as part of the Academics component. These variables comprise 17% (10%/60%) of a middle or high school’s Academics component measure and 10% of their Overall score. ↩
We treat missing data in the following ways. If a school does not have standardized test scores, it does not receive an Academics score. If a middle or high school does not have college readiness data, its Academics score defaults to its standardized test score. If a school is missing data for two Safety variables or three or four Learning Environment variables, the weights assigned when computing the Overall composite score decrease as follows. The Safety component measure for a school with only one variable is assigned a weight of 10% instead of 15%. For the Learning Environment component measure, a school is assigned a weight of 10% with only one variable and a weight of 20% for two variables (instead of 25%). In such situations, the weight originally assigned to the missing component measure are allocated proportionally to the other two measures. If a school is missing data for one Safety variable or one or two Learning Environment variables no changes are made to original weights assigned when computing the Overall composite score. ↩
Because states administer different assessments, a school’s standardized test performance is compared only to other schools of the same type in the state. For all other variables, including Safety and Learning Environment measures, a school’s results are compared nationally to the distribution of all schools of the same type. ↩
We do not compute an Overall score if a school does not have an Academics score. However, we do compute an Overall score and assign an Overall grade to schools missing a Learning Environment or Safety score. In such situations, the weight originally assigned to the missing component measure is allocated proportionally to the other two measures. Thus, the Overall score for a school missing their Safety score is weighted 71% Academics and 29% Learning Environment and the Overall score for a school missing a Learning Environment score is weighted 80% Academics and 20% Safety. If a school is missing both their Learning Environment and Safety scores, their Overall score defaults to their Academics score. ↩
In addition to having an Overall component score in the top 5%, a school must have grades of B or higher across all three subcategories to be eligible to receive an A+ grade. Similarly, a school must have an Overall component score in the bottom 5% as well as grades of C+ or lower across all three subcategories to receive a D grade. ↩