On July 5th the Arkansas Department of Education issued a press release and accompanying social media posts announcing the release of the 2018 ACT/Aspire Results. The headline on the release was:
Arkansas’ ACT Aspire Results Show Growth; ACT Results for Grade 11 Hold Steady
I could not wait to dig in and see all of the progress our schools were making.
The release continues with a brief summary of the results, reproduced here for your befuddlement:
Overall, the percent of students at or above ready in English remained steady or trended upward in all grades except for grades 6, 9 and 10. Reading results at or above ready trended upward in all grades except for grades 6, 9 and 10. For 2018, overall writing scores were not reported; however, writing scores were incorporated in the overall English Language Arts scores.
Arkansas’ percent of students at or above ready in math trended upward in grades 3 and 8 through 10 and downward in grades 4 through 7. Science scores at or above ready remained steady or trended upward in grades 3, 8 and 9 and downward in grades 4 through 7 and grade 10.
If you are anything like me, you will have to make multiple trips through those last two paragraphs if you care to actually understand what the ADE is trying to say.
My next thought was something along the lines of: “It’s really too bad that they didn’t have a nice table or graph so we could actually see the growth that the headline announces.”
I clicked open the Excel files the department provided. Well that’s weird, they don’t actually have any charts that compare this year’s results to last. It is especially odd since last year’s results were accompanied by tables summarizing the results and growth from the previous year.
What in the world? The Excel sheets released do not contain the measure that is continually referenced in the ADE “summary” of the results, “Percent of Students at or above Ready.” It suddenly felt as if whoever put this out, doesn’t actually want to give the public any real information about how the ACT/Aspire went.
I decided to build my own table that was similar to the one provided by the ADE with the previous year’s release of the scores. I added the colors so it would be easy to determine if the change in scores were positive or negative at a glance.
Figure 1. Percent Change of Students Who Met Readiness Benchmarks from 2016-17 to 2017-18
The ADE’s confusing summary of the scores, while technically correct once they are parsed and compared to the above chart, certainly don’t support the headline: Arkansas ACT/Aspire Scores Show Growth.
What would be the appropriate headline for results based on looking at the table above? Perhaps something about how the results were mixed, showed no obvious improvements, or maybe even something about the missing writing scores. Maybe it should have been about the change in the cut scores for STEM and ELA Readiness Indicators, the latter of which will have a significant impact on how schools are looked at on the new state accountability plan.
When any reasonable person looks at the table above, they quickly and correctly conclude that the results are at very best, mixed. In fact, several news organizations that picked up the press release recognized this and didn’t run the suggested headline as provided by the ADE, while some other smaller and/or less local news outlets ran the headline as is, adding to the misconceptions about the actual results.
What would be another way to look at overall change in scores on the ACT/Aspire based on the data that the ADE has provided?
The Office of Education Policy at the University of Arkansas in Fayetteville released the graph below via their blog on July 12th.
Figure 2. Percentage of Arkansas students meeting or exceeding expectations on the ACT Aspire, by content area, 2016 to 2018.
Overall, the percentage of students meeting or exceeding benchmarks stayed exactly the same in mathematics, science, English, and reading.
The ADE’s press release continues with a quote from Commissioner Johnny Key, which, similar to the misleading headline, paints a picture of the results that is simply not supported by the evidence:
The increase in overall growth reflects the hard work and dedication of Arkansas’ students and teachers,” Arkansas Department of Education Commissioner Johnny Key said. “The results reflect educators’ commitment to teaching strong, rigorous education standards and the positive effects of the R.I.S.E. Arkansas and Computer Science initiatives. The results show we still have work to do to improve education in Arkansas, but I am confident that as we move toward implementation of Arkansas’ Every Student Succeeds Act plan this fall, we will lead the nation in student-focused education.
The entire quote is based on the conclusion that there was an “increase in overall growth” in the test results. Perhaps the conclusion is drawn from some data that was not presented, or some other report that the ADE has seen but has not been released. If that is the case, hopefully the ADE will share it with those of us who are feeling misled.
Not coincidentally, the Commissioner’s quote also makes it clear that we are to attribute the good news of the “increase in overall growth” to the hard work of the students and teachers, and the “positive effects of the R.I.SE. Arkansas and Computer Science initiatives.” This is why it is really important to get this stuff right. If there is there is no “increase in overall growth” should we also assign blame where we used to give credit?
Any discussion about how to improve education in Arkansas must begin with using good measures of student achievement, then talking clearly and precisely with each other about the what the results honestly say. We can save the argument about whether ACT/Aspire is a good measure of student achievement for another time, but is it too much to ask that the ADE provide evidence to support their conclusions around the ACT/Aspire results? Not only is it the correct way talk about the state assessment, it would model the Arkansas Writing Standards for the students who struggled with the singular ACT/Aspire Writing Assessment item.