Let’s Not Let Perfect Be the Enemy of Good: Measurement, MESH, and Continuous Improvement

| Written by Sara Bartolino Krachman, Transforming Education

We are excited to be featured in yesterday's article on social-emotional skills (or “MESH”) on the front page of the New York Times. It's great to see the recognition for this important cause, and we love hearing about teachers like Ms. Cooney who are proactively supporting students in building the skills they need to succeed within school and beyond. Teachers like Ms. Cooney make us even more proud of the work we’ve done so far to support the CORE Districts, which have incorporated social-emotional skills and school culture/climate into their school accountability and continuous improvement system. As with any systems change effort, this has given rise to many questions. We understand the concerns about how MESH measures should be used, and we take those concerns seriously. We also know that part of having a Growth Mindset is committing to putting in the effort, reflecting on what we’re learning, and continuously improving the approach over time. We believe MESH measures aren’t perfect…yet. We also believe that:

  1. MESH skills matter immensely in shaping students’ outcomes in school, college, career, and life.
  2. Given how much time, money, and effort is already being invested to build students’ MESH skills in school, it is responsible to gather outcome data and share it with educators.
  3. The best MESH measures currently available (while still imperfect) can be helpful to both educators and students when used as part of an ongoing continuous improvement process.
  4. Reasonable people differ in their views of whether these skills should be included in formal accountability systems.

The Research is Clear: MESH Skills Matter

We have published the paper Ready To Be Counted to highlight the overwhelmingly strong case that MESH competencies support students’ success in academics, career, and life. The paper goes into significant detail about the predictive value of various MESH competencies, such as self-control and social competence. Included below is a sample of the findings from one of the seminal longitudinal studies in this field, the Dunedin Study, which focused specifically on self-control:

Academics: Even at the first major milestone in academic attainment—completing high school—differences between students with high self-control and those with low self-control were large. While approximately 95% of the top quintile in self-control earned a high school diploma, little more than half (58%) of those from the lowest quintile did so.

Career: Individuals’ levels of self-control in childhood are also powerfully predictive of their socioeconomic status, income, and financial stability in adulthood. For example, while 10% of those who were high in self-control as children lived in poverty at age 32 (earning less than ~$15,000 per year), more than three times as many of those low in childhood self-control (32%) lived in poverty as adults.

Well-being: Children in the lowest quintile of self-control were 2.5 times more likely (27% versus 11%) to suffer from multiple health problems by their 30s. Low self-control also strongly predicted recurrent depression and substance abuse. By age 32, almost half (43%) of those in the lowest quintile of self-control had been convicted of a crime, while barely more than one in 10 (13%) of those in the top quintile were convicted criminals.

This isn’t just about a single study though; there is a broad and compelling research base showing that MESH skills matter immensely and that they can be measured well enough to predict lifelong outcomes.

MESH Measures Provide Helpful Information

More than 90% of teachers believe that MESH competencies are important and teachable. And 88% of teachers report that their school is already implementing some type of program or practice to build students’ MESH skills. Nationally, school systems are spending an estimated $650 million per year, and teachers are spending more than 4 hours per week (worth about $30 billion annually) on activities intended to help build students’ MESH skills. 

Nonetheless, the vast majority of schools currently have no measures in place to provide feedback on whether the time and money being spent on MESH is having a positive impact on student outcomes. As such, we believe it is crucial to give educators a tool to help them understand the impact of these existing efforts. Assessing students’ MESH skills would provide teachers and school leaders with valuable feedback about which approaches seem to be supporting students most effectively, enabling them to double down on these practices and move away from others that prove to be less effective.

The CORE Districts are incorporating survey-based social-emotional measures into their accountability and continuous improvement system for the first time this year, after having piloted those measures with ~10,000 students in 2014 and field tested them with ~450,000 students in 2015. CORE will count the survey-based social-emotional measures as 8% of the total school quality improvement index. Weighting at this level will likely be more relevant for signaling the importance of social-emotional skills than for significantly changing the rating of individual schools. Incorporating these measures into a formal accountability system has been a way to highlight the importance of social-emotional skills and to ensure that data is systematically collected and shared back with every school. These data will not only help educators serve their students more effectively; they will also help us learn much more about what works and what doesn’t with respect to supporting students’ growth.

The large-scale field test we conducted with the CORE Districts has demonstrated strong evidence of the reliability and predictive validity of these measures.  For example, have learned that all four social-emotional skills the CORE Districts are measuring are statistically significantly correlated with students’ academic and behavioral outcomes.

Data from a field test with approximately 450,000 students

Data from a field test with approximately 450,000 students

Moreover, CORE is far from alone in exploring how to incorporate measures of these skills into how we think about student success and growth. PISA (the international test from OECD) and NAEP both include and present comparative data on MESH measures, and ACT and ETS use similar measures as well. These organizations are full of experts in educational assessment and psychometrics, and they have published significant evidence of the validity and reliability of their work using survey-based measures of SE skills.

Not all Accountability Systems are Created Equal

It’s clear that the word “accountability” means different things to different people. When it comes to accountability, one of the most relevant questions is what the consequences are for schools identified as lower-performing. In the case of CORE, the consequences are quite intentionally supportive rather than punitive.  Specifically, low-performing schools are paired with higher-performing schools, which provide support and mentorship to help build educators’ capacity and improve student outcomes over time. Perhaps most importantly, there are no consequences attached to the survey-based MESH measures for individual teachers or students.

With the passage of the Every Student Succeeds Act (ESSA), the federal government has empowered states to develop their own accountability systems. ESSA also acknowledges that schools and students are more than just test scores by requiring states to choose at least one other measure of school quality or student success as part of their state-determined systems. Furthermore, ESSA gives states the most latitude they’ve had in recent years to determine what the consequences of their accountability systems will be. We hope that states will use these new flexibilities to articulate a broader, more holistic definition of student success and school performance. We also hope that states will find ways to prioritize MESH skills and assess these skills systematically so that educators have access to the data they need to make informed decisions about how to support students most effectively. We think there are multiple ways to accomplish this goal, including incorporating these measures into a state-determined accountability system or collecting the data systematically for formative purposes only.

We understand that people have questions and concerns about how attaching even low stakes to survey-based MESH measures will play out in practice. Based on CORE’s thoughtful, phased approach to this work, we will have empirical evidence as early as this summer about what happens when (low) stakes are attached to MESH measures. If we see a significant increase in students’ self-reported MESH skills between school year 2015 (when no stakes were attached) and school year 2016 (when low stakes will be attached at the school level), then we’ll know that further exploration is needed to ensure that these measures aren’t being misused.

As we think about the next generation of accountability, it’s important to acknowledge that the reading and math tests we’ve used for decades are also imperfect. But, we still use them because we need a measure to determine the efficacy of our instructional methodologies and to know when to intervene to support students’ academic development.  New assessments are never perfect, but they improve with testing and iteration.  This happens by implementing assessments, evaluating their effectiveness, and getting feedback from students and practitioners to improve those assessments over time.  If we waited until measures were perfect whether for English/language arts, math, or MESH, we would have no data to draw upon to inform instruction, capacity-building for educators, or supports for students.

Given what we know about the importance of MESH skills for students’ lives, we can’t afford to wait for perfect measures. These skills matter: longitudinal research and a recent field test with almost half a million students have shown that they can be measured in valid and reliable ways. And perhaps most importantly, educators are already acting on their belief that these skills are important to teach in school. At TransformEd, we look forward to continuing to support educators and education systems in using the most recent research to assess and develop the MESH skills that we know students need to succeed in college, career, and life. We approach this work with optimism and rigor, but also with great humility and a deep commitment to learning from both the educators and the researchers we work with along the way.