One of the most important questions of any intervention or program is when do we know if it works? And can success be sustained? That applies to anything from medicine to classroom teaching.

In the United States, this question is gaining attention as school reform sweeps the nation.  To date, the reforms have been largely disappointing or the improvements fleeting, not to mention the evaluations have frequently sported problems that render the results questionable. To a cynical observer, it seems that nothing is working in the classroom.

But if we’ve learned one thing in the intervening years, it’s that not all learning happens in the classroom. And in a vibrant ecosystem of out-of-school learning opportunities like those on the ground in Pittsburgh (many of which are joining the Hive Learning Network), quantifying what happens in out-of-school time and whether, and how, that learning translates back into the classroom, is critical.

The work, however, is seldom easy. One of the most rigorous assessments of an out-of-school program with teens is Barton Hirsch and colleagues’ evaluation of After School Matters in Chicago. After School Matters has great reach and well-implemented program elements, and it targets a high-need group of teens. The program enrolled nearly 7,500 students in 2009, combining interest-driven classes with a paid apprenticeship.

Unfortunately, and despite high hopes, the results were, as education scholar Greg Duncan put it, “sobering.” The program had little effect on academics or job skills. It did, however, have a nominal impact on behavioral problems and self-regulation.

Despite the disappointing results, the findings underscore the importance of looking broadly at possible outcomes or impacts, beyond just “did the program improve test scores?”

Looking at outcomes such as whether programs improve engagement with school, or reduce juvenile delinquency or problem behaviors, or make neighborhoods safer when kids are engaged and not idle, or make teens feel more connected to positive role models, or even whether programs can improve physical health should be in the purview of evaluators.

And, impacts on positive behavior are important to include as well. Too often research and policy concerns itself with the problem behaviors only. All of these factors contribute indirectly, and perhaps directly, to school outcomes in the end.

One positive behavior that has caught researchers’ eye is a program’s influence on helping youth build an identity. Bill Penuel, professor of educational psychology and learning sciences in the School of Education at the University of Colorado, and director of evaluation research for the Center for Technology at SRI International, stressed the importance of this in an email:

“We want to know what consequences there are for who they are becoming. I think we need to think about these kinds of settings as potentially identity transforming sites, sites where people discover new interests, pursuits, images of who they can become. And then we can ask: How do these sites enable that? How do they help young people gain access to other sites where they can grow even more?”

[blockquote style=”normal”] If students begin to see themselves as web designers or photographers or engineers after taking part in out-of-school opportunities, that new identity may indirectly translate back to the classroom via a newfound excitement about math or literature.[/blockquote]

An art history major at the University of Pittsburgh, for example, told the Pittsburgh Activation Lab, which is conducting a large-scale survey of how kids become interested in and engaged with art, science and technology, that her participation in the Young Writers’ Institute in Grades 4-6 was pivotal, particularly the visit to the Carnegie Museum of Art. The freedom to choose one piece of art to view over an extended period and to write about her impressions was the one thing that focused her attention on becoming an art major.

Becoming involved in the actual planning and direction of the out-of-school programs can also help youth develop their identities and strengthen involvement. University of Pittsburgh’s Tom Akiva is currently mapping the resources and barriers to youth leadership opportunities in Pittsburgh’s out-of-school programs with the hope that his findings can help improve on and build more of these leadership opportunities. The study is in the field now. Read a recent Q&A with Akiva on civic participation.

Although evaluations can be difficult, and the best cost money, they are critical to sustainability. By better understanding how and where programs leave their impression, program designers can continue to improve the offerings.