A network can be a great example of when more really is more.

A practice becomes more powerful when it is implemented in multiple settings. An individual has a greater impact when she can share advice and resources with others.

So in the education world, networks—cross-sector groups of educators, schools, community programs, and businesses working in tandem toward a shared goal—are catching on, and it’s no surprise.

Networks “allow ideas that are isolated, either in a classroom or community space, to scale, through sharing, collaboration, communication, and iteration,” said Anne Sekula, the director of the Remake Learning Council. The council represents the Remake Learning Network, a group of formal and informal education organizations working to increase opportunities for all young people in the greater Pittsburgh region.

Yet while members of networks sing their praises, the benefits of a network are hard to systematically track. Take “sharing” in all its manifestations—how do you count that, and how can you tell that it improves learning outcomes? As education networks proliferate, many are wrestling with the question of how, exactly, to measure networks’ unique impact. What data should be collected, and through what methods? And perhaps most important, why measure in the first place?

“That’s the million-dollar question,” Sekula said.

In its 10 years of existence, Remake Learning has focused on providing information and resources that are more immediately practical, she said. That could be money for community programs, research on the use of classroom tools, or conferences. The network has not yet invested heavily in measurement, a deeply complex and expensive task.

“It’s a tremendous set of resources that we don’t want to lightly direct away from educators and impact on programs,” Sekula said.

But at 250 members and growing, the network is now considering the benefits of self-assessment.

“We at Remake Learning have said, ‘We know we have something that’s strong, but we also want to continually improve and expand that work,’” Sekula said. To do so, the network must take a close look at its strengths and shortfalls.

What would that entail? There are a couple of different, yet interconnected, approaches to measuring what a network is and does.

Remake Learning Network members gather annually. Photo/Ben Filio

Networks need to look in the mirror

“If you’re interested in networks,” said Jennifer Russell, a researcher at the University of Pittsburgh School of Education, “you want to have ways of tracking the health of the network itself.”

This type of “process measure” assesses the strength of a network—how connected its members are—rather than measuring the outcomes of its work.

Social network analysis comes in handy here. The process involves mapping a network—determining who is connected to whom, and how. Social network analysis is used in many fields, whether researchers are tracking the spread of a disease, mapping an online social media network, or studying the structure of an education network.[pullquote]What data should be collected, and through what methods? And perhaps most important, why measure in the first place?[/pullquote]

Evaluating the health of an education network is necessary, said Julie Stolzer, director of marketing and consulting at the Teaching Institute for Excellence in STEM. Mapping a learning network can reveal whether everyone working in the field knows one another and whether available resources are being used to their utmost. Stolzer has seen firsthand how that often isn’t the case until an intentional network is established and assessed.

Stolzer is one of the facilitators of the STEM Ecosystems Initiative, a program developing regional STEM learning networks in over 30 communities. Part of her job is to survey each community to get a sense of existing and potential relationships—as well as where relationships may not yet have been formed. In every single case, she said, at least one educator has insisted that everyone working in STEM education locally already knows one another. And in every single case, there is a different educator who insists there is no local STEM program.

“That was a big ‘aha’ moment,” Stolzer said. Qualitative self-assessment can help identify gaps, forge new partnerships, and include those previously left out of the loop.

Photo/Ben Filio

In Pittsburgh, an evaluation of the local maker education movement also demonstrated the importance of mapping the ties in a network. The analysis revealed the critical links—those people and programs that serve as “connective tissue” between others, Sekula said.

“You find these nodes of connectivity you can really mine,” she said. “You might be surprised to find one small organization that’s highly networked and an important player, but maybe they don’t get the resources they need to do that.”

Surveying members or mapping the connections in an existing network can help those critical actors expand their reaches.

“We’re trying really hard not to reinvent the wheel, but to figure out the gems that already exist,” Stolzer said of her work on the STEM Ecosystems Initiative.

But it is one thing to take the temperature of a network. It is a more difficult task to determine whether all its connection and collaborations are helping the students it serves.

Photo/Ben Filio

New kinds of measurement for new kinds of learning

“Student-level impacts are a hard thing to get your hands around,” Sekula said.

First of all, most programs in the Remake Learning Network intend to boost student engagement, equity, and innovation—and other outcomes that may be immeasurable through traditional methods like standardized tests or AP enrollment.[pullquote]”You find these nodes of connectivity you can really mine.”[/pullquote]

In Russell’s experience, one of the hardest and most critical steps in measuring student impact—of a network or otherwise—is coming up with a highly specific problem of practice. A project she worked on, for example, explored why many students were not progressing from a community college onto university or a career. But it was impossible to measure something so broad as “success after community college.” The researchers eventually figured out that many students were getting held up by a mandatory developmental math course. The lack of success in that math class became the problem of practice. The task then, said Russell, was to drill down to measurable indicators—ones that tracked long-term goals (passing the class) as well as short-term (attendance or participation in class).

“That’s a process itself—identifying those achievement outcomes that a specific network or community values,” she said.[pullquote]Measurement may be the key to the sustainability of a network.[/pullquote]

Often, the different sectors represented in a network initially have different goals and ideas about how to track them. Stolzer observed that dynamic in one of the new regional STEM networks. At an early meeting, a teacher said she thought test scores were the only way to measure student achievement. One of the business partners countered that a measure of career readiness would most accurately capture achievement. The STEM Ecosystems Initiative eventually developed a set of success indicators for all of the regional networks to use. The metrics—ranging from measures of parent engagement to indicators of professional development opportunities—are designed to apply to multiple groups’ visions of success.

Even with the right metrics, tracing those outcomes back to the network adds another layer of complexity.

The collaboration intrinsic to networks turns out to also be a barrier to measurement. Perhaps students at one of the maker afterschool programs in the Remake Learning Network become demonstrably more engaged in their creative work. Or say attendance at a career training program skyrockets. Can those outcomes be attributed to the network, versus to the autonomous programs?

Take a recent pilot study of Remake Learning summer program providers. The researchers tracked participation by demographics, interest level, and other student outcomes. But “this study wouldn’t reveal network impacts other than anecdotally,” said Carnegie Mellon University’s Marti Louw, the lead researcher. “Those kinds of studies require a much larger investment in research and evaluation.”

At the end of day, measuring networks is a challenge, but one that most involved are increasingly finding necessary—whether for the sake of self-improvement or to communicate successes to funders. Measurement, then, may be the key to the sustainability of a network and its capacity to make a difference where it matters most: for learners.