WNET and Teaching Channel: Measuring the Efficacy of Video-Based PD for Engineering Instruction

December 6, 2016

“We’re talking about a child’s future. We need to respond fast with a long term, slower, deeper approach.” -Matty Lau, 100Kin10’s Research and Innovation Fellow

When facing a problem as urgent as the lack of high quality engineering learning in K-12 education, there isn’t time to wait for lengthy scientific studies. An urgent problem requires immediate attention, as well as a collective, purposeful approach, like a Networked Improvement Community (NIC).

Simply put, improvement science is designed to accelerate learning-by-doing.

Launched in January 2016, 100Kin10’s first NIC was an experiment, a program designed for 100Kin10 partners by Dr. David Kanter,, 100Kin10’s first Director of Research and Innovation. He was inspired by the Carnegie Foundation’s NIC model of improvement science coupled with networked communities to continuously improve solutions to complex problems in education. David adapted and brought NICs to the network “to help 100Kin10 partners tackle their shared STEM challenges more quickly and effectively.”

Open to the entire network, 11 different 100Kin10 partner organizations signed up for the four-month process which brought partners together to carefully dissect issues, uncover data and design solutions to a systemic problem: lack of engineering instruction in K-12 classrooms. The 100Kin10 NIC process started virtually, via two webinars where partners were trained in foundational, structured NIC exercises, to help them focus on specific elements of the larger problem.

Like most 100Kin10 NIC participants, neither David Wisnieski of WNET nor Tom Jenkins of Teaching Channel had ever heard of a NIC or its technical jargon. Still, both organizations saw it as an opportunity to network with like-minded partners, learn about educational research and just “expand their reach.” Reflecting on the experience, Tom said:

“As far as how research was conducted, I really didn’t have a clue. It was interesting to understand how research groups frame and set things up, breaking it down to the essential questions. I’ll be more inclined to do so on my own now that it has been modeled for me, so I can mimic that in the future.”

The NIC experience brings with it new terminology, like “driver diagrams,” “root causes”and “fish bones.” David appreciated that “information wasn’t immediately dumped on us, it was actually a slow start, evenly paced. We were able to digest at monthly webinars first and get some practice doing some of these things.” Apparently, it worked. When asked to explain a fishbone diagram, David answered quickly:

“A fishbone diagram organizes all the little problems that you see in an area, and helps you group them together and see what the main [root] causes are that are contributing to the problem. Being able to see all the problems out there, and putting them together, helps see how they all contribute to the big issue.”

Describing a driver diagram, David said:

“It starts with a goal, something that you want to do, and then you know that there’s really big things that need to change, but you can’t tackle those big things, so it helps you break down those big things into smaller pieces and smaller pieces, which then helps you start making change by addressing the things that you actually can do something about.”

At the end of those first two webinars, with purposeful guidance from 100Kin10 facilitators and lots of practice on their individual NIC diagrams, each partner organization chooses the issue that they most wanted to tackle. As science educators working in media, David and Tom were both interested in exploring the role of media in teaching engineering.

The big kick-off came in April 2016, the day before 100Kin10’s Fifth Annual Partner Summit in Houston, Texas, when approximately 25 NIC participants gathered to work face-to-face for the first time. Partners were now asked to coordinate their work with another team, finding common ground and working for the greater good. They were put into smaller groups called Research Action Clusters (RACs), based upon their shared interest. “It was like matchmaking,” said David.

summit

Tom and David’s collaboration was a natural fit. The Teaching Channel makes videos about teachers teaching, while WNET mostly makes videos for PBS LearningMedia that teachers can watch with their students. Together, they decided to focus on “Real World Engineering Media” in the classroom. Tom remembered: “A lot of the other clusters had six to eight people, and then off in the corner was kind of David and I…. We just kind of bounced ideas back and forth, and then if it resonated we just went with it. If we were at an impasse and there’s something that we just kind of had different thoughts, then we’d bring over somebody from 100Kin10 like Matty Lau, or one of the consultants from American Institutes of Research [AIR].”

As they worked together on a driver diagram, their focus narrowed: they wanted to find a way to evaluate the impact of engineering video programs on teacher professional development and classroom instruction. As far as their diagram, Matty said, “Initially, they struggled to clarify what they wanted to do, but more important than a perfect driver diagram, was the fact that they were dedicated to working together to find that clarity. Actually, for a first time doing a NIC, their work was remarkable.” She explained:

“We just needed to help partners be okay with starting something and seeing where it goes.”

After the Summit, Tom and David continued working on their project virtually, with Tom often calling from the road. Occasionally, they checked in with 100Kin10 and AIR consultants for coaching support and to discuss the data they collected. Their first step was to construct and send a Google survey form to a group of science teachers, asking them what they found most difficult about integrating engineering into science classes. The survey also included links to several WNET engineering-in-action videos and asked teachers to name their favorites. David found that teachers have a clear idea of what they want in videos about engineering:

“Teachers want to be shown parts of the engineering design process, not just the end product. Videos need to talk about constraints, brainstorming, and prototypes.”

Their next step was designed to find out what kind of videos actually “increase K-12 educators understanding of engineering content.” To accomplish this, David and Tom used a new group of teachers, dividing them into two groups. Both groups watched a favorite WNET engineering-in-action video like: “Sitara the Swan,” which showed a doctor repairing a goose’s broken beak. One group also watched a Teaching Channel video which showed a science teacher (Tom) explaining the engineer design process to his own students. The group that watched both videos had higher engineering comprehension scores, suggesting that teachers benefit from watching engineering being taught in the classroom.

Video

Even more important than these early results themselves, was the fact that, in a relatively short period of time, David and Tom created a simple tool to help their media organizations measure the educational impact and effectiveness of videos relating to the engineering design process. As Tom put it:

“Just being part of the network improvement community was a tremendous experience. You’re tasked with this common problem that seems Herculean. Then together you can try to devise these solutions to where you can kind of chip away at different aspects of the problem and before long, therein lies the solution and we can make progress.”

Both partners agree that the NIC approach has changed the way they approach problems. Tom says that the Teaching Channel can now “integrate this tool into its framework, asking its customers to rate videos based upon more quantifiable learning criteria. This will help us design future content and determine which videos to highlight for our viewers. And we can share these measures with potential investors to help demonstrate value.”

Moving forward, The Teaching Channel and WNET plan to continue working together, building upon their discoveries. For example, they want to “break the study into different grade bands to give us more insight into how teachers at different levels approach teaching engineering.” According to David Kanter of 100Kin10, “that kind of ‘what works for whom’ is precisely the analysis that the NIC is designed to support.”

David and Tom may not be full-time researchers, but their brief, focused collaboration is already moving the needle to help teachers teach engineering with more confidence and efficiency. Together WNET and the Teaching supported one another’s efforts, while chipping away at a deeply embedded, potentially overwhelming problem: the lack of engineering teaching in schools. “It’s quite empowering,” says Tom, “and on a personal level it validates what your goals are, and it gives you hope that there are other like-minded folks out there working to accomplish the same goal.”