What happens after you evaluate and purchase a new edtech program? This article deep-dives the most important things to put in place in order to make sure the new program is effective.
Research studies always come with a friendly little reminder: You will only get the promised results if you implement the program with fidelity. But what, exactly, is fidelity? Is it as simple as using the program for the recommended dosage? Or does it require a coordinated and sustained effort by you, your teachers, and your edtech provider, all working together?
Let’s look at the four kinds of critical components:
- Program design
- Assumptions about teacher knowledge
- Expectations for how teachers interact with students
- Student engagement
Program Design
Program design is how an edtech provider intends for the program to be used. There are four areas of program design that have a bearing on fidelity:
- Dosage
- Sequencing of the learning
- Program content
- Best practices in design such as usability and accessibility
These design elements are the primary responsibility of your edtech provider. Let me share a few examples of program design at Speak Agent:
- Speak Agent is designed to have a flexible dosage, targeting 4 hours per month. Originally the dosage was 1 hour per week, but we found that providing teachers with more flexibility was beneficial. Independent research suggests that it’s okay to vary week by week, as long as it averages out to 4 hours/month over the marking period.
- In Speak Agent, the sequence of lessons and academic concepts matters. It mirrors the core curriculum of the school district in order to provide learners with timely context. This mirroring also reduces teacher workload. Within each lesson, we do have a recommended order of learning tasks that build upon one another in a learning sequence, but we purposefully allow students to circumvent that sequence to give them agency to find their own optimal path.
- In terms of content, one design goal we have is to use questions or prompts that challenge the students we serve. We know we’ve implemented this design principle because, on average, about one-third of student answers are incorrect on their first try. We adhere to this design principle because learning sciences research suggests that students need to be challenged with material that pushes them past the edge of their prior knowledge. Another part of our content design is to use learning scaffolds such as hints, visual aids, and other learning supports that help students acquire that new knowledge.
- Best practices can be more subjective, but there is research behind them. One best practice in Speak Agent’s design is low-stakes scoring: Points are ultimately awarded for participation and perseverance, since most activities require a 100% score to pass. There are no quizzes, but rather knowledge checks that students and teachers alike can use to gauge progress, areas where students are struggling, and common misconceptions.
- Other best practices we employ are universal design principles for usability and accessibility and giving teachers and students agency with the learning experience, rather than trying to be overbearing with the technology solution. These practices are expressed, for example, in our AA level accessibility compliance and our AI Design Principles. Other edtech providers have chosen a different path with their own practices, so this is an area you should examine closely when choosing a provider.
Teacher Knowledge
The second critical component is about the assumptions that we have for what teachers should know. What is their level of expertise in the curriculum content? How were they trained in pedagogy?
For example, with Speak Agent Math + Language, we do start with the assumption that math teachers understand that language is required to support the eight Standards for Mathematical Practice (SMPs)—at least after they’ve participated in our PD sessions. We also expect they understand basic, widely accepted teaching strategies such as the evidence that formative assessments support learning by providing feedback to both teacher and students.
We also expect they know that students achieve a deeper level of understanding when instruction is focused on concepts rather than memorization. And they should already know that a powerful element of learning occurs when students give verbal or written explanations of their thought processes and reasoning.
These are understandings a teacher should have when starting use of our program. It’s good for the school district and the edtech provider to be on the same page about these kinds of understandings before starting implementation. If you have opposing assumptions about what teachers should know, it’s going to be a real challenge to implement with fidelity.
So consider this: What are the most critical understandings for your curriculum?
Teacher-Student Interaction
The third critical component in fidelity of implementation is teacher behavior. This is about how we expect teachers to interact with their students when using a program.
For example, with Speak Agent, we expect teachers to model the program in whole class instruction before setting students loose with independent practice. We also expect them to vary the mode of instruction to include teacher-led, groups or pairs, and individual work with the program. This is important in our program, because language is a collaborative activity that requires interacting with one or more peers in order to make learning progress.
There are other more basic expectations, too, about how teachers teach. In math and science, we expect teachers will ask their students to explain their answers. This is implemented in the program itself, but if teachers do not set these expectations for their students outside the program as well, it’s going to be extra challenging for students if this is their first encounter with this way of learning.
We also expect that teachers will address common misconceptions. We provide tools such as the Level Up formative assessment report, which identifies common misconceptions, but it’s up to the teacher to use the data to close the loop and explain the most-common incorrect answers.
You can see that some of these expectations are ones that the edtech provider has and some may be expectations that the school district has. Again, these two sets of expectations need to align if you want to have a high-fidelity implementation!
Student Engagement
The last of the four critical components is student engagement. But it is most certainly not the last in importance!
Without students orienting their attention, keeping up their motivation and confidence, asking questions, discussing ideas, and developing problem-solving strategies, all the most wonderful programs and curricula in the world will not make an impact.
A well-designed program can draw student interest, pique curiosity, inspire ideas, and then get out of the way, removing barriers to usability and learning. But it cannot solve for a class full of students who are not in a learning mindset or who feel stuck without the intervention of the classroom teacher. So engagement really is an all-hands-on-deck effort.
Fortunately, Speak Agent has a great track record of partnering with districts to enhance student engagement.
Measures of Fidelity
So thinking about these four kinds of components—program design, teacher knowledge, how teachers interact with students, and student engagement—how do we know if we’re on the right track?
Honestly, this is the hardest part. There is an entire field of study on how to measure fidelity of implementation, but it can be so specific to the content area, program, or curriculum that it’s challenging to devise universal measures. But there are certainly some things we can measure.
Measuring Program Design
- Dosage is one of those things we can measure exactly. For Speak Agent, we can break down usage data by content area, by school, by class, by grade level, and by student. We can observe data trends over a time series as well.
- Learning progress with the content can similarly be measured. In Speak Agent we can measure acquisition of the specific academic concepts that students need to master for each unit of study in three discrete stages: vocabulary knowledge, comprehension, and ability to communicate reasoning. Learn more in a deep-dive of our Content + Language Framework.
- Best practices like usability can be assessed with data collection from experts or from users. Speak Agent collects input from students on usability, content, and accessibility for every activity and responds rapidly to improve based on student voices. We collect more than 10,000 student responses every month! (Fortunately, 97% report no issues.)
Measuring Teacher Knowledge and Behavior
We make sure to identify and fill any gaps during Speak Agent PD sessions, as well as self-paced professional learning modules and live coaching sessions that teachers have access to afterward. We also partner with school districts on large-scale implementations to integrate with their instructional guidance, unit maps, pacing guides, etc.
- We can measure attendance at PD sessions and teacher feelings as to their readiness.
- We measure usage of our self-paced modules, aka Speak Agent Academy.
- We track live e-coaching sessions as well.
We can also measure whether teachers use the program as expected when interacting with students. For example, we expect teachers to use our Preview feature to get familiar with the digital lessons during prep time or to model whole-class activities. We can measure use of this feature. We can also measure the frequency of assignments, use of due dates and bonus activities, and whether students were able to overcome misconceptions identified on their formative assessments.
Measuring Student Engagement
Speak Agent measures assignment completion, time on task, student ratings of activities, quality of open-ended answers, progress toward concept mastery, student ratings of peer work, and more. In addition, we often partner with districts to survey teachers about their perceptions and observations of student engagement.
Conclusion
At Speak Agent, we are always looking for ways to make implementation more seamless and more successful because, as a teacher-owned and -operated company, we care deeply about student outcomes.
You may also be interested in our white paper, The Big Shift in How to Evaluate EdTech Product Effectiveness, which gives you tools for discerning real research from, let’s call it “promotional research,” and explores why fidelity of implementation can be hard to achieve at scale.
