Accelerate Learning by 3X or More in Six Hours
In addition to the secondary evidence presented in our Learning Sciences page, there is mounting empirical evidence that shows Speak Agent's strong evidence of promise.
Impact Study Findings
A quasi-experimental design (QED), two-phase impact study (Li, 2018), sponsored by the National Science Foundation (award #1632488), found significant gains in elementary STEM concepts and vocabulary using Speak Agent. Five second-grade classes totaling 74 students, 59 (80%) of whom were ELLs, at two schools at a central Texas public school district were compared. The district administered randomized versions of pre/post-tests designed by independent evaluator Rockman et al in September and December 2017. A second set of pre/post-tests were administered in January and April 2018. Each test consisted of 16 to 20 varied question types covering 40+ academic concepts from second-grade STEM units such as matter, energy, force and motion, the natural world, the water cycle, weather, and living organisms.
Second-grade students in the Fall 2017 experimental group used Speak Agent for an average of less than six hours over a 12-week period (< 30 minutes per week). The control group received regular STEM instruction over the same 12 weeks. The experimental group acquired 3.1 times more STEM concepts than the control group did. In Spring 2018, both the experimental and control groups were allowed to use Speak Agent. On average, these students acquired 4.6 times more STEM concepts over 12 weeks in Spring 2018, as compared to the Fall 2017 control group. Again, students had access to Speak Agent for an average of less than 30 minutes of usage per week over a 12-week period. Both effects had significance at p < .01. The graph shows the average gains in STEM concept acquisition for each second-grade group. Speak Agent content addressed 102 second-grade concepts in seven units of study.
Testing with third graders had similar findings but were not statistically significant, likely due to constraints with population size, fidelity, and dosage. Read the full study report by Rockman et al >
Independent evaluator Rockman et al conducted eight teacher interviews as part of the NSF study. Teachers reported that Speak Agent helped students learn complex and challenging STEM concepts. Engagement with Speak Agent increased connections to the content in significant and impactful ways, as Li (ibid.) reports: "[Teachers] observed students engaging in more STEM content, asking questions about challenging topics, and wanting to look up additional information on their own time. Students would create their own experiments (e.g., model volcanos) based on content introduced in Speak Agent, or they would go to the library to find additional information about a concept discussed through the activities. Students also seemed to remember information from Speak Agent and apply it to class lessons and discussions." Teachers "liked that the Speak Agent tool focused on individual students’ needs and helped teachers assess where students struggled and thrived." They lacked time, however, for STEM teaching and also lacked equipment. The classrooms had student-to-device ratios of 1:2 to 1:5. Some rooms lacked working projectors. Students had access to Speak Agent for less than 30 minutes per week and teachers were allowed only one hour of training and professional development. Significant results were achieved in spite of these barriers.
The above teacher comments are verbatim, as recorded by interviews in the NSF study.
Results from Montgomery County, Maryland with a different second-grade STEM curriculum showed similar findings to the Spring 2018 NSF study (15% change score) with the same student population, dosage, and 12-week timeframe. Concurrently, the Maryland afterschool program (Excel Beyond the Bell) found a highly promising 31% growth in reading vocabulary over the same period.
This unexpected finding led us to pursue a small-scale study in middle-grade reading intervention in partnership with KIPP Kansas City and LEANLAB Education. The KIPP study found a gain of 25% in reading assessment scores among 7th graders over just a nine-week period. Find a study summary here. A larger-scale follow-on study is planned for SY20-21.