EVALUATING STEM SCOUTS: THE DESIGN OF A COMPREHENSIVE EVALUATION PLAN AND FEASIBILITY STUDY

The Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. An immediate challenge for informal STEM education is the lack of high quality program evaluation. What does high-quality evaluation look like and how can it be accomplished? High-quality evaluation necessarily begins with good evaluation planning. In addition, although several large-scale youth-serving organizations embrace STEM, there is limited research on the interaction between STEM outcomes and social-emotional learning (SEL). This pilot and feasibility project involves a collaboration between researchers with expertise in STEM education, out-of-school time (OST) programs, social-emotional learning, and evaluation, as well as practitioners and program developers. STEM Scouts participants are grouped into 'labs' of approximately 10-20 boys and girls with 3 or more leaders per group and often collaborate with local schools. The mission of STEM Scouts is to help young people grow in character and STEM skills as they explore their curiosity using experiential activities and interaction with STEM professionals. Building on prior NSF-funded work, this project expands the implementation of the Systems Evaluation Protocol (SEP), a three-stage evidence-based approach to evaluation planning involving (a) Preparation, (b) Model Development, and (c) Evaluation Plan Development. The proposed project has implications for other youth-serving organizations that also embrace a dual mission of supporting STEM and the social-emotional development of youth.The goals of this project are to: (1) develop a detailed theory of change for STEM Scouts that highlights the hypothesized interactions between STEM outcomes and Positive Youth Development/SEL outcomes; (2) pilot three new enhancements to the SEP (System Mapping, Ecosystem Modeling, and Model Validation); and (3) determine the feasibility of conducting a national scale STEM Scouts study. The project will utilize a research-practice partnership Evolutionary Evaluation approach and the enhanced SEP. A working group of STEM Scouts leadership and project researchers will work through initial stages of the SEP to generate a stakeholder map, logic model and pathway model (a causal depiction of how the program works). Ten STEM Scout Labs purposively selected from across the country with the goal of diverse representation will participate in focus groups. Using logic and pathway models generated by the working group, each Lab will complete a System Mapping activity to identify the larger system in which the Lab exists (e.g., stakeholders and decision-makers). Focus group participants will review and discuss their maps. Through Ecosystem Modeling they will identify the most important outcomes, key connections, and revisions they might make to their model, and discuss how the model reflects their STEM Scouts experiences. A coding dictionary will be developed to analyze focus group transcripts; findings will inform Model Verification and will be used to assess whether changes need to be made to the enhanced SEP. It is hypothesized the enhanced SEP will enable the working group to better understand factors hindering or enabling program and evaluation feasibility and success. Project findings will be disseminated to the evaluator and research community, OST program providers/developers and the public.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.>