Office of Superintendent of Public Instruction 21 st Century Community Learning (CCLC) Program Evaluation and Data Collection Update for Spring 2018 April 20, 2018 Introduction: We are in a busy time of year for statewide data collection and evaluation activities, for the federal APR21 data entry, as well as state and local evaluation efforts. To keep us on track and on time, I am providing an update to the evaluation and data collection guidance. First, I want to acknowledge that we have a complex system of data collection, and evaluation, and thank you for your ongoing efforts to help Washington State demonstrate positive outcomes for our 21 st CCLC Programs!! Important Changes in 21 st CCLC Data Collection Protocols and Timelines to Note in this Update Include: (See details below) The Federal 21APR data collection system launches April 23 rd for the 2017-18 data collection. New dates are provided for data collection windows OSPI is collecting PQA Satisfaction Surveys for Program Directors, Site Coordinators, Staff and Evaluators, April 24-May 22 for 2017-18. The timeline coincides with the timeline for Summer APR Data entry. Overview of Statewide Evaluation and Data Collection Activities and Timelines: February 15, 2018 December 15, 2018 Data Collection Activities and Timelines: I. AIR Leading Indicator Staff and Site Coordinator Surveys: February 15 March 31 II. Federal 21APR Data Reporting April 24 November 16, 2018 by Summer, Fall and Spring Term: New dates OSPI updates user accounts and permissions, grantees, allocations, and sets state specific outcomes, and timelines for grantee reporting April 15 23 Cohort 14 New Grantees Enter Profile, and Center Information April 23 May 23 Cohorts 10-13 Report APR Summer 2017 data April 23 May 11 Cohorts 10-14 Report APR Fall 2017 data May 23 July 6 Cohorts 10-14 Report APR Spring 2018 data August 1 November 16 APR State Data Certification November 17 December 15 III. Weikart Center PQA Satisfaction Survey for Managers and Staff April 24 May 23 New data collection IV. AIR 21 st CCLC Youth Motivation, Engagement, and Beliefs Surveys: April 2 June 7
V. AIR 21 st CCLC Program Attendee Module: June 5 July 31 VI. Local Evaluation Reports due to grantees and OSPI November 1 Statewide Evaluation Activities, Purpose, Additional Information, and Resources I. Leading Indicator Staff and Site Coordinator Surveys: February 15 March 31 URL: Leading Indicator Survey Project Director Login Help Desk Contact: wa21stcclc@air.org Used to populate the Program/Site Leading Indicator Reports: Used for Advanced Planning with Data and Development of Program Improvement Goals, reported in Scores Reporter. Informs Quality Improvement Efforts Collect information from site coordinators and staff who work directly with students in programming Process: 1. Program management and staff view recorded webinar and power point presentation posted at and attached PowerPoint slides to prepare for survey administration process from February 11, 2017 Emails from AIR were sent to project directors containing a link to the survey management system, along with a username and password for the director. These emails were sent from wa21cclc@air.org during the week of February 15 th. Complete surveys from site coordinators and afterschool program staff are due March 31 st 2. Once logged into the Project Director Survey Management page: Director must ensure the centers listed for a given grant are correct and send any corrections needed to WA21stcclc@air.org and cc Kelsey.Winters@k12.wa.us, and ssniegowski@air.org Under the name of each center, a survey Web link unique to each center is provided This link needs to be forwarded to the site coordinator for that center When the link is clicked on, the respondent will be asked to create a survey account and then be prompted to take the survey in question
Site coordinators and staff will utilize the same link to create a survey account and take the survey for their role in the program You can return to the Project Director Survey Management page to view the completion status of surveys and send email reminders 3. In addition to the site coordinator, ideally, a minimum of 8 staff per site should be identified per center 4. Staff selected to take the survey: Need to spend the majority of their time working directly with youth by leading activities Can be paid or a volunteer Can work for the grantee directly or for a partner organization, or on a contract Can be working in the program or recently ended their participation in the program (e.g., led activities in a previous session that was active at some point in program year) Should work (or did work) in the program at least once a week If there are less than 8 staff at a given center that meet these criteria, then all staff at the center meeting these criteria should be asked to take the survey 5. Directors Must monitor Survey Completion for Site Coordinators and Staff Project directors can see both: Who has completed a survey account Whether or not they have completed their survey from the Project Director Survey Management page Automated reminders can be sent directly from the Project Director Survey Management Page Project directors are required to monitor the survey process frequently to ensure submission by March 31. II. Federal APR Data Reporting for 2017-18 Data by Term Summer 2017, Fall 2017 and Spring 2018 April 24 November 16 URL: Federal 21 APR Data Reporting Login Help Desk: Federal 21 APR Data Reporting Login Help Desk You can also email them at 21apr@thetactilegroup.com or call 1-888-282-4589 Required Federal Data Collection for the Department of Education Used to report to Congress on the GRPA Measures for the 21stCCLC Program
Process: 1. The SEA (State Coordinator) assigns user accounts by program for grantee users including Grant Directors, and Evaluators 2. Grantees and Evaluators review Training and Technical Assistance Documents at Federal 21 APR Data Reporting Support 3. Grantees Login at Federal 21 APR Data Reporting Support using the Username (e-mail) and password you submitted. If you cannot remember your password, click change password and answer the challenge questions you set up in the system. If you cannot remember the answers to your challenge questions contact OSPI to request a new account login. Kelseywinters@k12.wa.us 4. Complete or update Center and Profile Data for 2017-18, and Activities, Staffing and Participation by Term for 2017-18. 5. Grantees Complete Data Reporting by Terms Summer 2017, Fall 2017, and Spring 2018 with each reporting period aggregating the previous period. Spring will include all data from Summer 2017 through Spring 2018. 6. Do not enter any outcome measures for Summer, and Fall. WA only reports outcomes for State Assessment Data and only in Spring term 7. Participation and Outcome data will be provided to you to enter for Spring by AIR, and derived from the data you submit in the Regular Attendee Module 8. Timelines for Reporting: New OSPI updates user accounts and permissions, grantees, allocations, and sets state specific outcomes, and timelines for grantee reporting April 15 23 Cohort 14 New Grantees Enter Profile, and Center Information April 23 30 Cohorts 10-13 Report APR Summer 2017 data April 23 May 11 Cohorts 10-14 Report APR Fall 2017 data May 23 July 6 Report APR Spring 2018 data August 1 November 16 APR State Data Certification November 17 December 15 9. Notes: There will be no windows of time for corrections. The windows are much longer and should provide time for corrections. There are new features added to the System that improve user friendliness and transitions between reporting sections. Each reporting segment closes at 11:59:59 PM Hawaii time. 10. Reporting timelines for terms will be as follows: Summer July 1-August 31, 2015 (or the dates of the summer program) Fall September 1- January 31, 2016 Spring February 1-June 30, 2016 11. All grantees profiles need to be updated annually in the system 12. There will be no export or reporting feature for grantees. 13. There will be no export/import feature for the states, so we will not be able to access the data after it is submitted, nor import our state assessment data. 14. AIR will send you a report by site to enter spring outcome and participation data. 15. We are still exploring a statewide system for collecting 21stCCLC data in 2018-19.
III. Weikart Center PQA Satisfaction Survey for Directors, Site Coordinators, Evaluators and Staff April 24 May 23 New Data Collection URL: will be sent April 23 from OSPI Help Desk contact: TBD The survey of staff, managers, evaluators, and site coordinators in our 21 st CCLC sites is a critical part of the continuous improvement cycle for program supports The survey provides an assessment of the past year s efforts and decisionmaking for next year regarding PQA and Professional Development supports and training offered by OSPI to 21stCCLC Grantees. Process: 1. Survey findings will be reported in aggregate and no identifying information will be included in the final report 2. As part of your Youth Program Quality Intervention, you will be asked to complete a 15-minute survey on your participation and experience from this past year 3. This survey will need to be completed by you (program directors), as well as evaluators, site coordinators and staff at your programs. The survey will be distributed on April 24 via a link to an online platform 4. Directors and Evaluators will receive the link to the online survey from OSPI 5. Directors will need to email the link out to your site coordinators and have site coordinators email it to staff members. 6. All participants must complete this survey no later than May 23. 7. Participants submit only one survey if they serve in multiple roles or support multiple programs or sites. 8. The questions included in this survey ask you about your participation in the following activities: Youth Program Quality Intervention (YPQI) Kickoff Conducting a Program Self-Assessment Using the Program Quality Assessment (PQA) Youth Work Methods Training and other Professional Development workshops Developing a Program Improvement Plan using PQA data Sustainability of the Quality Improvement System
IV. Youth Motivation, Engagement and Beliefs Survey April 2 June 7 URL: Youth Motivation, Engagement and Beliefs Survey Login Help Desk contact wa21stcclc@air.org To establish a new measure of young people s skills and beliefs, as a potential outcome measure that contributes to academic and behavioral outcomes. Measuring things like academic mindset, persistence, school belonging, personal identity, and social and civic values. Spring 2016 onward Implementing in all WA 21 st CCLC 4-12 grade sites, but with emphasis on surveying students who took the survey the previous year! Process 1. Survey data will be collected in a way that will allow OSPI to connect youth survey responses with data about that youth contained in the state data warehouses maintained by OSPI. 2. Each project director will receive an e-mail from the wa21stcclc@air.org in late March containing: A set of instructions for logins per site, Criteria for selecting young people to survey and Parent/guardian passive consent forms in English and Spanish. A webinar recording from 2016 is available for training and technical assistance on the OSPI 21stCCLC website 21st CCLC Evaluation and Accountability. Listen to the Student Motivation, Engagement, and Beliefs Survey webinar April 11, 2016, and see the instructions and permission forms available. 3. Directors should review the instructions and webinar with staff that will be administering the surveys, make lists of eligible students, and make preparations for computer labs, or alternate online access and a schedule for surveying students. 4. Each Director will receive an additional e-email with a username and password to access the online system for submitting student surveys. Paper surveys will not be accepted. This email will also contain survey links for each site. Each link should be passed along to the appropriate staff member administering the survey. 5. When staff click on this survey link, you will see a place where you will be asked
to identify the young person s name and SSID (10 digit state student ID number). Each youth will also be assigned a unique survey ID. 6. Enter the online System and follow the instructions. 7. Begin Surveying Students April 3, and finish by May 31. Try to avoid SBAC testing windows to access computer labs in schools. V. AIR 21stCCLC Attendee Module: June 5 July 31 URL: AIR 21st CCLC Attendee Module Login Help Desk contact wa21stcclc@air.org Collect participation data on all students who attended program during the program year. Collect Individually Identifiable Student Information such as SSID numbers, names and codes for schools of enrollment, and names, etc. Process 1. Timelines for reporting: June 5 July 31 Data is reported in three terms o Summer: July 1-August 31 (or when summer officially starts and ends in the district) o Fall: September 1-January 31 o Spring: February1-June 30 (or when summer officially starts and ends in the district) 2. Types of student data you will need to report per student in the Attendee Module: School District Identifier (District ID#) School District Name (Name) School of Attendance Identifier (School ID#) School of Attendance Name (Name) Unique 10-digit Student State Identifier (SSID Number) Students Unique District Identifier (Number) (Optional) Student Last Name Student Middle Name or Initial (Optional) Student First Name Student Date of Birth (MM/DD/YYYY format) Student Grade Level (preset drop-down options) Student Attendance Summer 2017 (integer value)
Student Attendance Fall 2017 (integer value) Student Attendance Spring 2018 (integer value) 3. Tasks to complete on a regular basis Continually update list of program enrollees Keep program attendance records up to date Obtain 10-digit Student State Identifier (SSID) for each student upon enrollment Obtain SSID numbers from School District monthly Obtain Grade Levels and DOB from the School District monthly Identify District and School Building names, and building and district codes where each student attends school during the regular school day, from WA State Title I school building list posted on 21stCCLC website OSPI 21st CCLC. 4. Review the Attendee Module PowerPoint Presentation and Module Demo Webinar, and Excel Template from June 2016 posted on the OSPI website under training at 21st CCLC Evaluation and Accountability. 5. Initial login process for the Attendee Module online server Invitation email sent from wa21cclc@air.org the first week in June to all directors and evaluators, with link to join the site, that contains your username Enter username and email address on the site and submit Receive second email with temporary password Login using username and temporary password You will be prompted to create a new password We will not have access to this password (use forgot password function if misplaced) Once new password is created, login in with username and new password 6. Follow the Instructions for entering and uploading information contained in the webinar and PowerPoint on the OSPI website for 2016. 7. Validate all data prior to submission District ID# Must be 5 digits School ID# - Must be 4 digits SSID# - Must be 10 digits, Duplicate or incomplete SSIDs entries not allowed Days Attended Must be an integer, even if it is 0. Must match when programming is offered (summer and school year) All fields required except for: Student s Unique District ID# Student Middle Name/Initial
VI. Local Independent Evaluation Mail and email a copy of the Report to OSPI and Program Directors by November 1 All 21st CCLC programs are required to conduct an annual local independent evaluation of the program s effectiveness and submit an annual evaluation report to the Office of Superintendent of Public Instruction (OSPI). The purpose of the evaluation is to support continuous program improvement at the local level. It should measure progress in meeting the program goals and objectives, and progress in meeting the state and federal performance indicators, including progress in increasing student achievement and behavior that supports student success for at risk students. The local evaluation report guidelines are designed to provide a uniform framework for reporting the annual findings of the 21st CCLC projects. Flexibility is purposefully built into the guidelines so that projects can maintain responsive local evaluations that serve the needs of local projects. Process 1. Evaluators should meet with project directors prior to the program year to review the projects goals and objectives, review the logic model or theory of change (or develop one), review any drafts of last year s evaluation report, and to set up the evaluation design for the upcoming program year. This may include a discussion of any issues outside of the stated goals and objectives that the project director or evaluator thinks is important to examine. Evaluators and project directors should visit the statewide leading indicators report to determine if there is any area of interest that would require further investigation during the upcoming program year. Newly Revised 2. Evaluators should check in with project director at least twice a semester to discuss any issues or concerns, or anything new and interesting that may affect the evaluation. Regular check-in calls could also serve as a tool to flag any important adjustments that need to be made midway through the evaluation process. 3. Evaluators should provide a draft of the local evaluation report to the project directors for review, allowing a chance to discuss any questions or concerns, as well as talk about recommendations for the upcoming year. Once the final report is submitted, evaluators should schedule a follow-up conversation to confirm all data collection plans and goals for the next program year. Evaluators might also schedule an on-site meeting with the project director and site staff to discuss the findings from the report and create an action/improvement plan for the next program year. 4. Refer to the WA 21stCCLC Evaluation Guidelines on the OPSI 21stCCLC website at WA 21st CCLC Local Evaluation Guidelines
for reporting requirements and format recommendations. Additional resources on selecting an evaluator or designing a 21stCCLC evaluation are also located at the WA 21 st CCLC Evaluation and Accountability website 21st CCLC Evaluation and Accountability. 5. Final Evaluation Reports are due to Program Directors and OSPI no later than November 1 of each year. Mail a hardcopy to: Attention: 21stCCLC Program Manager, Learning and Teaching Support, PO Box 47200, Olympia WA, 98504-7200, and email a copy to the OSPI, 21stCCLC Program Supervisor. Note: Questions regarding 21 st CCLC Data Collection and Evaluation should be directed to Debra.appleton@k12.wa.us. Please contact me if you have any concerns or questions Best, Debra Williams- Appleton Debra.appleton@k12.wa.us 360.725.6049