Massachusetts Pipeline Fund @Scale Initiative Report: Business Brief Presented to the Massachusetts Department of Higher July 2014
The genesis of the Massachusetts @Scale Initiative lies in the state s original 2010 Statewide Plan, A Foundation for the Future: Massachusetts Plan for Excellence in. A central theme of this document was the question of how to invest strategically and with an expectation of predictable and measureable outcomes through the plethora of competing education projects. 1 This was a concern raised by parties from multiple sectors (business, K-12 education, higher education, etc.). The result was an initiative that sought to provide guidance: 1) to educators with regard to identifying effective projects that are aligned with state frameworks and standards, and 2) to potential funders of such projects to ensure investments are directed to projects that meet a threshold for evidence of effectiveness and are strategically aligned with statewide goals. Keeping in mind that there are many high-quality projects across the Commonwealth that are unknown to the Governor s Advisory Council, it sought to develop a list of projects that both served the needs of educators and funders listed above, as well as met criteria for replication and/or scalability. The @Scale Initiative is a portfolio system approach intended to deliver a suite complementary projects as a group for the improvement of interest, ability and goals. This suite possesses both an investment process (which incentivizes donors whether they are businesses or foundations to support the group of projects) as well as a recruitment process (of the projects themselves to be awarded @Scale status and become participants in the initiative). The suite of projects aims to cover a wide spectrum of life stages (early education through workforce), developmental areas, types of learning, and subject areas, among other things, so that interest, ability, and goals can be improved throughout a system rather than through pockets. The @Scale Awards were made in four phases (although Phases III and IV were issued at the same time) over a period of approximately 20 months (January 2012, August 2012, and September 2013). As a result, as of the timing of this report, @Scale activities for Phase III and IV awardees may not have been completed. All of the selected projects committed to meeting a funding match ratio of $3 private for every $1 public they received as part of their award. Seven projects were funded through Phase I of the initiative, five were funded through Phase II, and five were funded through the joint Phase III and IV round for a total of 17 projects in the current portfolio (Figure 1). The @Scale Initiative was embedded in a theoretical framework centered around a pipeline system that conceptualizes an individual s development from early education into the workforce and across individuals interest, ability and goals (Figure 2). Effective practices for affecting interest, ability, and goals are then implemented though different types of programming models (Figure 3). There is a solid theoretical base underlying the Pipeline Theoretical Framework, the Portfolio Approach, as well as the funded projects. At this point the @Scale Initiative has been successful across several aspects of its pilot implementation, including: 1) Coverage of all three dimensions of : Personal interest, ability and goals. 2) Coverage of all four parts of : Science, Technology, Engineering and Math. 3) Coverage of both in-school and out-of-school time. 1 Cedrone, David. (September 2012). Massachusetts Plan @Scale Briefing Document. 1
4) Coverage of all geographic regions of the state. 5) Coverage of a variety of student learning and teacher professional development effective practices. 6) Coverage through a variety of programming models. 7) Coverage of all five of the Statewide Goals. The pilot implementation of the initiative yielded a number of lessons learned as well as recommendations for future improvement. These included: Lessons Learned: 1) Although the Request for Proposal process generated substantial interest and a diverse range of proposals, the Governor s Advisory Council (GSAC) was limited to working with those projects that applied for the grant award. As a result, some known and potentially effective projects did not apply and, as a result, were never considered for funding. It is unclear why these projects did not pursue an @Scale grant, but it may be theorized that: a. the financial carrot was not big enough to entice large projects, b. the application process was too involved/complicated for small projects, c. outreach about the initiative was not extensive enough, d. the initiative had a large match requirement which was outside the operational scope of many organizations, and/or e. larger groups may have considered themselves to already be @Scale and not appropriate candidates. 2) The spread of projects over so many life stages (early education, elementary education, middle school, etc.), coupled with their geographic diversity, resulted in too much dilution of the projects effects for results to be seen through standardized outcomes data (such as standardized test scores or SAT interest data). In essence, construction of the pipeline may require a degree of focus and continuity, such that it can function as a complete system. 3) There was very little communication among the projects selected to be part of the portfolio. As a result, they missed opportunities to (a) partner with each other for implementation in specific locations, or (b) further complement each other in terms of program development at the system level. 4) In retrospect, it appears that many project proposers did not understand the full meaning/implication of scaling up and simply equated it to expanding numerical reach as opposed to meaning an expansion or packaging of infrastructure such that the capacity of the entire program could be expanded or replicated. This resulted in distinct limitations on scaling potential for most projects. 5) Leaving project evaluation at the local/project level was ineffective, resulting in little to no useful data from some projects and disparate results from others. At a minimum, more rigorous standards should be set for the local evaluations in terms of what they cover and expected quality. Even better would be to implement this kind of standard and layer a standardized statewide evaluation over it, thus insuring that at least some common information is collected across all of the projects. 2
Considerations and Recommendations: 1) If resources are limited, concentrate efforts on limited geographies and/or life stages. This will decrease the likelihood of program effects being overly diffuse and increase the ability of projects to work with each other. 2) If consideration/recommendation #1 is adopted, focus on early education / early elementary level programming. This holds promise to counteract many of the stereotypes and issues that projects try to address later in the pipeline. 3) Create structures that promote inter-project communication and cooperation so that project implementers can recognize opportunities to complement and support each other at the system, organization, and implementation levels. 4) Cast a wider net for bringing projects into the system: do not simply wait and see who applies. Perhaps move to a nomination system rather than a self-application system. Another option would be to radically simplify the application process such that much more of the investigative work about the quality of the project lies with @Scale staff rather than on the project to present in an application. Perhaps have a simple application to be followed by a site visit with an evaluation rubric. 5) Improve the evaluation of grant applications to include criteria for how a project contributes to the whole portfolio, not just whether they are a quality project on their own. There are many projects that could potentially be included that are very strong on their own, but would only be duplicative of projects already within the portfolio (e.g., cover the same effective practices during the same life stage and through a similar format). Similarly, there are many projects that may appear to not be as strong at the individual level, but which could cover missing pieces within the portfolio and, as such, make the system stronger. 6) Strengthen the rubric for determining who becomes part of the portfolio, especially segments related to research basis, past evaluation results (and the standards of evidence for them), future evaluation plans, and scalability. In addition, confirm that funding will truly be used for scaling: that the project will be retaining past activities/sites and not just replacing them. 7) Consider funding/evaluating portfolio proposals rather than managing a portfolio of individual projects. That is, implement the scaling up premise at the initiative level: the goal of the @Scale Initiative is to promote portfolio implementation (that is, portfolios of projects proposed by groups of partner institutions) rather than project implementation. Portfolio proposals could be reviewed against a standard for variety of programming, coverage of effective practices, estimated participant total dosage, and other elements central to the theoretical basis of the portfolio and pipeline systems. The initiative would then not be so much about crediting individual projects as scalable but about promoting/scaling the portfolio approach generally. 8) Modify the match investment to be with the portfolio as a whole rather than with individual projects. This could alleviate at least one of the factors that potentially limited applications to the @Scale Initiative as well as increase the reach of donor funding (instead of affecting participants in one project they are affecting participants in the portfolio of projects). The @Scale portfolio approach has a strong research basis and a great deal of potential to affect the improvement of interest, ability and goals. Enhancement of this pilot program, taking into account the lessons learned and program considerations/recommendations would be of benefit to the Commonwealth and serve as a contributor to the improvement of our students and workforce. 3
Figure 1: @Scale Initiative Portfolio Coverage Map Phase I @Scale Awards Phase III @Scale Awards Phase II @Scale Awards Phase IV @Scale Awards @Scale Phase I Awards: @Scale Phase II Awards: @Scale Phase III Awards: @Scale Phase IV Awards: Focus on high school interest, achievement, and/or educator effectiveness (Statewide Goals 1, 2 and 3). Awards issued January 2012. Focus on college degree completion and/or workforce development (Statewide Goals 4 and 5). Awards issued August 2012. Focus on middle school interest, achievement, and/or educator effectiveness (Statewide Goals 1, 2 and 3). Awards issued September 2013. Focus on interest, achievement, and/or educator effectiveness in early education and/or out-of-school time programming for middle grades (Statewide Goals 1, 2 and 3). Awards issued September 2013. 4
Figure 2: Pipeline System Theoretical Framework The following model depicts a individual s (student s) progression through the Pipeline System and how their personal interests, abilities, and goals interrelate both within each life stage as well as over time. Interest Situational or Intrinsic Subject Orientation Engagement Perceived or Actual Knowledge Skills Goals Career or Personal Motivation Identity Early Workforce Elementary Middle Effect on Next Generation Pipeline High College 5
Figure 3: The @Scale Logic Model Implementation Short-Term Intermediate Long-Term Need Input Method Outcomes Outcomes Outcomes Not enough people are interested in, and prepared for, careers @Scale Initiative Student and/or teacher programming to increase student interest, ability and goals Increase number of students participating in projects that increase interest, ability, and goals Increase student interest, ability, and goals Increase number of people who are interested in, and prepared for, careers Early Workforce The above Logic Model depicts the shortterm, intermediate and long-term outcomes that are anticipated by implementing the @Scale Initiative. The model to the right provides an example of how an array of s might be distributed across the Pipeline, thus influencing an individual s (student s) personal interest, ability and goals over time. Elementary Middle Effect on Next Generation Pipeline High College 6