NASA Education Pogam 2007 Assessment Pogam Code 10002310 Pogam Title Pogam Type(s) Pogam Notes Assessment Yea 2007 Assessment Status Assessment Notes Assessment Rating NASA Education Pogam Competitive Gant Pogam Final Ready to publish/published. No changes will be made fom this point on. Results Not Demonstated Assessment Section Scoes Section Scoe Pogam Pupose & Design 100% Stategic Planning 88% Pogam Management 60% Pogam Results/Accountability 33% Pogam funding Level (in millions) Pio Yea $162 Cuent Yea $140 Budget Yea $154 Explanation of Composition of Funding Funding Teasuy Account Budget Resouces (millions) Cuent Yea Pio Yea Budget Yea 800114 $162 $140 $154 s/s Obligations (millions) Pio Yea Cuent Yea Not Not povided. povided. Budget Yea Not povided. Explanation Scoe Section 1
1.1 Is the pogam pupose clea? Scoe YES 20% Explanation: NASA's Office of Education, Mission Diectoates, and centes engaged in a compehensive stategic planning pocess duing FY 06. The planning pocess was diected and implemented by the NASA Education Coodinating Committee (ECC), which is composed of epesentatives fom all NASA oganizations involved in the implementation of education pojects. The ECC's stategic planning wok culminated in publication of the NASA Stategic Coodination Famewok: A Potfolio Appoach which clealy outlines the pupose of NASA's education pogam. The NASA Education Famewok is designed to suppot the NASA Stategic Plan (NASA Policy Diective [NPD] 1001) and the Vision fo Space Exploation. To achieve the Vision and on-going science and aeonautics activities, the Agency equies a highly skilled and divese wokfoce. NASA education investments ae an impotant component to ensuing the availability of that wokfoce. The NASA Stategic Coodination Famewok aticulates thee majo education goals, as follows: 1. Stengthen NASA and the Nation's futue wokfoce; 2. Attact and etain students in Science, Technology, Engineeing, and Mathematics (STEM) disciplines; and 3. Engage Ameicans in NASA's mission. Ou investments ae oganized to facilitate a student's pogess though a pipeline of oppotunities pogessing fom inteest to academic pepaation in STEM subjects, leading eventually to employment in the aeospace wokfoce at NASA, with academia, o in the pivate secto. Evidence: The following two documents aticulate the puposes of NASA's education pogam. Thee puposes ae defined in the NASA stategy while thee outcomes ae defined in the education famewok. NASA Stategic Plan, NPD 1001; NASA Stategic Education Famewok, NP-2007-01-456-HQ, http://education1.nasa.gov/about/stategy/
1.2 Does the pogam addess a specific and existing poblem, inteest, o need? Scoe YES 20% Explanation: The NASA Education Pogam addesses both intenal NASA needs and the extenal needs of the education community. NASA's needs ae elated to wokfoce development while the nation's needs in STEM education have been identified in numeous authoitative epots. Both the NASA Stategic Plan, 2006, and the NASA Stategic Coodination Famewok clealy aticulate the needs to be addessed by the education pogam. These needs ae the development of the highly educated and well-pepaed wokfoce that is citical to the success of the Agency's mission. The NASA education stategy is ultimately designed to identify and develop the citical skills and capabilities needed to achieve the Vision fo Space Exploation. As stated in the NASA Stategic Plan, 2006, the Agency's human capital management challenge is geate than eve. NASA must develop and implement Agency-wide human capital management initiatives to ensue that the wokfoce contains the ight skill mix and an appopiate balance between civil sevice, contacto, and othe components to achieve the Agency's Vision. Thus, the pogam tagets development of individuals who become pepaed fo employment in disciplines needed to achieve NASA's mission and stategic goals. Beginning in the ealiest gades, though high school, and then though intenships, fellowships, and othe pofessional taining, NASA offes individuals a pipeline of oppotunities to become paticipants in the Vision fo Space Exploation. NASA's education pojects contibute to paticipating students' pepaation fo employment with NASA, industy, o academia. Moe geneally, since the inception of the education pogam in the ealy 1960s it has contibuted to the "expansion of human knowledge of Eath and space phenomena" as povided fo in the Space Act of 1958. Evidence of the needs of the extenal education community egading STEM education is included in the NSF publication, Science and Engineeing Indicatos, 2006. Additionally, in a 2005 epot by the National Academies, Rising Above the Gatheing Stom: Enegizing and Employing Ameica fo a Bighte Economic Futue, two ecommendations wee made that elate diectly to the value fo the nation of NASA's education pogam: (a) incease Ameica's talent pool by vastly impoving K-12 science and mathematics education; and (b) sustain and stengthen the Nation's taditional commitment to long-tem basic eseach. NASA's Stategic Education Famewok and ou pojects ae designed to addess and implement these ecommendations. Evidence: The following two documents aticulate the puposes of NASA's education pogam: NASA Stategic Plan, NPD 1001; NASA Stategic Education Famewok, NP-2007-01-456-HQ, http:// education1.nasa.gov/about/stategy/. The epot, Rising Above the Gatheing Stom, is available fom the National Academies, http:// www7.nationalacademies.og. Science and Engineeing Indicatos is available fom the NSF, http://www.nsf.gov/statistics/seind06/. The Academic Competitiveness Council (ACC) epot povides additional evidence.
1.3 Is the pogam designed so that it is not edundant o duplicative of any othe Fedeal, state, local o pivate effot? Explanation: On the whole, NASA's Education Pogam fills a unique niche by poviding educatos and students the oppotunity to gain diect expeiences using ou state-of-the-at equipment and facilities, the agency's unique mission of scientific discovey and exploation, and ou highly skilled aeospace wokfoce. These attibutes of NASA enable the implementation of education pojects that cannot be diectly duplicated by othe U.S. oganizations. One of the cental goals of the pogam is to stengthen NASA's futue wokfoce; to this end, the pogam funds space-specific intenships, eseach gants, and othe education pogams. To minimize edundancies among the multiple NASA oganizations involved in education, the pogam has implemented a pocess, fomalized though a stategic famewok, to coodinate intenally with ou mission diectoates and centes and eliminate unwaanted duplication of effot via an Education Coodinating Committee (ECC) with boad stakeholde paticipation. An impotant accomplishment in FY 06 was the competitive selection of specific centes to lead designated pojects, thus eliminating poject duplications and management stuctues. Each investment in the education potfolio, except Space Gant and EPSCoR has now been tansfeed to a lead cente. Within the fedeal govenment, we coodinate though the Academic Competitiveness Council (ACC). While the ACC has thus fa only conducted an inventoy of govenment-wide pogams, we expect to use the foum to collaboate with othe agencies and seek ways to wok togethe to minimize edundancies and eplicate effective pactices. NASA also elies upon patneships as a way to minimize edundancies with othe effots. To fomalize NASA's ageements with othe agencies, we have established Memoanda of Ageement with the National Science Foundation (NSF), the National Pak Sevice (NPS), and the Fedeal Aviation Administation (FAA). NASA has established a patneship foum to ceate synegy with a significant numbe of education and industy oganizations. By building stategic patneships and linkages between STEM fomal and infomal education povides, the pogam expects to ceate a mechanism to coodinate pogams and shae esouces. Even with many coodinating pocesses in place, many of the gants pogams conducted by the pogam ae simila to the type of eseach suppot that othe fedeal agencies (e.g., Dept. of Education, Dept. of Enegy, National Science Foundation) awad; theefoe, it is not entiely clea that all NASA education pogams can only be conducted by NASA. In addition, some pogams, such as the Faculty Awads fo Reseach, duplicate the eseach gants aleady offeed by othe NASA pogam offices. Evidence: MOUs between NASA and the NSF, the NPS, and the FAA, Detail on the NASA-NSF MOU is posted at http:// www.education.nasa.gov/divisions/highe/oveview/ F_One_Giant_Step_STEM_Education.html Scoe YES 20%
1.4 Is the pogam design fee of majo flaws that would limit the pogam's effectiveness o efficiency? Scoe YES 20% Explanation: The pogam is designed to be fee of majo flaws. NASA's education outcomes wee developed with boad stakeholde input, including input fom OMB. Ou audiences ae clealy defined and they ae seved though an effective mix of gants, eseach oppotunities, fellowships and scholaships, and pofessional development activities deliveed diectly to schools. Additionally, the eoganization of the NASA Office of Education and the development of the new Education Famewok was specifically intended to addess inadequacies identified by OMB in NASA's FY 04 PART esults. Since the FY 04 PART, we have teminated seveal pojects to educe the total numbe of pojects and concentate esouces moe efficiently. Temination decisions wee the esult of an annual eview of the education potfolio and wee based on seveal consideations including, (a) insufficient evidence of effectiveness, (b) edundant investments, o (b) non-competed investments. Appoximately 50% of the pojects in the education potfolio wee teminated, phased out, o estuctued. In FY 04, six pojects wee teminated; in FY 05 thee wee teminated and thee wee phased out; and in FY 06-07, seven wee phased out o teminated, and eight wee estuctued. To futhe educe the numbe of investments, we have conducted an inventoy and established atings by pee eviewes in pepaation fo decision-making. Additional aeas fo impovement wee identified by a eview team composed of intenal and extenal stakeholdes led by NASA's Office of Pogam Analysis and Evaluation (PA&E) in FY 05. The PA&E team was chateed by the NASA Administato to: 1. develop an implementation plan to suppot the Vision fo Space Exploation; 2. define a coodinated potfolio; 3. descibe oganizational elationships and esponsibilities; and 4. define an evaluation pocess. Among the impovements ecommended by the team wee to evise NASA's education goals to impove alignment with the NASA Stategic Plan. The team also poposed a new govenance model to impove communication and coodination as well as to claify oles and esponsibilities. Actions taken since then espond to the eview team's ecommendations. The stategic famewok, evised management appoach, and aticulation of outcomes, objectives, and measues epesent citical steps taken to addess flaws identified in NASA's FY 04 PART esults. Notably, NASA conducted an extensive seach leading to the appointment of a new Assistant Administato fo education with authoity fo all aspects of the total NASA education pogam. Evidence: NASA Stategic Education Famewok, NP-2007-01-456-HQ, http://education1.nasa.gov/about/stategy/. Repot fom the PA&E eview team and NASA Administato's chate to the team. Summay epot of poject teminations.
1.5 Is the pogam design effectively tageted so that esouces will addess the pogam's pupose diectly and will each intended beneficiaies? Scoe YES 20% Explanation: Foemost, the pogam tagets NASA's human capital needs. NASA's fist pioity outcome is to develop a wokfoce to suppot the NASA mission and ou budget allocations eflect this pioity. To ensue coodination with human esouces, a epesentative fom the Office of Human Capital Management paticipates in the ECC. NASA's education famewok identifies a pogession of oppotunities to inspie, engage, educate, and employ students. To accomplish this goal, the pojects taget students at all levels of the education systems, teaches, faculty, and institutions. Thus, we have defined a specific population of beneficiaies. Institutions, including schools, univesities, and infomal education oganizations, eceive the majoity of NASA's education financial esouces which ae invested in the fom of competitive gants awaded to education institutions. The Space Gant, EPSCoR, and Minoity Univesity pojects make all awads to the states and institutions designated in legislation though competitive pocesses. Additionally, the highe education pogam ecently conducted a pincipal investigato confeence, in collaboation with the National Science Foundation, to infom new faculty how to benefit fom and paticipate in NASA's education pogam. Evidence: Minoity Univesity Reseach and Education Pogam povides annual epots documenting that HBCUs, Tibal College, and HSIs ae seved as intended. (Repots posted at http://mued.nasaps.com/ epot/ ) The Space Gant and EPSCoR pojects epot annually to document that ecipients ae eached in states as mandated in legislation. The evaluation pocess aticulated in NASA's education famewok is designed to validate that ou pojects ae eaching ou intenal NASA and extenal audiences. The potfolio assessment pocess will enable an analysis of ou pogam to not only ensue we ae eaching designated segments of the education community but also to ensue that thee ae no gaps in the potfolio to pevent othe potential beneficiaies in the education community fom eceiving sevices. 100% Section 2 Scoe
2.1 Does the pogam have a limited numbe of specific long-tem pefomance measues that focus on outcomes and meaningfully eflect the pupose of the pogam? Scoe YES 12% Explanation: Duing the 2006 fiscal yea, NASA established two woking goups to develop measues and analyze the existing potfolio. The Potfolio Woking Goup, conducted a poject inventoy and chaacteized it with espect to alignment with the outcomes and objectives aticulated in the stategic famewok. The Evaluation Woking Goup, woking closely with the potfolio goup, developed an aticulated set of measues linked to each objective to enable pefomance measuement. The 15 measues that wee developed and adopted by the ECC wee subsequently appoved by OMB. Included ae long-tem outcome measues designed to detemine the degee to which NASA-suppoted students gain employment in the aeospace wokfoce in NASA, academia, o industy. Othe long-tem outcome measues document inteest gains among K-12 student paticipants; anothe long-tem measue tacks the degee to which K-12 teaches use the knowledge and mateials acquied fom paticipation in NASA education pojects. Evidence: NASA's thee pioitized outcomes; 16 objectives, each linked to an outcome; and 15 measues, each linked to an outcome and objective, ae documented in the measues section of the PART and in NASA's annual pefomance plan. The 15 PART measues wee appoved by OMB (see the PARTweb measues section). 2.2 Does the pogam have ambitious tagets and timefames fo its long-tem measues? YES 12% Explanation: NASA has documented ambitious tagets fo its PART measues as documented in the measues section of PARTweb. Fo new measues, data will be collected and baselines will be established in FY 07. NASA's objective is to continue seving the same numbe of paticipants, despite any funding eductions in the agency's budget fo education. Evidence: PARTweb measues section shows the pogam's measues, ambitious tagets, and timefames though 2011. The tagets show that the pogam is measuing long-tem success by the degee to which pogam paticipants ente NASA and STEM-elated caees.
2.3 Does the pogam have a limited numbe of specific annual pefomance measues that can demonstate pogess towad achieving the pogam's long-tem goals? Scoe YES 12% Explanation: Yes, the pogam has a limited numbe of specific annual pefomance measues that can demonstate pogess towad achieving the pogam's long-tem goals. Annual measues tack, fo example, annual numbes of students involved in NASA education pogams as well efficiencies gained in pogam execution each yea, such as cost to NASA pe paticipant. Evidence: PARTweb measues section. The annual measues fo the pogam will measue the degee to which the pogam eaches intended paticipants and the extent to which it does so in an efficient and costeffective manne. In geneal, data ae collected and will be epoted by calenda yea. 2.4 Does the pogam have baselines and ambitious tagets fo its annual measues? YES 12% Explanation: The pogam has established baselines and ambitious tagets fo the majoity of its annual measues. Whee baselines have not yet been established, they will be detemined by the end of FY07. Evidence: See PARTweb measues section. Baselines and ambitious tagets have been established fo the pogam; the tagets ae intended to impove on and at least not fall below histoical pefomance of the pogam. Many of the tagets intend to show incemental impovement in pefomance o efficiency in eaching the pogam's intended paticipants and beneficiaies.
Scoe 2.5 Do all patnes (including gantees, sub-gantees, contactos, YES 12% cost-shaing patnes, and othe govenment patnes) commit to and wok towad the annual and/o long-tem goals of the pogam? Explanation: The NASA gants handbook specifies that gantees ae accountable fo the activities specified in thei poposal. The poposal epesents the commitment of the institution to the goals aticulated in the poposal. Since poject manages ae accountable fo ensuing the poposal tagets the pogams goals, it epesents the gantee commitment. Annually, poject manages povide guidance to gantees egading changes to NASA pefomance goals fo education. Futhemoe, the Office of Education's stategy is to manage Congessionally diected pojects as integal components of its education potfolio. Thus, each poject must contibute to achievement of one of thee defined education outcomes and must be implemented accoding to the objective(s) of the elevant categoy. Pojects ae accountable fo the same pefomance equiements, including pefomance measuement epoting, as ae all othe pojects in the education potfolio. Guiding pinciples include: 1) Implement a poject that fulfills the intent of Congess; 2) Implement a poject that contibutes to the achievement of the outcomes aticulated in NASA's Stategic Education Famewok; and 3) Ensue taxpaye funds ae spent esponsibly, efficiently, and in accodance with applicable legal, egulatoy, and policy guidance, including the NASA FY 2006 Eamak Guidance. A NASA Poject Office fom the Office of Education o a cente education office is designated fo each eamaked poject. The Poject Office is the pimay point of contact at NASA fo the ecipient oganization; the office eviews the poposal, pepaes the technical justification and the gant package, and monitos the gant, ensuing that the poject is implemented as poposed and that equied pefomance measuement data ae submitted. Fomal mechanisms, such as Space Act Ageements (SAA) and Memoanda of Undestanding (MOU), ae used to establish stategic patneships with specific long-tem goals of the pogam in mind. Tems of these ageements include defining oles and esponsibilities of paties as tageted towads specific pupose that establishes commitment to pogam goal(s) and epoting on measues and/o esults. Futhemoe, foums and summits, have been effectively convened to communicate pogam diection and goals with inteested oganizations, that have included govenment, academia, and industy. Evidence: The gants awad document epesents institutional commitment to woking towad Agency education goals. Additionally, guidance was tansmitted to each manage esponsible fo education gants that diects them to ensue poposals ae eviewed and monitoed fo pefomance. All Eamak ecipients must submit a poposal with the Office of Education. Poposals must include the following elements: (a) a detailed desciption of the wok to be pefomed, including a clea listing and schedule of the specific activities suppoted by the NASA gant; (b) a desciption of the esouces equied, clealy listing expenditues to be funded by NASA; (c) an adequate justification of the poposed budget, including a bief naative desciption of majo budget items; and (d) a detailed desciption of the elevance of the poject to the NASA education pogam. Space Act Ageements and Memoanda of Undestanding Highlights http://www.education.nasa.gov/divisions/highe/oveview/ F_One_Giant_Step_STEM_Education.html http://www.nasa.gov/home/ hqnews/2004/ma/hq_04083_honeywell_launch.html http://
Scoe
2.6 Ae independent evaluations of sufficient scope and quality conducted on a egula basis o as needed to suppot pogam impovements and evaluate effectiveness and elevance to the poblem, inteest, o need? Scoe YES 12% Explanation: NASA has developed and submitted to OMB a planning schedule to accomplish evaluations of all investments in the Office of Education budget. Each investment will be evaluated at least once duing a five-yea peiod using the most independent, igoous, and eliable methods possible. Pojects will be selected on the basis of multiple citeia including the size of the investment, the elative impotance of the investment within the education potfolio, and the length of time since the last evaluation was conducted. Seveal poject evaluations that ae independent and of sufficient scope to guide impovements and evaluation effectiveness have been conducted within the past five yeas o ae now neaing completion. An outcome evaluation of the Exploe Schools poject will be completed in FY 07. This evaluation is extemely impotant in that it tests the feasibility of using Randomized Contol Tials (RCTs). To date, five pogess epots have been submitted by the independent evaluato who woks closely with poject management staff to ensue findings ae implemented. As a esult of this evaluation, seveal impovements to the poject have been made. These impovements include management consolidation at a NASA cente fo geate efficiency, implementation of specific instuction modules to suppot the school cuiculum, and a geate focus on pofessional development. An evaluation of the Gaduate Student Reseaches Pogam (2000 and 2006) documented its effectiveness and led to an impovement to stipend levels based on ecommendations. Othe evaluations that evaluate effectiveness and suppot impovements have been conducted by univesity-based evaluatos using pofessionally-accepted methods on the Aeospace Education Sevices Pogam (2004), the NOVA Pesevice Education Pogam (2003), EathKam (2006), and the Faculty Fellowship Pogam (2006). The Evaluation Wok Goup (EWG), chateed by the NASA Education Coodinating Committee to define an evaluation stategy, designed a thee-component pefomance measuement pocess. The second component, defined in this context as the eview pocess, establishes a pocess fo conducting outcome evaluations of effectiveness. To the geatest extent possible, we will implement evaluations though cedible, objective evaluatos using techniques based on RCT methodologies, as ecommended by the Academic Competitiveness Council (ACC). In situations in which an RCT-based method is not appopiate, methodologies will be adopted that ae igoous, objective and in confomance with ecognized pofessional standads. Evidence: Evaluation epots of each of the pojects cited above ae available. The NASA education famewok documents ou appoach to pefomance measuement and evaluation. An evaluation of the NASA Exploe Schools poject is being conducted by an extenal, univesitybased evaluation team. Peliminay esults have been obtained though peiodic pogess epots and these epots have been shaed with OMB. Beginning in FY 08 an additional evaluation is planned fo the Science, Engineeing, and Mathematics Aeospace Academy (SEMAA) poject. An evaluation of the NASA elementay and seconday education potfolio is being conducted by an expet panel convened by the National Reseach Council (NRC). Two meetings of the NRC panel have been conducted with two moe to be held in CY 2007; the panel's pe-publication epot is
Scoe 2.7 Ae Budget equests explicitly tied to accomplishment of the annual and long-tem pefomance goals, and ae the esouce needs pesented in a complete and tanspaent manne in the pogam's budget? YES 12% Explanation: The potfolio eview pocess established by the stategic famewok is designed to ensue pefomance is factoed as a key element in budget decisions. Additionally, the thee education outcomes have been pioitized and this pioitization scheme is used as a significant facto in making decisions about esouce allocations. NASA's annual Integated Budget and Pefomance Document links budget equests, made in full cost, to specified pefomance standad. Evidence: NASA's budget documents ae located at http:// www.nasa.gov/about/budget/ Pogam plans have been submitted by each NASA cente that is esponsible fo implementation of one o moe pojects. These plans clealy descibe and budget fo all necessay pogam costs, including pesonnel and financial equiements, and manages ae held accountable fo confomance to the plan. Similaly, gant pojects must submit a budget establishing majo cost categoies. This budget is eviewed, appoved, and monitoed fo gantee compliance. The NASA Office of Education fully complies with all equiements of the NASA poject management pocess aticulated in NASA Policy Diective, NPD 7120.
2.8 Has the pogam taken meaningful steps to coect its stategic planning deficiencies? Scoe NO 0% Explanation: In the pevious PART eview, suppoted by intenal eviews, seveal stategic deficiencies wee outlined fo this pogam including an unde developed pefomance management system, unclea alignment to NASA's wokfoce needs and an unde developed stategic investment famewok that aligns the pojects and tasks to the pogam goals. Albeit the pogam has made significant pogess on addessing these deficiencies it has not completed the wok. The pogam has adopted new long-tem and annual pefomance measues that eflect its pupose and content. Duing FY 2006, a igoous pocess was initiated and completed by the NASA Education Coodinating Committee (ECC). The pincipal pupose of the ECC's wok was to addess deficiencies that had been identified in the 2006 PART and that had been identified though an intenal eview. With espect to stategic planning, the ECC engaged in a thoough eview of the existing stategy and substantially evised it. The evision aticulated thee specific outcomes with five to six objectives to addess each outcome. Additionally, a management evaluation pocess was defined. In the Potfolio eview pocess we will be using a goup decision suppot softwae application called Expet Choice. Expet Choice was selected to povide a systematic decision-making tool to achieve bette, faste, and moe justifiable decisions while also ensuing oganizational and stategic alignment of all pojects. Expet Choice was used by the ECC in the second quate of FY 2007, to pioitize outcomes and objectives to guide futue decisions, though a mapping of the Education pojects to these pioities. This impotant wok has been stated but is not yet complete. Decsions have not been made fom this analysis yet but will be. Planned completion is late this calenda yea. Evidence: The NASA Education Famewok: A Potfolio Appoach is the pincipal document that defines the stategic plan. Measues coesponding to each objective and outcome has also been defined. The potfolio and famewok have been communicated to all manages. Additionally, the National Reseach Council evaluation of ou cuent elementay and seconday pojects is expected to esult in objective conclusions and ecommendations that we will facto into planning eviews. 88% Section 3 Scoe
Scoe 3.1 Does the agency egulaly collect timely and cedible pefomanceno infomation, including infomation fom key pogam patnes, and use it to manage the pogam and impove pefomance? 0% Explanation: The Office of Education does collect timely and cedible pefomance infomation as shown below. The office fully complies with the pocess and pinciples of stategic management outlined in the NASA Stategic Management and Govenance Handbook (NASA Policy Diective 1000 and with the NASA Pogam and Poject Management Pocesses and Requiements (NASA Pocedual Requiement 7120) equiing the development of Pogam and Poject Plans. Pogam and poject plans have been established fo Highe Education, Elementay & Seconday Education, Infomal Education, and e-education. These plans equie pojects to conduct annual pefomance planning and epoting; submit monthly epots and conduct quately eviews; conduct annual eviews with esults of the eview used to develop an impovement plan to be integated within its next annual pefomance plan; and submit pefomance data to the education data system. NASA has established a data collection system, compised of thee linked databases, to collect data fom each poject egulaly with the goal that poject manages will use the data to make poject impovements as waanted. Inconsistencies exist, howeve, in that the pojects do not all epot the same data elements. The pogam also does not collect consistently data on pogam beneficiaies and paticipants o tack fo all of its pojects beneficiaies and paticipants' schooling o caee paths afte thei involvement in the pogam; the pogam theefoe does not have solid infomation on how well it pefoms in tems of eaching and having lasting impacts on its intended beneficiaies. In addition, the pogam has not yet established baselines fo all of its pefomance measues o begun to collect data fo most of the measues. Evidence: The NASA gants web site ( http://nodis3.gsfc.nasa.gov/ ) povides clea guidance stipulating data epoting equiements, esponsibilities, and pocedues is disseminated annually. Responsibility fo compliance with data epoting equiements is clealy delineated in the Education Famewok. NASA poject manages ae equied though thei individual pefomance plans to use these and othe data epoted to assess pogess towad goals and make adjustments via the annual pefomance planning and epoting, and pefomance goal development peiods. Poject patnes collect and epot pefomance data. Examples of data collected which suppot pefomance measues include: National Space Gant College and Fellowship Poject - fo 2005 data (most ecently available), longitudinal fellowship/scholaship esults - 31% employed by NASA, industy, academia, o othe STEM fields, 49% moved on to an advanced degee pogam, and 18% wee seeking STEM employment. Jenkins Pe-doctoal Fellowship Pogam - cumulative esults total 26 Ph.D. and 33 M.S. degee ecipients in STEM with 8% of gaduates enteing the NASA wokfoce. Undegaduate Student Reseach Poject - fo 2001-2003 data (most ecently available), longitudinal esults show 26% of paticipants in aeospace-elated employment (vs. 21% of applicants who wee not selected); Univesity Reseach Centes - fo 2005 data (most ecently available), longitudinal awadee esults - 8% of undegaduate paticipants moved on to advanced degees, 7% of undegaduate paticipants wee employed in STEM; 11% of maste's degee paticipants moved on to doctoal pogams, 16% of maste's degee paticipants wee employed in STEM.
Scoe 3.2 Ae Fedeal manages and pogam patnes (including gantees, sub-gantees, contactos, cost-shaing patnes, and othe govenment patnes) held accountable fo cost, schedule and pefomance esults? YES 10% Explanation: Since NASA's FY 04 PART was conducted, actions taken have ensued that manages and gantees ae held accountable. In accodance with the NASA Employee Pefomance Communication System (EPCS), all poject manages' pesonal pefomance plans document thei esponsibilities fo poject o pogam pefomance and they ae accountable fo effective pefomance. To maximize accountability, the Assistant Administato will povide input to Cente Diectos egading the pefomance of all Cente Education Diectos. Gantees who fail to fulfill thei equiements isk having thei pojects discontinued. Fo example, five Space Gant state consotia wee equied to estuctue thei consotium and document esults as a condition fo futhe suppot. The NASA Education Stategic Coodinating Famewok aligns the poject management accountability fo cost, schedule and pefomance esults to the expected outcomes of each poject. Each level of poject management that includes the Fedeal pogam manage, poject manages, gantee/contactos, and othe patnes, fully complies with NASA Pocedual Requiement (NPR) 7120.7 Institutional/Infastuctue Pogam/Poject Management Pocesses and Requiements, which is detailed in each Education Poject Plan. Poject manages have submitted annual pefomance plans, detailing accountability measues. The ECC ovesees the potfolio though annual eviews, using a fomal decision-making softwae tool, Expet Choice, to addess accountability and to specify coective actions when necessay. Evidence: Poject Manages ae accountable fo cost, schedule and pefomance esults though the implementation of compliance appoved poject plans maintained by the Office of Education. In geneal, action is taken egading gant pojects that fail to meet schedule and pefomance objectives because gants epesent at least two-thids of the education budget. Specific management changes and coective actions have been taken. Fo example, in FY 06 five pojects wee teminated o phased out due to inadequate pefomance and a lack of stategic alignment with education objectives and an additional thee pojects wee estuctued. These plans specify the citical elements of esponsibility fo pefomance at all poject management levels. Pojects plans meet the compliance equiements of NPR 7120.5C, NASA Pogam and Poject Management Pocesses and Requiements (http:// nodis3.gsfc.nasa.gov/). Futhe, poject management compliance is a key facto in decisions used to detemine gantee awad enewals and poject continuations. Manages ae accountable fo poject pefomance though thei individual pefomance plans. These plans specify the citical elements of esponsibility and, in the case of education poject manages, pefomance elements ae specified fo each poject fo which he o she is accountable.
3.3 Ae funds (Fedeal and patnes') obligated in a timely manne, spent fo the intended pupose and accuately epoted? Scoe YES 10% Explanation: Funds ae lagely obligated in a timely manne and spent fo the intended puposes. Fo all pojects, manages eview poposals, maintain egula communications, and evaluate pogess epots to ensue funds ae spent accoding to plan and fo intended puposes. The exception is that funds ae not obligated in a timely manne fo Congessionally-diected pojects because poposals ae often not eceived fom the ecipient institutions in a timely manne. All Congessionally-diected pojects ae eviewed against Agency citeia (meit, stategic alignment, & cost) as well as education citeia in accodance with guidance tansmitted to manages. Howeve, diectly appopiated pojects must be awaded egadless of meit eview, ensuing the poposal is at least minimally acceptable. New financial contols have put in place to allow tacking of pogam expenditues at centes. Evidence: NASA's budget system is based on full-cost pinciples and includes wok beakdown stuctues. The system povides full visibility into obligation status. Contactos ae equied to submit monthly o quately costing epots that documents the funds ae being used accoding to equiement. Additionally, the Office of Education fully complies with the equiements of NPD 7120, which establishes guideline fo the appoval and obligations of funds.
3.4 Does the pogam have pocedues (e.g. competitive soucing/ cost compaisons, IT impovements, appopiate incentives) to measue and achieve efficiencies and cost effectiveness in pogam execution? Scoe YES 10% Explanation: The pogam's has efficiency measues with baselines o plans to establish baselines, in addition to output and outcome measues. These measues will, fo example, demonstate the pogam's ability to educe the cost pe paticipant of its pogams and show how well the pogam leveages funds contibuted by pogam patnes. The e- Education pogam is specifically designed wok closely acoss NASA education initiatives within the Office of Education and acoss the Mission Diectoates to facilitate the incopoation of technology and distance education delivey methods as well as existing libaies of pogamming fo pofessional development and intenet-based poduct dissemination with the goal of eaching a boad audience of paticipants at a elatively low cost. Additionally, ou patneship initiatives ae intended to achieve synegy among paticipating oganizations and to leveage the esouces and extenal expetise of these oganizations in ode to achieve ou planned outcomes with a geate level of efficiency. Fo example, FMA Live engages middle-school students in math and science though a live stage show that demonstates science in kids' eveyday lives. The entie pogam is funded by Honeywell. To impoved efficiencies we ae also consolidating pojects. Fo example, seveal elements of the Minoity Univesity Reseach and Education Pogam (MUREP) have been meged (e.g. we have consolidated the Patneship Awads fo Integation in Reseach (PAIR) and the Cuiculum Impovement Patneship Awads (CIPA) pojects). Evidence: NASA developed efficiency measues with input fom and the appoval of OMB. Additionally, NASA is making extensive use of the libay of distance education pojects to delive infomation, pofessional development taining and to conduct vitual symposiums and kick off events fo membes of the fomal and infomal education community. Distance education allows NASA to seve a much highe numbe of paticipants pe unit of investment. Vitually all of ou education investments ae competitively selected to facilitate selection of the most efficient and effective pojects.
3.5 Does the pogam collaboate and coodinate effectively with elated pogams? Scoe YES 10% Explanation: NASA employs thee pincipal methods to collaboate and coodinate with elated pogams. Intenally, the Education Coodinating Committee (ECC) is the mechanism to ensue coodination among the NASA offices involved in education. These offices include the Office of Education, the Mission Diectoates, and the NASA Field Centes. The NASA Education Stategic Coodination Famewok (http:// education.nasa.gov/about/stategy/index.html) documents each oganization's equiements fo collaboation and coodination. Second, fedeal Govenment collaboations include paticipation on the govenment-wide Academic Competitiveness Council (ACC), NASA has developed ageements with STEM-elated agencies and oganizations acoss the govenment and with the pivate secto. We have designated staff to epesent NASA on each of the ACC woking goups and this pocess facilitates coodination at the woking level. And thid, NASA convened two confeences to bing togethe ou extenal collaboatos fom fedeal agencies, industy, foundations and the extenal education community. As a esult, "Futues Panels" at each NASA Cente have been fomed to analyze the status of STEM education, leading to a epot expected to be completed in Decembe, 2007. The epot will identify additional methods fo collaboation. We collaboate with seveal extenal fedeal agencies as evidenced by MOUs and joint activities. Fo example, a model patneship with ExxonMobil, leveaged funds to suppot the Benad Hais Summe Science Camp (BHSSC). NASA povided stat-up funding fo the BHSSC and now contibutes content and expetise with majo funding now assumed by ExxonMobil. Anothe example of a specific collaboation is with the National Science Foundation with which we implemented the Joint NASA-NSF Reseach and Education Oppotunities Confeence fo Pincipal Investigatos, Faculty, and Patnes on Febuay 22-24, 2007. NASA also collaboates with pofessional STEM education oganizations, including (a) the National Science Teaches Association, a key patne in the Exploe Schools poject; (b) the National Alliance of State Science and Mathematics Consotium, with which NASA conducts state STEM education impovement initiatives, and (c) the Intenational Technology Education Association, with which NASA assisted in the development of new technology standads, published in 2004. NASA woks with the Intenational Space Univesity (ISU) by coodinating student intenship placements at NASA centes. Evidence: Extenal Patnes: The extenal collaboations wee extensive though the two meetings of the emeging NASA patneships. Confeences wee held & documented at (http://education.nasa.gov/ about/team/patneship.html). Patneship Foum on Septembe 12, 2006 & Patneship Summit on Januay 17, 2007. Gil Scouts U.S.A. & NASA collaboated to aise compehension & inteest of gils, leades, & leade-taines in science & science-elated topics, & to encouage gils & women to pusue STEM caees. The MOU esulted in a STEM confeence held at the US Depatment of Education that was sponsoed by NASA, Gil Scouts, & the US Depatment of Education. MOUs suppot Educato Astonaut, Babaa Mogan, on STS-118: Ameica OnLine. Kids Sevice & Mad Science have collaboated to develop the Pennant Design Challenge ; Obital Technologies Copoation & Planet LLC fo Educational Gowth Chambes have developed an engineeing design challenge; Imaginay Lines (www.sallyridescience.com) fo thee Educato Institute; & Pak Seed Company fo "SEEDS in Space" (Space Exposed Expeiment
Scoe 3.6 Does the pogam use stong financial management pactices? NO 0% Explanation: The most ecent Independent Audito epot fo NASA identified two (2) mateial weaknesses, all of which ae epeats, as well as noncompliance with the Fedeal Financial Management Impovement Act. Evidence: NASA's FY 2006 Pefomance and Accountability Repot (www.nasa.gov/about/budget/index.html) includes the communication fom the NASA Inspecto Geneal and the epot of the Independent Audito. In addition, the GAO has published numeous epots identifying shotcoming in NASA's new financial management system as well as its financial management pocesses (example is GAO-04-754T eleased on May 19, 2004). 3.7 Has the pogam taken meaningful steps to addess its management deficiencies? YES 10% Explanation: Duing FY 2006, a igoous pocess was initiated and completed by the NASA Education Coodinating Committee (ECC). The pincipal pupose of the ECC's wok was to addess deficiencies, including management issues, that wee identified though an intenal eview. The ECC engaged in a thoough eview of the existing management stuctue and substantially evised it. A specific management pocess was defined with a clea aticulation of oles, esponsibilities, and oganizational elationships. Seveal management evisions have been identified by the ECC and ae being implemented. These changes ae documented in the Education Famewok. The office oganization has been evised with new positions specified to moe clealy define oles and esponsibilities. In paticula, a new Assistant Administato fo Education has been identified and has assumed leadeship. One significant change is the designation of Outcome Manages who ae esponsible fo ensuing pojects ae implemented in a way that ensues the outcomes will be achieved. Anothe significant change is the tansfe of poject management esponsibilities to the field centes. Cente education offices povide expetise in state standads and equiements in thei geogaphic aea. These offices also wok closely with thei egional custome base to suppot systematic school impovement pogams. Evidence: Repot of the NASA Administato-chateed team to eview the education pogam; NASA Stategic Coodination Famewok: A Potfolio Appoach.
3.CO1Ae gants awaded based on a clea competitive pocess that includes a qualified assessment of meit? Scoe NO 0% Explanation: The pogam is committed to awading gants though full and open competition to the geatest extent possible and encouages the paticipation of new and less expeienced paticipants. All of ou majo pojects, such as Space Gant, EPSCoR, Univesity Reseach Centes, and the NASA Exploe Schools, select paticipating institutions though a competitive pocess. In some instances, such as EPSCoR, the competition is esticted by legislation to a designated categoy, such as the defined EPSCoR states, but the specific gants selected fo awad ae detemined though a competitive pocess. In the case of Space Gant, the enabling legislation equies that each state have a Space Gant awad but all poposals ae eviewed fo meit. Congessionally diected appopiations epesent an exception to the use of competitive pocesses. Congessional inteest items ae designated to a specific oganization, such as a univesity, obviating the need fo a competitive pocess. (Howeve, in all cases, the pogam equies that the gantee submits a poposal befoe funding is eleased. Poposals ae eviewed to ensue the poject is an effective investment and aligns with NASA famewok and outcomes.) In FY 06, appoximately 50% ($78 million) of NASA's education budget was eallocated to Congessional inteest items as diected in the appopiations epot. Evidence: The equiements fo competition ae fomally specified in the NASA Gants Handbook. Pogam solicitations and poposal guidance also define the equiements fo competition. Each yea, guidance is issued specific to the equiements fo Congessionally diected appopiations.
3.CO2Does the pogam have ovesight pactices that povide sufficient knowledge of gantee activities? Explanation: Each poject in the NASA potfolio is assigned a designated poject manage. It is this manage who is esponsible fo poviding ovesight. The manage's esponsibility is fist to thooughly eview poposals and appove only those poposals that ae found acceptable. Afte awad, manages ae esponsible fo maintaining fequent communications, making site visits as necessay, eviewing status epots, and eviewing pefomance data. The gant poject diectos ae equied, though a povision in the awad document, to submit annual pogess epots. Examples of how these ovesight pactices ae used can be illustated by the Space Gant and EPSCoR pojects which collect highly eliable and compehensive data in compliance with the poject manage's guidance. These data ae used each yea to issue guidance to each state consotium fo impovements. These data ae also used to conduct eviews evey five yeas as equied by legislation specific consotia impovement plans o e-competition of those consotia that do not demonstate effectiveness. As a esult of the last five-yea eview five states wee equied to submit impovement plans o wee teminated. The MUREP potfolio is closely monitoed esulting in an annual epot fo each poject. In FY 05-06, NASA equied each institutional ecipient of a Congessionally Diected Appopiation to submit a poposal accoding to guidance povided by an designated NASA manage which was then eviewed fo compliance pio to awad. Evidence: Ovesight of gantees is in compliance with the NASA Gants and Coopeative Ageement Handbook (Subpat C, 14 CFR Pat 1273) and the guidance of the NASA Shaed Sevices Cente and NSPIRE. Poject manages conduct annual pefomance eviews which ae followed with on-site gantee visitations as necessay. Gantees obtain enewal of thei awads based upon the accomplishment of objective and outcome measues. The equiements fo ovesight ae fomally specified in the NASA Gants Handbook. The annual pefomance measuement guidance to poject manages specifically cites the manages' ovesight esponsibilities. The gant poject diectos ae equied, though a povision in the awad document, to submit annual pogess epots. Scoe YES 10% 3.CO3Does the pogam collect gantee pefomance data on an annual basis and make it available to the public in a tanspaent and meaningful manne? NO 0% Explanation: The pogam cuently does not make gantee pefomance data available to the public. Beginning in FY 07, NASA will publish a publicly accessible annual epot, coveing the pefomance of all gantees, on the NASA education web potal. Evidence: None. 60%
Scoe Scoe Section 4 4.1 Has the pogam demonstated adequate pogess in achieving itsno long-tem pefomance goals? 0% Explanation: As a esult of the estuctuing and stategic planning effots duing FY 2006, a new set of outcomes, objectives, and measues wee defined and eviewed and appoved by OMB. At this point, the longtem measues do not have baselines and no data have been collected (although the FY07 goal fo each long-tem measue is to establish a baseline). Meanwhile, the pogam has continued to assess duing this tansition peiod whee pefomance data fom existing pojects could be mapped to the new set of long-tem goals. Notable achievements by specific pojects exist. Evidence: Data collection potocols must be developed fo the new measues and data must be collected fo many of the new measues. Notable achievements by specific pojects, but not the full Education pogam, exist. Fo example, the NASA Exploe School povide valuable instuctional esouces and pofessional development activities to 251 schools. NES has been ecognized with a "Top 50 Innovations in Govenment Awad" and the SEMAA poject has advanced to finalist status in the 2007 Innovations in Govenment competition. The Jenkins Fellowship pogam has poduced a cumulative total of 21 Ph.D. and 33 M.S. degee ecipients in STEM and notably succeeded in moving five of those gaduates into the NASA wokfoce. National Space Gant College and Fellowship Poject; 2005 data show the following esults: longitudinal fellowship/scholaship awads - 31% employed by NASA, industy, academia, o othe STEM fields, 49% moved on to an advanced degee pogam, and 18% wee seeking STEM employment. Jenkins Pedoctoal Fellowship Pogam - cumulative esults total 26 Ph.D. and 33 M.S. degee ecipients in STEM with 8% of gaduates enteing the NASA wokfoce. Undegaduate Student Reseach Poject - fo 2001-2003 data (most ecently available), longitudinal esults show 26% of paticipants in aeospace-elated employment (vs. 21% of applicants who wee not selected). 4.2 Does the pogam (including pogam patnes) achieve its annual pefomance goals? Explanation: As a esult of the estuctuing and stategic planning effots duing FY 2006, a new set of outcomes, objectives, and measues wee defined and eviewed and appoved by OMB. NASA met the tagets associated with the annual-only output measues. The annual efficiency measues lack data fo FY 2006. SMALL EXTEN T 7% Evidence: See pefomance measues section. NASA met the tagets associated with the annual-only output measues. The annual efficiency measues lack data fo FY 2006.