DEPARTMENT OF DEFENSE Developmental Test and Evaluation FY 2016 Annual Report

Transcription

1 DEPARTMENT OF DEFENSE Developmental Test and Evaluation FY 2016 Annual Report MARCH 2017 Dr. J. Brian Hall Acting Deputy Assistant Secretary of Defense Developmental Test and Evaluation The estimated cost of this report fo r the Department of Defense is approximately $276,000 in Fiscal Years This includes$ I 02,000 in expenses and $ 174,000 in DoD labor. Generated on 20 17Janl0 ReflD: E-03 1C8F2 DISTRIBUTION STATEMENT A: Approved for publ ic release

2 Department of Defense Developmental Test and Evaluation FY 2016 Annual Report Deputy Assistant Secretary of Defense Developmental Test and Evaluation 3090 Defense Pentagon 5A1076 Washington, DC

3 Contents 1 EXECUTIVE SUMMARY Developmental Test and Evaluation Test and Evaluation Workforce DoD Test Resource Management Center Adequacy of Resources DASD(DT&E) ACTIVITIES DASD(DT&E)/TRMC Focus Areas T&E Policy and Guidance Activities T&E Workforce Development Activities Program Engagement Other Activities DOD COMPONENT ASSESSMENTS Updates from DoD Component Assessments DASD(DT&E) Assessment of the DoD Component Reports T&E Acquisition Workforce ADDITIONAL REPORTING REQUIREMENTS Adequacy of Resources for DASD(DT&E) Adequacy of Resources for DoD Component Lead DT&E Organizations DOD TEST RESOURCE MANAGEMENT CENTER TEMP Review Strategic T&E Roadmaps Test Infrastructure Studies T&E/S&T Program Central Test and Evaluation Investment Program (CTEIP) Joint Mission Environment Test Capability (JMETC) Program EA for Cyber Test Ranges National Cyber Range (NCR) Range Sustainability MRTFB T&E Workforce Budget Certification and Trends International Facilities PROGRAM ENGAGEMENT AND ASSESSMENTS DoD Programs Army Programs Navy Programs DASD(DT&E) FY 2016 Annual Report iii

4 Contents 6.4 Air Force Programs ABBREVIATIONS AND ACRONYMS Figures Figure 2-1. DASD(DT&E) FY 2017 to FY 2019 T&E Workforce Development Road Map Figure 3-1. FY 2016 T&E Certification Rates Figure 3-2. FY 2016 T&E Personnel Breakdown Figure 3-3. FY 2016 T&E Personnel Breakdown by DoD Components Figure 5-1. NCR FY 2016 Events Tables Table 2-1. Updates to the FY 2017 T&E Curriculum Table 3-1. FY 2016 T&E Acquisition Workforce Required Certification Levels Table 3-2. Acquisition T&E Workforce Comparison, FY 2015 vs. FY Table 3-3. MDAPs/MAIS Programs and CDTs in FY 2015 vs. FY Table 3-4. MDAPs/MAIS Programs and CDT T&E KLPs in FY 2015 vs. FY Table 4-1. DASD(DT&E) FY 2016 Budget ($K) Table 4-2. DASD(DT&E) Workforce and Contractor Support Table 4-3. List of Lead DT&E Organizations and Programs Table 4-4. Assessment of Adequacy of Resources for Army Lead DT&E Organizations Table 4-5. Assessment of Adequacy of Resources for Navy Lead DT&E Organizations Table 4-6. Assessment of Adequacy of Resources for Air Force Lead DT&E Organizations Table 4-7. Assessment of Adequacy of Resources for MDA Lead DT&E Organization iv DASD(DT&E) FY 2016 Annual Report

5 Index to Program Engagement and Assessments PROGRAM ENGAGEMENT AND ASSESSMENTS...81 DoD Programs Assembled Chemical Weapons Alternatives (ACWA) Ballistic Missile Defense System (BMDS) Defense Agencies Initiative (DAI) Increment Department of Defense Healthcare Management System Modernization (DHMSM) F-35 Lightning II Joint Light Tactical Vehicle (JLTV) Key Management Infrastructure (KMI) Increment Public Key Infrastructure (PKI) Increment Army Programs Abrams M1A2 System Enhancement Package Version 3 (SEPv3) Engineering Change Proposal (ECP) 1a Army Integrated Air and Missile Defense (IAMD) Bradley Family of Vehicles (BFoV) Engineering Change Proposal (ECP) Indirect Fire Protection Capability Increment 2 Intercept (IFPC2-I) Joint Air-to-Ground Missile (JAGM) Joint Tactical Radio System (JTRS) Handheld, Manpack, and Small Form Fit (HMS) Manpack (MP) Radio Joint Tactical Radio System (JTRS) Handheld, Manpack, and Small Form Fit (HMS) Rifleman Radio (RR) M109A7 Family of Vehicles (FoV) Self-Propelled Howitzer (SPH) and Carrier, Ammunition, Tracked (CAT) Vehicle Mid-Tier Networking Vehicular Radio (MNVR) (AN/VRC-118(v)1) Stryker Family of Vehicles (FoV) Engineering Change Proposal (ECP) Navy Programs CH-53K Heavy-Lift Replacement Helicopter COLUMBIA-Class Submarine Distributed Common Ground System Navy (DCGS-N) Increment GERALD R. FORD Class Nuclear Aircraft Carrier (CVN 78) Littoral Combat Ship (LCS) and Mission Packages (MPs) MQ-4C Triton Unmanned Aircraft System (UAS) Multi-Mission Maritime Aircraft (P-8A Poseidon) Trident II Life Extension (D5LE) U.S. Navy Integrated Air and Missile Defense (IAMD) VIRGINIA-Class Submarine ZUMWALT-Class Destroyer (DDG 1000) Air Force Programs Air and Space Operations Center Weapon System (AOC-WS) Increment DASD(DT&E) FY 2016 Annual Report v

6 Index to Program Engagement and Assessments B61-12 Life Extension Program (LEP) Tail Kit Assembly (TKA) Combat Rescue Helicopter (CRH) Defense Enterprise Accounting and Management System (DEAMS) F-22 Increment 3.2B Modernization Family of Advanced Beyond Line-of-Sight Terminals (FAB-T) Global Positioning System (GPS) Enterprise Joint Space Operations Center (JSpOC) Mission System (JMS) Increment KC-46A Tanker Modernization Small Diameter Bomb Increment II (SDB II) Space-Based Infrared System (SBIRS) vi DASD(DT&E) FY 2016 Annual Report

7 1 EXECUTIVE SUMMARY The Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) submits this Annual Report for Fiscal Year (FY) 2016 in response to section 139b(d)of Title 10, United States Code (U.S.C.). This report addresses activities related to the Major Defense Acquisition Programs (MDAPs) including the following: A discussion of the extent to which MDAPs are fulfilling the objectives of their developmental test and evaluation (DT&E) plans. A discussion of the waivers of and deviations from requirements in the Test and Evaluation Master Plans (TEMPs) and other testing requirements that occurred during the preceding year with respect to such programs, any concerns raised by such waivers or deviations, and the actions that have been taken or are planned to be taken to address such concerns. An assessment of the organization and capabilities of the Department of Defense (DoD) for DT&E with respect to such programs. Any comments on such report that the Secretary of Defense considers appropriate. This report includes a separate section that covers the activities of the DoD Test Resource Management Center (TRMC) during FY 2016 and a separate section that addresses the adequacy of resources available to the DASD(DT&E) and the Lead DT&E Organizations of the Military Departments to carry out the responsibilities prescribed by law. This report provides an assessment of the test and evaluation (T&E) workforce and also highlights the engagement activities and assessments of 40 programs (MDAPs, Major Automated Information System (MAIS) programs, and special interest programs designated by the Under Secretary of Defense for Acquisition, Technology, and Logistics (USD(AT&L)). These selected programs reached significant milestones or had significant DT&E activities in FY Developmental Test and Evaluation In FY 2016, DASD(DT&E) continued activities to address its focus areas, initiated in 2013, to be more responsive to acquisition decision makers, to the test community, and to emerging requirements. DASD(DT&E) continued to assist programs in applying the Developmental Evaluation Framework (DEF) concept to their DT&E planning and in developing a DEF that is tailored to the program s unique needs and circumstances. DASD(DT&E) DEF Core Teams engaged with 35 acquisition programs across the DoD warfare areas at various stages within the acquisition development life cycle. DASD(DT&E) developed the DEF as a guide to increase analytical rigor in the development of DASD(DT&E) program assessments of the system s performance, interoperability, cybersecurity, and reliability evaluation. DASD(DT&E) continued to improve guidance to better support program office personnel in developing robust reliability growth programs (RGPs) that include a reliability growth curve (RGC) with appropriate DT&E activities and resources identified, planned, and executed during each phase of system development. The guidance was provided to the Defense Acquisition University (DAU) to DASD(DT&E) FY 2016 Annual Report 1

8 Executive Summary update the T&E acquisition workforce curriculum. DASD(DT&E) also continued to assist the program offices in developing the TEMP, with special attention given to the Milestone (MS) A and MS B TEMP. DASD(DT&E) continued to mature the cybersecurity DT&E process and encouraged programs to plan for and conduct cybersecurity DT&E early and throughout the development process. Early discovery of system vulnerabilities through DT&E can facilitate remediation and reduce the impact on cost, schedule, and performance. DASD(DT&E) used several outreach opportunities to brief the DoD Components on the cybersecurity T&E process and inform the DoD T&E community about T&E activities that are most likely to improve the security posture of fielded systems. DASD(DT&E) also continued to develop, refine, and promote an interoperability guidance that calls for programs to plan and execute interoperability testing earlier in the program life cycle. Conducting interoperability T&E early in the development process provides valuable feedback and information to systems engineers and allows them to make system modifications more efficiently. Of the MDAPs monitored by DASD(DT&E) during FY 2016, including those reported in this annual report, none requested a waiver or deviation from requirements in the TEMP in FY As in the past, this report uses the self-assessment reports provided to the DASD(DT&E) by DoD Components with MDAPs, MAIS programs, and USD(AT&L)-designated special interest programs. For FY 2016, the DoD Components provided updates to their previous reports regarding T&E involvement in early acquisition activities, T&E planning and execution, and T&E personnel. 1.2 Test and Evaluation Workforce This report includes DoD Component-specific information on the designation of Chief Developmental Testers (CDTs), which are the T&E Key Leadership Positions (KLPs) for MDAPs and MAIS programs; the use of Defense Acquisition Workforce Development Fund (DAWDF) funding in support of the T&E workforce; and the adequacy of resources available to the Government organizations serving as Lead DT&E Organizations for the programs being assessed in this report. DASD(DT&E) routinely monitors and reviews the composition of the T&E workforce. As in previous years, DoD Components continue to rely on support contractors and developer T&E support. Non-acquisition-coded, and specifically non-t&e-coded, personnel are still the major contributors to T&E activities. A significant number of T&E resources remain outside this Defense Acquisition Workforce Improvement Act (DAWIA)-certified workforce. DASD(DT&E) continues to work with the DoD Components to ensure that T&E acquisition positions are properly coded (T&E acquisition career field). DASD(DT&E) convened the third T&E KLP Joint Qualification Board in December The board identified 16 individuals, out of 19 applicants reviewed, as qualified to be assigned in a T&E KLP. 2 DASD(DT&E) FY 2016 Annual Report

9 Executive Summary 1.3 DoD Test Resource Management Center The TRMC is responsible for oversight of the DoD test resources. These responsibilities include annual certification of the Service and Defense Agency T&E budgets and wide-ranging studies on topics such as fifth-generation threat requirements; a hypersonics business case; T&E of autonomous systems; costs associated with long-range, open-ocean strategic weapons tests; and test range capabilities to support testing against emerging advanced threats. The TRMC also oversees management of the Central Test and Evaluation Investment Program (CTEIP), the T&E/Science and Technology (S&T) Program, the Joint Mission Environment Test Capability (JMETC) Program, and the National Cyber Range (NCR). In March 2016, the Director, TRMC was designated as the DoD Executive Agent (EA) for Cyber Test Ranges. This report provides descriptions of TRMC activities and initiatives during FY The T&E/S&T Program made significant progress across the eight Test Technology Areas. The CTEIP again made significant progress in development and deployment of test infrastructure capabilities. Fifteen CTEIP projects were successfully completed and 46 projects continued in execution. The JMETC Program made strides toward providing the T&E community with an infrastructure that supports testing across the full spectrum of the acquisition life cycle with a focus on distributed interoperability and cybersecurity test and training. In addition to network connectivity, JMETC also provides tools, services, and subject matter expertise to aid in the planning and execution of distributed test and training events. Work is ongoing on a variety of congressionally directed plans and studies. In the follow-up to an FY 2013 National Defense Authorization Act (NDAA)-directed study, the TRMC led development of a DoD report and plan on the requirements and proposed investments to cost-effectively meet DoD needs through Test range sustainability has grown in importance with the emphasis on renewable energy projects on or near the ranges. In 2016, the TRMC conducted the fourth biennial Test Infrastructure Encroachment Survey. Data from the survey, annual Infrastructure Range Reviews, and field visits to the ranges reaffirmed that the top three factors affecting testing are adjacent land use, airspace limitations, and spectrum encroachment. The TRMC continued to develop required T&E infrastructure improvement solutions and to focus on cybersecurity test capability, with continued development of the NCR. In FY 2016, the NCR supported 58 events for MDAPs, training, and operational exercises. 1.4 Adequacy of Resources The Office of the DASD(DT&E) has a staffing level of 11 organic Government personnel, six detailees, and additional contractor support. Working within available resources, DASD(DT&E) focuses its activities on MDAPs, with additional support to MAIS programs and special interest programs as designated by the USD(AT&L). DASD(DT&E) also devotes resources to support T&E acquisition workforce development. In FY 2016, DASD(DT&E) continued the process to assess the adequacy of resources available to the Lead DT&E Organizations to carry out their responsibilities. The DoD Components provided information on the designation of Lead DT&E Organizations for 40 programs. DASD(DT&E) DASD(DT&E) FY 2016 Annual Report 3

10 Executive Summary assessed the T&E expertise and capabilities provided by these Lead DT&E Organizations and funding to support DT&E activities. 4 DASD(DT&E) FY 2016 Annual Report

11 2 DASD(DT&E) ACTIVITIES 2.1 DASD(DT&E)/TRMC Focus Areas DASD(DT&E) and TRMC focus areas encompass those actions initiated in 2013 that the two organizations are developing to be more responsive to acquisition decision makers, to the test community, and to emerging requirements of the Nation s test infrastructure. These areas are in alignment with the principles of Better Buying Power (BBP) 3.0, introduced on September 19, Implementing the Developmental Evaluation Framework (DEF) Background. DoD Instruction (DoDI) , Operation of the Defense Acquisition System, instructs program managers (PMs) to describe a developmental evaluation methodology in the TEMP starting at MS A that will provide essential information on programmatic and technical risks as well as information for major programmatic decisions. Starting at MS B, the TEMP will include a DEF. The DEF is a tailorable, standardized, tabular means for acquisition programs to clearly and succinctly articulate the TEMP s DT&E strategy, which is focused on system evaluation to inform acquisition, programmatic, and technical decisions. The DEF identifies key data that will contribute to assessing system performance, interoperability, cybersecurity, reliability, and maintainability; the DEF shows the correlation and mapping between decisions, information/data requirements, test events, and key resources. The DEF is the basis for the DT&E plan and is designed to improve DT&E planning and streamline the TEMP. To enable proper application of the DEF concept, DASD(DT&E) developed, chartered, and deployed DEF Core Teams, beginning in FY 2015, to engage with the acquisition programs at their locations to assist in developing a DEF that is tailored to the program s unique needs and circumstances. In FY 2016, DASD(DT&E) DEF Core Teams engaged with 35 acquisition programs across the DoD warfare areas at various stages within the acquisition development life cycle. Since FY 2015, DASD(DT&E) DEF Core Teams have engaged with 60 acquisition programs. The initial DEF developed during these program engagements was further refined by program office personnel, and then inserted into the TEMP to assist in defining the DT&E strategy. In addition to direct program engagements for socializing and instantiating the DEF, DASD(DT&E) developed a DEF section that will be included in the next release of the Defense Acquisition Guidebook (DAG) to provide detailed guidance to the PM and T&E practitioners. DASD(DT&E) presented educational sessions on the DEF at acquisition centers, professional conferences, and symposiums. DASD(DT&E) continues to increase the analytical rigor of DT&E program assessments by using the DEF as a guide to define the system s performance, interoperability, cybersecurity, and reliability evaluation needed to inform the decision point. Next Steps. DASD(DT&E) will continue to assist programs in applying the DEF concept to their DT&E planning. In FY 2017, DASD(DT&E) will ensure that a DEF feedback loop is in place to improve a program s subsequent DEFs. Future annual reports will document updates, as needed. DASD(DT&E) FY 2016 Annual Report 5

12 DASD(DT&E) Activities Implementing the TEMP at MS A Background. DoDI requires that a TEMP be developed and approved beginning with the MS A decision. To assist policy implementation in FY 2016, DASD(DT&E) continued the development and refinement of the TEMP checklist for use with the MS A TEMP. The checklist provides guidance to address DT&E strategies and methodologies focused on assessing technology maturity and early program risk reduction, DEFs, T&E decision support matrixes, database management, modeling and simulation (M&S), and early identification of required program T&E resources. Next Steps. DASD(DT&E) will continue to develop and refine the guidance for implementing the TEMP at MS A and will finalize the guidance and disseminate it to the T&E community in the DAG update. Future annual reports will document updates, as needed Improving Reliability T&E Background. Beginning in FY 2015, DASD(DT&E) emphasized early DT&E to assess reliability program risks and influence design for reliability planning and reliability growth programs (RGPs). The focus is to ensure that program offices develop a robust and effective RGP, which includes a reliability growth curve (RGC) with appropriate DT&E activities and resources identified, planned, and executed during each phase of system development to inform key decisions from analysis of alternatives through completion of Engineering and Manufacturing Development (EMD). RGP execution and tracking RGC progress would enable sufficient knowledge for decision makers to judge risks about whether reliability requirements will be met throughout the developmental life cycle for each mission-critical subsystem, software build, and integrated system in a mission context. DASD(DT&E) goals for improving reliability T&E include the following: Provide guidance on and promotion of tools and techniques for tracking RGC progress, and evaluation and reporting for hardware and software reliability. Ensure that RGPs, which include RGCs, are collaboratively developed with systems engineering (SE) and included in the MS B TEMP and that reliability growth is monitored and reported throughout the acquisition process. Foster close relationships between CDTs and Chief Engineers within the program offices so that the initial reliability estimate is accurately determined before system-level reliability growth testing begins. In FY 2016, the DASD(DT&E) issued a memorandum providing guidance for DASD(DT&E) Staff Specialists to use during their engagements with program offices to improve reliability T&E. DASD(DT&E) provided DAU with information to improve reliability T&E in the T&E curriculum. Next Steps. DASD(DT&E) continues to improve guidance to better support program office personnel in developing robust RGPs during each phase of system development. The guidance was provided to DAU to update the T&E acquisition workforce curriculum and will be included in the next DAG update. Future annual reports will document updates, as needed. 6 DASD(DT&E) FY 2016 Annual Report

13 DASD(DT&E) Activities Improving Cybersecurity DT&E Background. DASD(DT&E) has emphasized the importance of cybersecurity testing within acquisition programs. DT&E must ensure that cybersecurity measures designed and implemented within systems perform as intended and provide adequate protection and resiliency. Systems built with security in mind and tested for security deficiencies will be more resilient, more trustworthy, easier to defend, and more effective for Warfighters. In FY 2016, DASD(DT&E) continued to encourage programs to plan for and conduct cybersecurity DT&E early and throughout the development process. Early discovery of system vulnerabilities through DT&E can facilitate remediation and reduce the impact on cost, schedule, and performance. DASD(DT&E) used several outreach opportunities to brief the DoD Components on the cybersecurity T&E process and inform the DoD T&E community about T&E activities that are most likely to improve the security posture of fielded systems. DASD(DT&E) has an individual, through the Intergovernmental Personnel Act, to support implementation of the DASD(DT&E) cybersecurity T&E guidance. DASD(DT&E) contributed to development of the Directive-Type Memorandum (DTM) regarding cybersecurity, released in January The DTM assigns, reinforces, and prescribes procedures for acquisition responsibilities related to cybersecurity in the Defense Acquisition System, including how cybersecurity is addressed in DT&E. DASD(DT&E) also worked with the Joint Staff; the Deputy Assistant Secretary of Defense for Systems Engineering (DASD(SE)); the Deputy Assistant Secretary of Defense for Command, Control, Communications, Cyber, and Business Systems; the DoD Chief Information Officer (CIO), and other DoD organizations to develop a guide that will help requirements writers define the cyber survivability component of the System Survivability Key Performance Parameter (KPP). Currently, cybersecurity requirements for most programs are compliance requirements in accordance with the DoD Information Assurance Certification and Accreditation Process (DIACAP) or the Risk Management Framework (RMF). These compliance requirements, although necessary, need to be augmented with the cybersecurity DT&E process to measure performance against any given threat. Having a System Survivability KPP, with a cybersecurity component, will enable testers to evaluate threshold and objective measures of performance. Next Steps. In FY 2017, DASD(DT&E) will encourage programs to use cybersecurity-focused activities such as tabletop exercises to determine areas in which programs should focus cybersecurity DT&E activities. These cybersecurity tabletop exercises bring together engineers, testers, and cybersecurity subject matter experts to conduct a system analysis and identify security threats and risks to the system. DASD(DT&E) staff works with programs to provide consistent cybersecurity T&E assistance, advice, and best practices to acquisition programs, particularly with regard to TEMPs, requests for proposals (RFPs), and DEFs. Future annual reports will document updates, as needed Improving Interoperability DT&E Background. In accordance with DoDI , demonstrated interoperability is one of the criteria considered by the Milestone Decision Authority at MS C. DoDI also requires the DASD(DT&E) FY 2016 Annual Report 7

14 DASD(DT&E) Activities DASD(DT&E) to submit a DASD(DT&E) program assessment, including an evaluation of interoperability activities conducted to date, to the USD(AT&L) to support a MS C decision. For a DoD system to interoperate effectively and complete its mission, interoperability must be carefully planned, designed, and built into systems during development. A well-planned, wellresourced, and well-executed DT&E program provides waypoints along the development path to ensure that a system is on course to meet its interoperability requirements. These developmental test (DT) activities will generate data that can be evaluated, assessed, and reported to systems engineers and decision makers. With the information provided, the Milestone Decision Authority, at MS C, will determine whether a system has demonstrated interoperability adequately enough to proceed (interoperability being one of many criteria). In FY 2016, DASD(DT&E) continued to develop, refine, and promote interoperability DT&E guidance that calls for programs to plan and execute interoperability DT earlier in the program life cycle. Conducting interoperability T&E early in the development guidance provides valuable feedback and information to systems engineers and allows them to make system modifications more efficiently. When designing tests, greater emphasis has been placed on identifying and using an endto-end set of activities to accomplish the mission as well as the supporting critical functions, interfaces, and information exchanges. DASD(DT&E) continues to emphasize that whenever possible, DT&E should be planned and executed in such a way that the Joint Interoperability Test Command (JITC) can leverage the data to support interoperability certification (occurring in initial operational test and evaluation (IOT&E) after MS C). DASD(DT&E) briefed the Services and the Defense Information Systems Agency (DISA) on this guidance and received helpful feedback. The DT&E guidance stresses that interoperability DT&E activities should be included in the overall DT&E strategy from the onset of planning. The TEMP should outline interoperability DT&E activities that support each milestone and provide early feedback at design reviews (preliminary design review (PDR) and critical design review (CDR)). Testing should occur in representative environments, and T&E resource planning must consider contested environments, security mechanisms, mission traffic, and system-of-systems (SoS) infrastructures. DT&E planning should match the limitations and critical information requirements in the Interface Control Document. Next Steps. In FY 2017, DASD(DT&E) will continue to stress the importance of examining interoperability during cybersecurity activities to develop DT&E plans in the TEMP. Including interoperability in the cybersecurity tabletop exercises will assist in identifying interoperability issues that should be addressed in the test program. DASD(DT&E) will continue to reach out to programs and incorporate lessons learned on the interoperability DT&E process, and DASD(DT&E) will continue to ensure that TEMPs contain an interoperability DT&E plan that will demonstrate interoperability by MS C. DASD(DT&E) will work with DAU to ensure that courses include the new guidance on interoperability T&E. Future annual reports will document updates, as needed Improving and Maintaining Hypersonic Test Infrastructure Background. The Office of Science and Technology Policy (OSTP), working with the Secretary of Defense and the Administrator of the National Aeronautics and Space Administration (NASA), completed a study, as directed by the FY 2013 NDAA, on the ability of the national T&E 8 DASD(DT&E) FY 2016 Annual Report

15 DASD(DT&E) Activities infrastructure to effectively and efficiently mature hypersonic technologies for defense systems development in the short term and long term. The study evaluated the capabilities of existing ground test facilities and open-air ranges (OARs). It found that although many existing facilities provide substantial capability for testing weapon systems in the hypersonic flight regime, capability gaps exist in ground test facilities, OARs, and M&S. Existing facilities were created to evaluate strategic deterrent, missile defense, and space access systems; however, sustained controlled flight in the hypersonic flight regime requires a more realistic emulation of this extreme environment to better understand the physics and chemistry. Testing that can replicate the broad spectrum of operational environments encountered within the hypersonic flight regime will require special considerations. The T&E community identified a need to develop a T&E methodology capable of supporting the effective and efficient development of maturing hypersonic technologies for defense systems and reduce risk before initiating program operation in these speed regimes. The set of expected test requirements will be based on the likely characteristics to be validated for near-term developmental hypersonic systems. In the follow-up to the OSTP study, the TRMC led development of a DoD report and plan on the requirements and proposed investments to meet DoD needs through In FY 2016, the TRMC sent the report to the appropriate congressional committees, and it provided an effective business case for a successful request of $350 million over the Future Years Defense Program (FY 2017 FY 2022) to develop a suite of infrastructure for hypersonic weapon system research and development (R&D). Next Steps. DASD(DT&E) and the TRMC will develop a suite of infrastructure for hypersonic weapon system R&D. Future annual reports will document updates, as needed Incorporating/Improving Big Data/Knowledge Management (KM) in T&E Background. The current KM capabilities and processes used to gain, collect, and analyze the information necessary to conduct acquisition assessments and evaluations are deficient and ineffective for today s world. Embracing an enterprise approach to T&E KM leveraging commercially proven big data analytics technologies can efficiently and continuously improve the knowledge base throughout the life cycle of a system. Improved analytical tools and methods are needed to quickly and accurately reduce the data and analyze performance within these environments. In addition, T&E data are currently compartmentalized with little discovery or usage outside of the specific event, acquisition program, and organization for which the data were collected. T&E data must be shared and leveraged across programs and throughout the life cycle to allow learning from the knowledge attained by others. In FY 2016, efforts to improve DoD s ability to quickly and thoroughly analyze the large amounts of test data it collects were focused on development of improved KM capabilities and the introduction of big data analytics. Work in this area, led by CTEIP, seeks to increase the speed of development, testing, and deployment of systems by leveraging commercial and Government development tools, techniques, and best practices. The Joint Strike Fighter KM Joint Improvement and Modernization (JIM) project is a proof of concept to ascertain how well KM capabilities and big data tools can assist large acquisition programs in discovering trends, dependencies, and unknowns through analysis of DASD(DT&E) FY 2016 Annual Report 9

16 DASD(DT&E) Activities multiple sets of flight test data at distributed test locations. The Collected Operational Data Analytics for Continuous T&E JIM project is consolidating and analyzing very large data sets across multiple commodity areas (e.g., automotive test vehicle data and theater-collected performance/ reliability data) to diagnose and visualize complex trends and undiscovered issues. To date, both proof-of-concept projects have led to cost savings resulting from earlier issue identification and avoiding retest. Next Steps. The TRMC will continue efforts to develop reference architecture and an investment road map for delivering big data analytic services to the DoD test ranges. Future annual reports will document updates, as needed Understanding and Improving T&E of Autonomous Systems Background. As DoD weapon systems are implementing more and more autonomous capabilities, T&E methods and approaches need to adapt to enable DoD to acquire and effectively employ systems that work as expected. DASD(DT&E) and the TRMC have embarked on a multifaceted initiative to develop a DT&E paradigm to establish trust in the validity of T&E methods. DoD leadership, users, and the public need assurances that autonomous systems will perform their missions as safely and securely as possible. An output of the initiative is to identify the T&E capabilities that will be needed to adequately test and evaluate the emerging variety of increasingly capable self-operating, and possibly self-learning, systems. The initiative will address T&E methodologies and resources as well as improve the T&E training curriculum needed for the T&E of autonomous systems. Ensuring that weapon systems will respond appropriately to all of the possible inputs will exceed the capability of the current conventional T&E methods and be very costly. DASD(DT&E) and the TRMC believe that addressing this challenge will require a combination of M&S to explore thousands of test cases, statistically measuring system performance against the desired standard, and then conducting a set of real-world test cases of the system to ensure that the modeled and real-world behaviors match predictions across the range of system performance. In FY 2016, DASD(DT&E)/TRMC contracted for two studies specific to T&E of autonomous systems: one to address the issues of testing advanced synthetic capabilities to perform functions independent of human control in four general domains: perception, cognizance, decision making, and execution; the other to address gaps in the T&E infrastructure and to develop an investment strategy. Next Steps. DASD(DT&E) and the TRMC will begin activities to develop a paradigm for DT&E of autonomous systems throughout their entire life cycle, develop an investment plan to address immediate test capability gaps, and determine T&E acquisition workforce education and training needs. Future annual reports will document updates, as needed. 2.2 T&E Policy and Guidance Activities Background. In FY 2016, DASD(DT&E) drafted updates and restructured the T&E section (Chapter 8) of the DAG to align with DoDI and several other updated DoDIs. 10 DASD(DT&E) FY 2016 Annual Report

17 DASD(DT&E) Activities DASD(DT&E) made significant updates to the sections on cybersecurity DT&E, interoperability DT&E, reliability T&E, the DEF, scientific test and analysis techniques (STAT) in T&E, and the TEMP. Next Steps. Based on guidance from the USD(AT&L), DASD(DT&E) will formalize the draft update to Chapter 8 of the DAG for distribution. 2.3 T&E Workforce Development Activities The DASD(DT&E) serves as the functional leader for the T&E career field. In this capacity, the DASD(DT&E) role is to establish, oversee, and maintain education, training, and experience requirements including competencies and certification standards, the T&E position category description (PCD), and the T&E content of DAU courses as current, technically accurate, and consistent with DoD acquisition policy. During FY 2016, DASD(DT&E), the DAU T&E Performance Learning Director, and T&E course managers conducted an annual review of the T&E curriculum. The T&E Functional Integrated Product Team (FIPT) reviewed the T&E Workforce Competency Model, the T&E PCD, T&Especific requirements for the CDT, T&E Certification Guides, and T&E training standards. Based on the review, several FY 2017 curriculum updates were made. DAU added cybersecurity to the DoD T&E Workforce Competency Model; updated TST 102, TST 204, and TST 303 course material; and added the EVM 101 (Fundamentals of Earned Value Management) course to the Level III Core Plus Development Guide for assignment to Headquarters and Staff and PM/Matrix Support. T&E experience, training, and education requirements; the T&E PCD; and T&E Certification Guides for T&E Level I and Level II remained unchanged. In addition, DASD(DT&E) recommended to DAU that CLM 016 (Cost Estimating) and CLE 003 (Technical Reviews) be prerequisites for TST 204. DASD(DT&E) also recommended changes to the T&E Overview section of ACQ 101 (Fundamentals of Systems Acquisition Management) to reflect current policy and guidance, as well as an update to CLR 101 (Introduction to the Joint Capabilities Integration and Development System (JCIDS)) to reflect the latest JCIDS process. DASD(DT&E) updated the T&E workforce development road map (FY 2017 to FY 2019) to assist in T&E workforce development through annual improvement blocks. The goal is to continuously improve the curriculum so that T&E professionals are prepared and capable of performing their critical roles throughout the acquisition life cycle. This road map is reviewed annually and updated as required. Figure 2-1 depicts the road map for FY 2017 to FY DASD(DT&E) FY 2016 Annual Report 11

18 DASD(DT&E) Activities Figure 2-1. DASD(DT&E) FY 2017 to FY 2019 T&E Workforce Development Road Map T&E Curriculum Updates Background. The DASD(DT&E) reviewed and certified the FY 2017 T&E core curriculum on June 14, The T&E workforce development road map guided the review and identified the updates needed for the T&E courses (TST 102, TST 204, and TST 303). Table 2-1 shows key updates to the FY 2017 T&E curriculum. Table 2-1. Updates to the FY 2017 T&E Curriculum Course TST 102 TST 204 TST 303 Updates Test Readiness Review (TRR) Expand information on TRRs Risk Management Introduce concept of T&E Risk Management TEMP Format T&E Interoperability Introduce concept Reliability T&E MS A TEMP DEF TEMP Format T&E Interoperability Expand on concept DEF Reliability T&E MS A TEMP Cybersecurity TEMP Format 12 DASD(DT&E) FY 2016 Annual Report

19 DASD(DT&E) Activities Next Steps. DASD(DT&E) will continue to monitor and annually review the curriculum in accordance with the functional leader responsibilities assigned in DoDI , Operation of the Defense Acquisition, Technology, and Logistics Workforce Education, Training, and Career Development Program. DASD(DT&E) will continue to implement the DASD(DT&E) road map for the T&E workforce (see Figure 2-1). In addition, DASD(DT&E) will continue to review student feedback on T&E courses and incorporate changes to improve the overall quality and usefulness of the material Acquisition Support to AT&L Workforce Development Background. Section 1706 of Title 10, U.S.C., establishes a goal to have a properly qualified military officer or civilian employee of the Department to serve as a CDT for each MDAP and MAIS program. The November 8, 2013, USD(AT&L) memorandum, Key Leadership Positions and Qualification Criteria, designated the CDT as a mandatory KLP for each MDAP and MAIS program and initiated a requirement for KLP Qualification Boards to be established and convened in This focus area is one of the initiatives of the USD(AT&L) BBP. The initiative is to improve the professionalism of the total acquisition workforce by establishing higher standards for KLPs and stronger professional qualification requirements for all acquisition specialties. T&E was the lead career field to develop and implement the KLP Qualification Board process. The DASD(DT&E) has convened three T&E KLP Joint Qualification Boards (December 2014, December 2015, and December 2016). The three boards identified 59 individuals, out of 89 applicants reviewed, as prepared to fill mandatory T&E KLPs based on their training, education, and experience. The results were provided to the USD(AT&L) via the Director, Human Capital Initiatives. Eleven of the 59 individuals are currently serving in CDT positions coded as T&E KLPs. Initially, board qualification will be a discriminator in KLP selection, but over time, it is expected to become necessary, with rare exceptions. Qualification will allow the DoD and Components acquisition leadership to create a pool of Level III certified personnel who are ready to fill KLPs as well as assist in acquisition workforce talent management and succession planning. Next Steps. DASD(DT&E) will continue to support the USD(AT&L) BBP initiative and will conduct follow-on boards annually to select and increase the pool of T&E professionals qualified to be assigned in a T&E KLP. Future annual reports will document updates, as needed. 2.4 Program Engagement DASD(DT&E) assists acquisition decision makers by providing an impartial evaluation of a program s status and risks prior to a key milestone decision. Program insight comes from early and continuous engagement with MDAPs, MAIS programs, and USD(AT&L)-designated special interest programs. In FY 2016, DASD(DT&E) advised 24 Defense Acquisition Boards (DABs) and 31 Overarching Integrated Product Teams (OIPTs). DASD(DT&E) completed 30 DASD(DT&E) program assessments and engaged closely with program offices to help develop 28 TEMPs. DASD(DT&E) worked with the TRMC to assess the adequacy of resources available to the programs. DASD(DT&E) FY 2016 Annual Report 13

20 DASD(DT&E) Activities 2.5 Others Activities Scientific Test and Analysis Techniques (STAT) in T&E Background. DASD(DT&E) continues to support execution of the STAT in T&E Implementation Plan. The STAT in T&E Center of Excellence (COE), a key component of the implementation plan, continues to assist acquisition programs in the use of STAT to generate T&E efficiencies; provide rigorous, defensible T&E strategies and results; and improve the level of knowledge for the DT planning, execution, and analysis process. In FY 2016, the STAT in T&E COE provided support to 38 programs: Department of the Army (6) o Armored Multi-Purpose Vehicle (AMPV) o Common Infrared Countermeasures (CIRCM) o Indirect Fire Protection Capability Increment 2 Intercept (IFPC2-I) o Integrated Air and Missile Defense (IAMD) o Logistics Modernization Program (LMP) Increment 2 o Stryker Family of Vehicles (FoV) Engineering Change Proposal (ECP) Department of the Navy (12) o Aegis Flight III o Distributed Common Ground System Navy (DCGS-N) Increment 2 o GERALD R. FORD Class Nuclear Aircraft Carrier (CVN 78) o LHA(R) Amphibious Assault Ship (Flights 0 and 1) o Maritime Tactical Command and Control (MTC2) o MQ-25 Stingray Unmanned Aerial Vehicle (UAV) o Multi-Mission Maritime Aircraft (P-8A Poseidon) o Navy Electronic Procurement System (EPS) o Next Generation Enterprise Network (NGEN) o Next Generation Jammer (NGJ) o Ship-to-Shore Connector (SSC) o Ship s Signal Exploitation Equipment (SSEE) Modifications Department of the Air Force (17) o Advanced Pilot Training (APT) o Air and Space Operations Center Weapon System (AOC-WS) Increment 10.2 o Air Force Integrated Personnel and Pay System (AF-IPPS) 14 DASD(DT&E) FY 2016 Annual Report

21 DASD(DT&E) Activities o B61-12 Life Extension Program (LEP) Tail Kit Assembly (TKA) o Combat Rescue Helicopter (CRH) o Enhanced Polar System (EPS) o Global Positioning System (GPS) Generation III (GPS III) o Ground-Based Strategic Deterrent (GBSD) o Intercontinental Ballistic Missile (ICBM) Fuze W78/W88-1 o Joint Space Operations Center (JSpOC) Mission System (JMS) Increment 2 o KC-46A Tanker Modernization o Military GPS User Equipment (MGUE) o Next Generation Operational Control Segment (OCX) o Presidential Aircraft Recapitalization (PAR) o Space Fence o Space-Based Infrared System (SBIRS) o SBIRS Survivable/Endurable Evolution (S2E2) Department of Defense (3) o F-35 Lightning II o Joint Light Tactical Vehicle (JLTV) o Next Generation Diagnostic System (NGDS) In FY 2016, the COE s key contributions to the Acquisition Category (ACAT) I programs included the following: For the B61 program, the COE worked with the program and contractor test teams on building M&S and live-flight experimental designs for the tail kit. Interaction with this composite test team ensured that the program test team s objectives were achieved. For the GBSD program, the Lead DT&E Organization leveraged the COE s expertise in evaluating multiple methods for determining and comparing sample size of performance and reliability metrics. The COE assisted in the development of an analysis and reporting strategy to quantify risk and variability for the decision makers. For the CIRCM program, the COE developed multiple iterations of experimental designs for several test venues. The designs cover all stages of the test process from M&S to hardware-inthe-loop (HWIL) testing to field testing. The collaborative relationship with the program led to the development of an overarching test strategy. In addition to direct program support and assistance to requests through the Ask a STAT Question option on the COE s website, the COE launched three sponsor-funded research and education initiatives from the Air Force and the Navy. The COE also developed numerous products aimed at providing relevant information to T&E practitioners. The products (best practices, case studies, DASD(DT&E) FY 2016 Annual Report 15

22 DASD(DT&E) Activities practitioner tools, STAT lesson, and journal articles) are available on the COE Website ( Next Steps. Based on available funds, DASD(DT&E) plans to continue support of the STAT in T&E COE. The COE is an important resource for CDTs and Lead DT&E Organizations to use when developing more cost-efficient and effective DT&E programs. DASD(DT&E) views the COE as a key contributor over the long term in improving acquisition outcomes Lead DT&E Organization Background. DoDI requires that each MDAP be supported by a governmental test agency, serving as the Lead DT&E Organization for the program. Section 4.2 of this report describes FY 2016 activities related to implementation of the Lead DT&E Organization. Next Steps. DASD(DT&E) will continue to report on Lead DT&E Organizations in future reports Cost of DT&E Background. DoD R, Department of Defense Financial Management Regulations (FMRs), instructs formulation of the T&E Exhibit dash one (T&E-1) needed for review and analysis of DoD Component T&E funding requirements. In FY 2016, DASD(DT&E), with assistance from the TRMC, reviewed the FY 2017 budget submissions for T&E funding to ensure that T&E resources are adequately funded; programs are properly identifying funds for Lead DT&E Organizations and DT; DoD is not maintaining unwarranted test capabilities at private industry facilities, and unwarranted duplication does not exist among DoD Component assets; test facilities and capabilities required are adequately funded and supported; and new major test facilities are warranted and meet the needs of the DoD Components. In addition, the information in the T&E-1 budget submissions is used to document the test resource requirements in the TEMPs. Next Steps. DASD(DT&E), with assistance from the TRMC, will continue to review future budget submissions for adequacy of T&E funding and resources. Future annual reports will document updates, as needed Modeling and Simulation (M&S) Background. DASD(DT&E), in coordination with the T&E Modeling and Simulation Working Group, conducted a major revision to the DAU Continuous Learning Module (CLM) on M&S for T&E during FY 2015 FY In FY 2016, DASD(DT&E) worked with DAU and its contractor to finalize course development. The updated M&S for T&E CLM became part of the DAU curriculum in April Next Steps. DASD(DT&E) will work with DAU to monitor student comments and make revisions, as appropriate, to the CLM. Future annual reports will document updates, as needed. 16 DASD(DT&E) FY 2016 Annual Report

23 DASD(DT&E) Activities HALT and HASS Methodology Report to Congress Background. House Report , page 189, accompanying H.R. 1735, the NDAA for FY 2016, requested the USD(AT&L), in consultation with the acquisition executives of each Service, to assess the value, feasibility, and cost of greater utilization of highly accelerated life testing (HALT) and highly accelerated stress screening (HASS) methodology to shorten design and development timelines, reduce system and component testing and life cycle costs, and enhance reliability of critical military system components and subcomponents. In FY 2016, DASD(DT&E), in coordination with Service representatives, assessed the value, feasibility, and cost of greater utilization of HALT and HASS methodology in DoD. The submitted report also included a description of plans regarding the use of HALT and HASS methodology in ongoing and future defense programs, along with recommendations to improve the Department s efforts. This information was also provided to DAU to improve the T&E curriculum material. Next Steps. This action is complete Common Aviation Command and Control System (CAC2S) Case Study Background. The USD(AT&L) directed DASD(DT&E) to develop a case study, based on the CAC2S program T&E activities, to showcase the success and value of integrated developmental and operational test planning and execution. CAC2S is an ACAT I MAIS program designed to replace the current Marine Corps direct air support center, tactical air operations center, and tactical air command center functionality. The integrated test provided shared data to support independent analysis, evaluation, and reporting by the DT&E and operational test and evaluation (OT&E) communities. In FY 2016, DASD(DT&E) conducted a kickoff meeting with DAU and the CAC2S Program Management Office (PMO). The PM CAC2S provided key reference material (TEMP, detailed test plan, test reports) to DAU for use in developing the case study, and DAU began to conduct interviews with PM CAC2S and Director of Operational Test and Evaluation (DOT&E) personnel. Next Steps. In FY 2017, the case study working group (DASD(DT&E), DOT&E, DAU, and PM CAS2S) will review the draft case study and conduct formal coordination. Future annual reports will document updates, as needed. DASD(DT&E) FY 2016 Annual Report 17

24 This page intentionally blank. 18 DASD(DT&E) FY 2016 Annual Report

25 3 DOD COMPONENT ASSESSMENTS The following DoD Components provided self-assessments in support of the DASD(DT&E) annual report: Department of the Army (Army), Department of the Navy (DON), Department of the Air Force (Air Force), Defense Information Systems Agency (DISA), and Missile Defense Agency (MDA). For FY 2016, the DoD Components provided updates regarding T&E involvement in early acquisition activities, T&E planning and execution, and T&E personnel. In addition, the DoD Components provided details of the T&E workforce composition including all categories of T&E personnel and addressed specific areas of concern from the DT&E FY 2015 Annual Report. Summaries and assessments of the DoD Component responses are provided in the following sections. The DoD Components continued to actively participate in DASD(DT&E)-led working groups, such as the T&E Working Group (TEWG), the T&E FIPT, and groups updating T&E policy and guidance. During FY 2016, these groups continued to support efforts to develop and implement the T&E KLP Joint Qualification Board, draft the T&E chapter of the DAG, and revise CLMs for T&E certification. 3.1 Updates from DoD Component Assessments The DoD Components reported on progress and improvements in T&E acquisition workforce certification rates across the T&E acquisition workforce. The DASD(DT&E) overall goal for certification is for 90 percent of the T&E acquisition workforce to be either certified or within the 24-month grace period for certification. The overall T&E acquisition workforce is exceeding this goal with 95 percent either certified or within the grace period. The rates shown in Figure 3-1 are taken from the AT&L Workforce Data Mart as of the end of FY Figure 3-1. FY 2016 T&E Certification Rates DASD(DT&E) FY 2016 Annual Report 19

26 DoD Component Assessments Table 3-1 shows the composition of the T&E acquisition workforce required certification levels in FY The table is based on data from the AT&L Workforce Data Mart and includes only the T&E-coded positions at the Military Departments, MDA, and DISA. The DON had a reduction in T&E positions coded at Level III from 54 percent in FY 2015 to 37 percent in FY Similar to previous reports, the Air Force has the lowest percentage of T&E positions coded at Level III. The Air Force considers the balance of Level I, II, and III positions within the T&E portfolio to match the duty responsibilities required and continues to review and ensure that all T&E positions are properly coded and at an appropriate certification level. The DASD(DT&E) has stated the position that achieving Level III training and certification should be a goal for the DoD Components in the management of their T&E acquisition workforce positions. DASD(DT&E) remains concerned about the limited number of Level III positions in Air Force T&E. DASD(DT&E) believes that more positions than the number currently reflected in the Air Force unit manpower documents can benefit from the level of training, experience, and expertise that T&E Level III-certified personnel provide. Table 3-1. FY 2106 T&E Acquisition Workforce Required Certification Levels Level Army DON Air Force 4th Estate* Total Level I 2% 20% 15% 1% 13% Level II 36% 43% 70% 36% 51% Level III 62% 37% 15% 63% 36% *4th Estate refers to DoD organizations, other than the Military Services, having DoD manpower resources. Military personnel assigned to the 4th Estate organizations are tracked by the Services. The DoD Components reported on their use of DAWDF funding. DAWDF funds permit the DoD Components to hire new T&E personnel, provide training for new and existing personnel, develop training courses, provide incentives and awards for T&E, and facilitate outreach programs. During FY 2016, DoD Components provided detailed accounts of DAWDF funding used to advance the T&E workforce. The Army used DAWDF funds to train, develop, and retain its acquisition workforce members at all levels, with the majority of the funding directed at journeymen hiring and functional skills development. The DON allocated DAWDF funding to systems commands (SYSCOMs) for execution in accordance with the DON Acquisition Workforce Strategic Plan. DON T&E coordinated the call for DAWDF T&E funds to improve identification and prioritization of critical training needs in SYSCOMs and Warfare/System Centers (W/SCs). DAWDF funds were used to develop and deliver core and specialized technical training for the T&E workforce, strengthen educational partnerships, and establish recruitment incentives for hard-to-fill positions. The Air Force used the funding to support recruitment and retention, training, and salary for a new highly qualified expert position. MDA used DAWDF funds to support salaries, retention incentives, and career-broadening initiatives. DISA used the funding to deliver on-site training courses to enhance the skills of the T&E and overall acquisition workforce. DASD(DT&E) will continue to encourage the DoD Components to use DAWDF funds to recruit and hire, develop and train, and recognize and retain their T&E workforce. 20 DASD(DT&E) FY 2016 Annual Report

27 DoD Component Assessments 3.2 DASD(DT&E) Assessment of the DoD Component Reports In September 2016, DASD(DT&E) requested that the DoD Components provide a self-assessment of capabilities and organizations to support DT&E activities. The self-assessment included the following: New efforts and challenges with T&E involvement in acquisition activities. Updates to processes and procedures that specifically address attracting, developing, retaining, and rewarding T&E personnel and any new challenges in this area. Efforts to develop and mentor future CDTs. Composition of the current T&E workforce. Organizational changes regarding T&E and details of how the changes impact the T&E workforce. Use of DAWDF funding and specific T&E initiatives in support of the T&E acquisition workforce. Listing of MDAPs and MAIS programs in its portfolio, and for each program, the CDT by name and current T&E certification status, whether the position is coded T&E, whether the position is designated as a KLP, and whether the CDT currently meets the KLP requirements. Status of implementation of the November 8, 2013, USD(AT&L) memorandum, Key Leadership Positions and Qualification Criteria (hereafter referred to as the USD(AT&L) KLP memorandum ). Listing of MDAPs in its portfolio, and for each MDAP, the Lead DT&E Organization. Current T&E capabilities; current and future gaps and activities to address the gaps. Initiatives in the following areas: o Cybersecurity T&E o Interoperability T&E o Reliability T&E o T&E of Autonomous Systems Efforts to address DASD(DT&E) recommendations and concerns from previous DT&E annual reports. For FY 2016, DASD(DT&E) assesses that the DoD Components have adequate T&E organization and capabilities to support DT&E activities Army Report Summary The Army report indicates that the state of personnel and other resources required to conduct DT&E within the Army is adequate to support the needs of its acquisition community. The Army is concerned about maintaining this state in future years, under fiscally constrained environment DASD(DT&E) FY 2016 Annual Report 21

28 DoD Component Assessments projections. The Army stated that its T&E enterprise continues to increase efficiencies throughout acquisition programs life cycles. The Army Test and Evaluation Command (ATEC) is the Army s recognized expert developmental tester. ATEC continuously reviews internal policies and procedures to ensure that the structure, framework, and workforce of the organization are aligned to accomplish Army acquisition requirements. This flexibility is essential during these decreased budgetary times. In FY 2016, ATEC continued to support the Army Requirements Oversight Council by ensuring that senior Army leadership understands the data requirements and evaluation strategy that are derived from the critical operational issues and critical operational issue criteria (COIC) as well as the KPPs identified in the Capability Requirements Document. ATEC continued to play an active role in early T&E planning by supporting Army, multi-service, and joint systems efforts including the Materiel Development Decision, Analysis of Alternatives studies, development of the COIC, as well as the Capability Development Document (CDD). ATEC wrote System Evaluation Plans that identified and documented the data required to support each of the acquisition milestones, providing critical input to the TEMP. The system evaluation is focused on operational outcomes rather than technical specifications and identifies whether the materiel solution fulfills the original operational intent. Critical information is provided to the PMs early enough to support the Program Objective Memorandum (POM) process. By participating early in the acquisition process, ATEC can ensure that the long-lead items required for T&E are documented and well understood in the TEMP for MS A. ATEC efforts in FY 2016 also include the following: Partnering with other organizations to establish a Greening Program on Aberdeen Proving Ground, Maryland. The goal is to acculturate Army civilians and give them a greater appreciation and understanding of the rigors and challenges associated with being a Soldier. ATEC also established effective employee engagement strategies to support the Army s effort to improve in the areas of supervisor effectiveness, professional development, and awards and recognition, contributing to the overall priority of readiness. Continuing to review ATEC T&E capabilities to ensure that a comprehensive and viable test infrastructure is maintained while adjusting capacities and requesting that low-usage capabilities be mothballed to comply with fiscal constraints. Initiating an aggressive workforce planning effort to streamline business processes that will allow ATEC to develop a manpower strategy to comply with FY 2019 manpower levels in an effort to minimize any adverse impact on the assigned T&E mission while maintaining core capability and capacities. Continuing to implement cybersecurity best practices to continuously improve the cybersecurity posture for the command as well as providing cybersecurity T&E information earlier in the acquisition life cycle. Maintaining a leadership role in the standardization of test methodology within the Army and continuing to lead DoD efforts to develop munitions safety and suitability for service (S3) assessment test procedures. 22 DASD(DT&E) FY 2016 Annual Report

29 DoD Component Assessments Continuing to support the Army Evaluation Center (AEC)/Army Materiel Systems Analysis Activity (AMSAA) Center for Reliability Growth (CRG). The CRG is funded to develop and share lessons learned from reliability T&E, develop reliability tools and methodology, and contribute to reliability policy development. The Army utilizes its program executive offices (PEOs) in support of the DT&E mission. The Army provided information about PEO Missiles and Space, PEO Ground Combat Systems, PEO Enterprise Information Systems, and Joint PEO Chemical and Biological Defense. Several specialized laboratories and groups across the U.S. Army Research, Development, and Engineering Command (RDECOM) also provide T&E services and expertise. They include the Aviation and Missile Research, Development, and Engineering Center (AMRDEC), where the majority of RDECOM s T&E workforce is located; the Edgewood Chemical Biological Center (ECBC); and the Tank Automotive Research, Development, and Engineering Center (TARDEC). The Army supports the Ronald Reagan Ballistic Missile Defense Test Site (Reagan Test Site) that provides defensive and offensive DT&E for the MDA integrated family of systems, ICBMs, boostglide systems, and space T&E. The Army s military, civilian, and contractor workforce is sufficient to sustain moderate risk to the T&E mission. In FY 2016, the Secretary of the Army directed reassignment of the West Desert Test Center Life Sciences Division from ATEC to RDECOM ECBC in an effort to improve overall biosafety and biosecurity, and directed the PEO Simulation, Training, and Instrumentation Project Manager for Instrumentation, Targets, and Threat Simulators (PM ITTS) to provide oversight to cyber blue teams supporting Army acquisition. To handle this responsibility and other new work related to cyber training, PM ITTS stood up a new organization: Project Lead for Cyber Resiliency and Training. In the DT&E FY 2015 Annual Report, DASD(DT&E) recommended that the Army continue its efforts to identify T&E KLPs to achieve the goal of having a CDT assigned to each of its MDAPs and MAIS programs. The Army stated that the U.S. Army Acquisition Support Center Army Director for Acquisition Career Management (DACM) Office will track and report required KLPs and incumbent data quarterly as required by the USD(AT&L) Human Capital Initiatives Office. The Army DACM Office will provide a quarterly KLP Status Report to responsible agents (PEO human resources points of contact). The KLP Status Report tracks all KLP hiring actions and KLP vacancies to ensure that every KLP selectee/incumbent has met the position requirements or has an approved position waiver on file. The Army DACM Office has developed the draft Army KLP Hiring and Implementation Policy and Procedures, expected to be signed in the 1st quarter FY Cybersecurity T&E T&E and cooperative vulnerability and penetration assessments are providing acquisition partners with actionable information in support of finding and fixing cybersecurity vulnerabilities. The research, development, test, and evaluation (RDT&E) workforce supporting cybersecurity continues to be stretched for adversarial assessments conducted by certified red teams, provided by PM ITTS. The PM ITTS Threat Systems Management Office conducts a protect, detect, react, and restore analysis, which then becomes a part of the adversarial assessment. The adversarial assessment is also provided to the Army Research Laboratory (ARL) Survivability/Lethality Analysis Directorate DASD(DT&E) FY 2016 Annual Report 23

30 DoD Component Assessments (SLAD), which performs the cooperative vulnerability identification and adversarial cybersecurity DT&E. PEO Assembled Chemical Weapons Alternatives (ACWA) established a cybersecurity program to meet DASD(DT&E) and DOT&E cybersecurity guidelines and requirements and collaborated with other cybersecurity organizations, such as the National Security Agency, PM ITTS Threat Systems Management Office, and ARL/SLAD, to plan and implement the cybersecurity program. PEO Missiles and Space, in partnership with AMRDEC, formed a Cyber Security Test Bed initiative to support the integration of cyber architectures, hardware, software, simulations, and connectivity through a specialized framework and virtualization capability to provide a controlled, stable environment to enable cyber experimentation, cyber analysis, and cyber risk reduction for PEO Missiles and Space developmental programs. Interoperability T&E RDECOM s AMRDEC interoperability T&E has evolved as a focus area in AMRDEC s support to the IAMD Project Office. The complexities of working T&E in an SoS context has created an environment in the IAMD Project Office that requires T&E workforce members to develop new skills, processes, and methodologies for assessing system performance internally to the IAMD system with all constituent programs of record and externally with other Army/joint forces. ATEC maintains a leadership role in the standardization of test methodology within the Army and continues to lead DoD efforts to develop munitions S3 assessment test procedures. These documents harmonize test procedural requirements between the tri-services and North Atlantic Treaty Organization (NATO) allies and increase the acceptability of test data. Standardization potentially avoids the need to retest and improves the interoperability of Warfighters at a time of heightened joint operations. Reliability T&E RDECOM s TARDEC is addressing reliability T&E through the ongoing Design for Automotive Reliability in Test effort between AEC and TARDEC, which is addressing the use of physical simulation in the formal suitability evaluation of Army technologies. A second initiative is a TARDEC, PEO Ground Combat Systems, and PEO Combat Support and Combat Service Support Small Business Innovation Research effort that is researching the creation of a test optimization software package/application to weigh the risk of technology immaturity against the test schedule and cost to create a simulation/test track hybrid to maximize the effectiveness of test while minimizing cost and schedule. The Army CRG, co-chaired by AMSAA and ATEC, is a group of dedicated reliability experts who develop the critical tools, methodology, policy, formal guidance, and education needed for improved Army weapon system reliability. The CRG tackles critical reliability challenges on specific Army programs. The intent of the CRG is to significantly increase the reliability of Army weapon systems. For more than 7 years, the CRG has provided a reliability short course to key stakeholders within the community of practice (from the action officer level up to the general officer/senior executive service (SES) level). The primary objective of each reliability short course is to share tools, methods, policy, 24 DASD(DT&E) FY 2016 Annual Report

31 DoD Component Assessments and lessons learned with the acquisition community to help improve the reliability of systems under acquisition. Since inception, the CRG has provided more than 40 reliability training courses to the Army and defense acquisition communities, including four sessions in FY The offerings have also evolved to include in-person and web-based sessions to reach a wider audience. As part of the CRG, AMSAA is responsible for development and maintenance of several reliability tools and models. These tools and models address reliability growth planning, reliability growth tracking, reliability growth projection, early reliability assessment (i.e., scorecards for general systems and software-intensive systems), and RGC risk assessment. In addition, AMSAA has developed reliability-focused contractual language for hardware- and software-intensive programs. The language establishes a structured approach to help product managers contract for key reliability elements. AEC is also responsible for archiving and analyzing historical reliability T&E results for lesson learned and incorporating reliability lessons learned, policies, tools, and methodologies into reliability T&E. In FY 2016, the AEC CRG team focused efforts on several initiatives that impact reliability T&E. T&E Capabilities ATEC continues to operate in a highly resourced-constrained environment and to focus on increased efficiencies, reduced overhead, and capacity reduction to reduce costs while maintaining a complete and viable test capability infrastructure. Efficiencies are also gained through investment in automation and modernization of test capabilities and facilities. Funding reductions and an aging test infrastructure have required a shift in emphasis in ATEC s major instrumentation program from development of new test technologies and capabilities to enterprise-level replacement of existing test assets that have become obsolete and very costly to sustain. ATEC s civilian manpower has been assessed a significant reduction over the POM, which requires a strategic and dynamic plan given that ATEC continues to be faced with an era of increasing uncertainty. ATEC s diverse command-wide missions are supported by a highly technical civilian and contractor workforce. In addition, ATEC s military personnel provide an invaluable infusion of military knowledge and relevant field experience. Such a diverse workforce is deemed critical because it provides the Soldier interface and operational realism with the field Army. As a result of the 2011 Budget Control Act, the Army significantly reduced ATEC s operations and investment funding. This situation continues throughout the current POM, with no major relief in sight. ATEC s POM budget development process for operations analyzes the resources needed across ATEC to accomplish planned workload and sustain the ranges at an adequate level of funding for test capability sustainment and range operations support. T&E of Autonomous Systems RDECOM s TARDEC is collaborating with ATEC (and others) to develop robust/accepted M&S capabilities to support the T&E of autonomy-enabled systems. Autonomous system test capabilities currently exist within ATEC, and enhancements to this test capability are taking place as test requirements evolve and the autonomy of systems under test becomes increasingly more sophisticated. The Product Manager for Unmanned Ground Vehicles (UGVs) is currently investing $1.5 million in a small UGV reliability test course as reliability is a DASD(DT&E) FY 2016 Annual Report 25

32 DoD Component Assessments pressing need for the Product Manager. In partnership with PM ITTS, ATEC is defining a set of requirements for an autonomous robotics both land and air major instrumentation effort beginning in FY The Robotics/UAS Instrumentation Suite is an FY 2017 FY 2025, $27.8 million effort that will develop methodologies and instrumentation for including the safe abort (emergency stop) of systems under test. The Aberdeen Test Center is the ATEC lead for this effort, with the Redstone Test Center and Yuma Proving Ground in support. Re-use is key. ATEC will leverage ongoing efforts in industry (such as autonomous automobiles) and Air Force, Navy, and Marine Corps T&E UAS/robotic instrumentation plus associated safe testing methodologies (e.g., sense and avoid). Initial operational capability (IOC) is scheduled for FY 2020, and the suite should be available to support TARDEC s autonomous convoy S&T effort when it transitions to a program of record, approximately in FY DASD(DT&E) Assessment of the Army Report Based upon the report submitted by the Army and subsequent discussions, the DASD(DT&E) assesses that the Army has adequate T&E organizations and capabilities to support the Army T&E mission. The Army did note that the fiscally constrained environment is beginning to adversely affect test resources. DASD(DT&E) will continue to monitor the effect of the FY 2017 budget on test resources. As in the FY 2015 report, the Army has identified CDTs for MDAPs and MAIS programs with some vacancies. The Army has coded certain positions as KLPs and has aligned all of the KLPs to the programs in the PEOs. Overall, most of the CDTs have the required level of certification and are expected to meet the requirements described in the USD(AT&L) KLP memorandum. DASD(DT&E) recommends that the Army continue its efforts to identify T&E KLPs to achieve the goal of having a CDT assigned to each of its MDAPs and MAIS programs. The Army should consider the pool of qualified CDTs when filling future T&E KLPs. The Lead DT&E Organizations for the Army MDAPs and MAIS programs are all within ATEC AEC, whose mission includes DT&E and OT&E activities. DASD(DT&E) continues to monitor the DT&E capabilities needed by AEC to perform the activities of a Lead DT&E Organization DON Report Summary The DON report indicates that the DON T&E workforce and test support to programs were effectively managed in 2016 across the DON T&E enterprise, and the DON workforce is assessed to be adequately structured and trained to support the needs of DON acquisition programs. The DON assessed its T&E workforce, facilities, processes, and practices as adequate to support DT&E activities for acquisition programs. Naval SYSCOMs, PEOs, PMOs, and Naval W/SCs utilized a Competency Aligned Organization and Integrated Product Team (IPT) business model in the area of T&E. SYSCOM commanders structure and staff their organizations to meet workload demands and provide required T&E technical expertise and infrastructure capability. The Deputy DON T&E Executive; Deputy Assistant Secretary of the Navy for Research, Development, Test, and 26 DASD(DT&E) FY 2016 Annual Report

33 DoD Component Assessments Evaluation (DASN(RDT&E))/Office of the Chief of Naval Operations (OPNAV) N94C; and staff provided oversight, policy direction and process information, and monitored activities. Highlights of 2016 are as follows: The Naval Sea Systems Command (NAVSEA) continued to execute the Engineering and Test and Evaluation (ET&E) Competency, with its Competency Domain Manager (CDM) and T&E Functional Advisory Board, to improve alignment of SYSCOM T&E support activity for affiliated PEOs and PMOs. Emphasis has been on early T&E, interoperability and integration (I&I) engineering, mission-based testing, and cybersecurity T&E. The Naval Air Systems Command (NAVAIR) continued to increase development of test capabilities in operationally relevant live, virtual, and constructive (LVC) test environments; cybersecurity T&E; integrated warfare; capabilities-based test and evaluation (CBTE); and autonomous systems T&E. NAVAIR expanded its College of Test and Evaluation (CT&E) curriculum and throughput to support workforce development for these initiatives. The Marine Corps Systems Command (MCSC) DT&E Division completed its first year beyond full operational capability with the focus shifting to building and sustainment of critical specialty skills for the DT&E Competency. The T&E-coded workforce is now fully embedded within MCSC, PEO Land Systems, the Marine Corps Tactical Systems Support Activity (MCTSSA), and the Amphibious Vehicle Test Branch. The growth of the DT&E Division and its impact on the Marine Corps acquisition community has been noticeable and measurable in terms of support activity. The Space and Naval Warfare Systems Command (SPAWAR) continued to provide significant support to cybersecurity DT&E workforce development efforts defining key roles, tasks, duties, and responsibilities. SPAWAR, as the lead cyber technical warrant in the DON, defined the knowledge, skills, and abilities needed for the cybersecurity T&E workforce in response to DoDI , Risk Management Framework (RMF) for DoD Information Technology (IT), and the DoD Cybersecurity T&E Guidebook. The DON T&E Enterprise Improvement Process (TEIP) remains in use for strategic planning and management of DON T&E enterprise efforts. The TEIP has the following thrust areas: workforce, infrastructure/capability, and policy/acquisition program support. The Deputy DON T&E Executive serves as the national lead for the T&E career field and is the designated TEIP lead. The TEIP lead is supported by a planning team from the DON T&E office (composed of DASN(RDT&E) and OPNAV N942/N943 senior T&E personnel). In September 2016, OPNAV reorganized and N84 was recoded to N94 with no changes in roles and responsibilities for T&E support to acquisition programs and oversight of Navy T&E infrastructure. In 2016, the DON T&E Workforce Competency IPT was again involved with the following continuous improvement initiatives: Use of metrics to track and monitor T&E KLP qualifications, T&E certifications, and continuous learning (CL) compliance for the DON T&E workforce. Update and continued fielding of the DON T&E Training Course, Strategies for Effective and Efficient T&E, which has now reached more than 850 T&E personnel across SYSCOMs/PEOs and W/SCs. DASD(DT&E) FY 2016 Annual Report 27

34 DoD Component Assessments Cybersecurity T&E process training to SYSCOMs and W/SCs in coordination with DOT&E and DASD(DT&E) action officers. The DON T&E Awards Program, which is in its fourth year, to award and honor individuals and groups for their outstanding T&E achievements. In 2016, DON T&E personnel continued to support early acquisition activities involving contracting, requirements development, source selection activities, and translation of operational capabilities into contract specifications. The DON T&E office, and in turn SYSCOM T&E offices, continued to promote early T&E involvement and best practices. In 2016, DON T&E personnel support to program planning and strategic execution continued to be effectively managed by SYSCOMs/PEOs/PMs. In the planning and strategic execution phases, the PM has the responsibility for resourcing T&E expertise. The DON directs the use of T&E Working Integrated Product Teams (WIPTs) for planning and executing program testing. PMs are accountable for assigning a program T&E lead (e.g., CDT for an ACAT I/IA program) to chair the T&E WIPT. The DON T&E office provides guidance and qualification requirements for T&E KLPs and monitors the filling of these critical positions. Within the DON, the PM and program T&E staff are accountable for planning and executing DT&E and the personnel providing test execution, data analysis, evaluation, and reporting activities. SYSCOMs, W/SCs and, in some cases, dedicated non-don test commands (e.g., ATEC, DISA) support the PMs in planning, executing, and reporting results of DT events. As a result, management of T&E personnel in support of program execution is conducted at the local SYSCOM/PEO/PM level. As part of the effort to improve workforce recruitment, development, retention, and awards, DON T&E continually assesses and addresses workforce policies and strategies. SYSCOMs perform periodic capabilities health assessments that address the supply, demand, and knowledge stewardship of the T&E workforce. Gaps identified through this assessment are used to develop recruiting, hiring, and training strategies to mitigate impacts. SYSCOMs employ numerous metrics to tailor the overall size and composition of the T&E workforce, including hiring forecasting and allocations, workforce demographics, workforce targets, and outsourcing levels. The Deputy DON T&E Executive serves as the national lead for the T&E career field for the Naval Acquisition Career Field Council. The role of the national lead is to serve as the senior subject matter expert (SME) and advise the Assistant Secretary of the Navy for Research, Development, and Acquisition on T&E career field health, career path policies, development, and overall guidance pertaining to the DON T&E acquisition workforce. For the DON T&E national lead, DASN(RDT&E) and N942 senior T&E staff participate in the Office of the Secretary of Defense (OSD) T&E FIPT and provide working-level support for T&E workforce initiatives. The DON T&E Workforce Competency IPT is used as a vehicle to provide SYSCOM T&E offices with awareness of T&E FIPT initiatives and as a means to obtain inputs from the field. The Deputy DON T&E Executive and staff provide policy guidance and oversight of SYSCOMs and affiliated PEOs/PMs to ensure compliance with T&E KLP/CDT requirements for ACAT I and IA programs. DON T&E continues to have a very high fill rate for program T&E KLP, or CDT, 28 DASD(DT&E) FY 2016 Annual Report

35 DoD Component Assessments positions with assignment of program-dedicated T&E Level III certified personnel with the necessary education, experience, T&E competencies, and leadership skills. In 2016, NAVAIR continued to conduct CBTE and improve interoperability with the development of LVC architectures and cyber-resilient integrated warfighting capabilities. NAVAIR has continued to develop the frameworks, tools, scenarios, and environments to provide robust LVC capabilities. NAVSEA continued to exercise its T&E Competency under the ET&E Directorate using the NAVSEA 05 T&E CDM as a focal point. The T&E CDM s role is to provide greater visibility on the role and responsibilities of NAVSEA T&E to programmatic and technical authority decision making. MCSC s T&E-coded workforce is now stable and has maintained a high percentage of T&E Level III and CL compliance. In the DT&E FY 2015 Annual Report, DASD(DT&E) recommended that the DON continue its outstanding efforts to achieve the goal of having a CDT assigned to each of its MDAPs and MAIS programs. In FY 2016, the DON stated its continued commitment to provide highly qualified and experienced personnel to fill T&E KLPs and provide CDTs for its MDAPs and MAIS programs. However, there are circumstances that prevent the goal of having a single CDT assigned to each and every program. Oversight and monitoring of T&E KLP assignments are provided by the DON T&E office to ensure that policy and program support requirements are being met. The DON T&E office conducts talent management studies to address T&E coding, certification, CL compliance, and incumbent T&E KLP/CDT qualifications fulfillment. For the many DON ACAT I and IA programs, more than 90 percent have T&E KLPs/CDTs assigned. For those that do not, some pre-mdaps have a T&E KLP vacancy that is in the process of being filled. Other programs, that are very early or late in their acquisition cycle, have a capable T&E lead assigned, if the billet has not been established as a T&E KLP/CDT position. For the MDAPs/MAIS programs that have a T&E KLP assigned, there is a small number of CDTs who support two programs. The moderate level of T&E activity for each of those programs makes this type of assignment feasible. Cybersecurity T&E NAVAIR CT&E has responded to the demand for cyber training by adding courses specializing in cybersecurity T&E and has aggressively pursued developing a workforce with knowledge and skills for supporting cyber T&E. NAVAIR has embraced a philosophy of learning by doing and is incorporating knowledge gained from these efforts to grow and mature the cyber T&E process for aircraft and weapon system programs. SPAWAR personnel completed the DoD CIO Cyberspace Workforce Framework development Phase 1 assessments of duties, knowledge, skills, and abilities based upon National Initiative for Cybersecurity Education standards. SPAWAR supported additional programs to refine the role of cybersecurity-related functions in the T&E community. SPAWAR continued to lead the transition for the Navy s Task Force Cyber Awakening and its transition into the CYBERSAFE Program. SPAWAR provides highly credentialed and experienced testers to support advanced penetration testing against unclassified and classified DON and DoD systems in development and production environments. DASD(DT&E) FY 2016 Annual Report 29

36 DoD Component Assessments In NAVSEA, cybersecurity is the fourth pillar of the command s Strategic Business Plan, and the purview of Navy control systems is assigned to the ET&E Competency. NAVSEA continues to focus on its top priorities for creating a cybersecurity culture across the Navy, establishing cybersecurity education, and identifying cybersecurity best practices. MCSC s DT&E Division along with MCTSSA and the Marine Corps Operational Test and Evaluation Activity achieved significant progress in the area of cyber DT&E. Command leadership support in this expansion of knowledge, skills, abilities, and capabilities is a high priority. MCTSSA is formally establishing a cyber DT&E capability and solidifying its position within the Marine Corps acquisition community as the Lead DT&E Organization for command, control, communications, computers, and intelligence (C4I); intelligence; and all things cyber. Interoperability T&E For NAVSEA, the Distributed Integration and Interoperability Assessment Capability (DIIAC) leverages existing facilities, skilled resources, live assets, and network connectivity and the Automated Test and Analysis/Automated Test and Retest applications to test and assess battle force interoperability. DIIAC is being merged with the PEO Integrated Warfare Systems and its multi-site test process to enhance I&I of systems. This merge facilitates testing Strike Force Interoperability over a common test architecture using the Secure Defense Research and Engineering Network and results in a combined interoperability T&E team that will support near-, mid-, and long-term Integrated Surface Warfare System interoperability requirements. NAVAIR utilized these capabilities to conduct distributed environment LVC testing to allow SoS to be linked together to perform end-to-end interoperability and mission-context testing of the systems under test. This effort has led to early identification of deficiencies and provided a method for risk reduction for programs. Reliability T&E The DON reliability and maintainability (R&M) engineering lead in the DASN(RDT&E) SE office continues to highlight the importance of R&M in engineering and the supporting role of T&E support for MDAPs and MAIS programs. The DON R&M engineering lead continues to support development and operation of SYSCOM R&M offices, and in turn supports program T&E leads and CDTs on major programs. NAVSEA has an R&M engineering manager assigned for the SEA05S Technical Warrant. The R&M engineering manager is responsible for related policy planning, guidance, standards, and training development. The R&M engineering manager reviews all products that go to the Chief Engineer for review and approval such as acquisition strategies and Systems Engineering Plans. NAVAIR continued to ensure that naval aviation systems are reliable and maintainable through the application of proven R&M engineering and T&E processes. In 2016, progress continued on the Reliability Growth Development and Management Standard Work Package. The R&M Engineering Competency worked closely with the T&E community in regard to R&M planning and assessment. NAVAIR continued to provide R&M instruction, focusing on reliability data collection and growth, and to train CDTs in the Assistant Program Manager T&E Boot Camp. 30 DASD(DT&E) FY 2016 Annual Report

37 DoD Component Assessments T&E Capabilities The DON has robust T&E facilities, ranges, target capabilities, and threat simulators for testing platform-centric and SoS capabilities. As threats emerge or new test capabilities are needed because of system technological advances, they are addressed using the Navy Major Range and Test Facility Base (MRTFB) and Navy Working Capital Fund test ranges and facilities capabilities. With the development and installation of laser weapon systems on Navy platforms, NAVSEA is pursuing the advancement of engineering and T&E career field skills to ensure that high-energy laser (HEL) test capabilities exist to support higher powered laser systems. NAVSEA is investing in additional T&E infrastructure to support the advancement of HEL capabilities. NAVSEA, SPAWAR, and NAVAIR continued working with DoD cyber ranges and accomplished two major watershed events in FY USS Secure Pilot Test I and Serial Test II were completed in March and August 2016, respectively. These events provided the DON s first-ever multi- SYSCOM test bed with systems from Tier 0 Tier 3 and multiple enclaves representing cyber key terrain below the C4I enclave by interconnecting and gaining accreditation for 13 different labs from Hawaii to Lakehurst, New Jersey. NAVSEA successfully initiated the process of collaboratively evaluating undersea warfare (USW) test range capabilities, a process to be extended to focus on additional capabilities in the future. R&D, T&E, and training range capability needs in the USW domain are very similar, which presents an opportunity to do more for less with sufficient enterprise collaboration. NAVAIR and the Naval Integrated Fire Control Counter Air (NIFC-CA) program effort is connecting the pillar platform system integration labs, other LVC systems, and aircraft to demonstrate and accomplish early investigations of SoS interoperability and functionality testing. The NIFC-CA effort is also conducting combined data analysis reviews across pillar test teams and collaborating on the best deficiency solutions to benefit the overall SoS performance. SPAWAR expanded the Interactive Test Procedure (ITP) program to use on the Large Deck New Construction Radio Room program. The ITP application is an unclassified, web-based software application executed in a common access card (CAC)-enabled secure environment. ITP transitioned the test procedure life cycle from a paper-based format, requiring a high degree of manual workflow interaction and validation, to an electronic format that uses the power of database management to streamline the entire workflow and life cycle of a test program from test development and maintenance through test execution and data validation. T&E of Autonomous Systems The demand signal for T&E of autonomous air vehicles and functions has been building and is expected to continue ramping up rapidly in the near future. Much of the DON activity relating to autonomous systems is in R&D and S&T demonstration efforts, but it does include some programs of record with extensive T&E activity. T&E-supported activity at SYSCOMs and W/SCs also includes science, technology, engineering, and mathematics (STEM) outreach and T&E workforce education efforts. Examples include the following: Each of the Naval SYSCOMs participates on the Autonomy Test, Evaluation, Verification, and Validation (TEVV) Working Group, sponsored by the DoD Autonomy Community of Interest. DASD(DT&E) FY 2016 Annual Report 31

38 DoD Component Assessments The group coordinates and reports on needed TEVV standards, best practices, and resources (including technical competencies and test ranges) required to enable future autonomous and self-governing defense systems. The Autonomous Aerial Cargo/Utility System is an Office of Naval Research Innovative Naval Prototype program exploring advanced autonomous capabilities for reliable resupply and casualty evacuation by an unmanned air vehicle under adverse conditions. NAVAIR T&E and the U.S. Naval Test Pilot School have been involved in reviewing test plans and test strategies, monitoring quarterly performance reviews, and capturing methods to successfully test autonomy. NAVAIR sponsors the STEM4UAS high school competition in collaboration with the Association for Unmanned Vehicle Systems International Seafarer Chapter. NAVAIR is in its second year of an element of a University of Maryland partnership to provide master s-level courses in robotics available to anyone in DoD via distance learning. The Unmanned Vehicles and Autonomous Systems Working Group, composed of members from each Warfare Center Division, continues to exploit collective technical capabilities to rapidly field the most cost-effective, reliable, and Warfighter-relevant unmanned systems possible. Numerous Warfare Centers are participating in the TRMC project involving the Unmanned and Autonomous Systems Testing (UAST) focus area DASD(DT&E) Assessment of the DON Report Based upon the report submitted by the DON and subsequent discussions, the DASD(DT&E) assesses that the DON has adequate T&E organizations and capabilities to support the DON T&E mission. The DON has identified CDTs for MDAPs and MAIS programs with only a few vacancies. The DON identified which positions are coded T&E and the level of T&E certification. The few vacant positions have personnel under consideration for fill. Overall, positions are filled with personnel expected to be qualified and meet the requirements described in the USD(AT&L) KLP memorandum. The DON continued its commitment to code and fill CDTs (T&E KLPs) with highly qualified and experienced personnel. DASD(DT&E) commends the DON on its efforts to fill CDTs (T&E KLPs) and recommends that the DON continue its outstanding efforts to achieve the goal of having a CDT assigned to each of its MDAPs and MAIS programs. The DON should consider the pool of qualified CDTs when filling future T&E KLPs Air Force Report Summary The Air Force report indicates that overall, the Air Force T&E enterprise is adapting to support the needs and requirements of Air Force acquisition programs. The Air Force continues to rise to the challenges presented by budget and personnel reductions as well as the uncertainties of repeated continuing resolutions. Through refinement of its internal processes and organizations, the Air Force T&E enterprise is working to minimize the impacts of these recent challenges and execute T&E activities, recapitalize critical infrastructure, and build and develop the T&E workforce. 32 DASD(DT&E) FY 2016 Annual Report

39 DoD Component Assessments The Air Force report describes the activities, processes, changes, and initiatives that the Air Force has implemented or will implement to ensure greater efficiency and effectiveness in the Air Force T&E enterprise. The Air Force report describes ongoing Air Force efforts to implement congressional direction to designate CDTs and Lead DT&E Organizations for MDAPs. The report also covers the implementation of KLP qualifications for CDTs. The Air Force Directorate of Test and Evaluation (AF/TE) continues its active participation in OSDled working groups such as the TEWG, T&E FIPT, and DAG rewrite team. AF/TE made significant contributions to DAU by helping to write new courses on integrated testing and by teaching in localarea T&E sessions. Additionally, AF/TE participated in revamping the DAWIA T&E certification requirements by updating the list of needed experience and coursework. Similar to its FY 2015 report, the Air Force expressed several concerns, including the following that are directly related to the T&E workforce: Maturation of cyber testing efforts is a focus of the Air Force T&E enterprise. Key aspects of cyber testing include properly educated and trained personnel, adequate resources, necessary infrastructure, and appropriate guidance at Air Force and DoD levels. o Personnel who meet the requirements necessary for cyber testing are in high demand, and recruiting and retaining such personnel remain a significant challenge. o More fully developed guidance by DoD and the Air Force will be required to ensure that cyber capabilities are sufficiently tested, evaluated, and assessed. o Resources, infrastructure, and close cooperation will be required to create and implement an adversarial testing capability that demonstrates resiliency and survivability of cyber systems and provides confidence to Warfighters. OSD oversight lists have been separated into three lists (USD(AT&L) list, DOT&E list, DASD(DT&E) list) that are not aligned with each other. This situation has caused some confusion to AF/TE oversight responsibilities. The Air Force asks that DoD reconsider reducing the multiple oversight lists to help limit confusion across the Services and programs. Although the Air Force continues to support the Joint Qualification Board process for identification of qualified CDTs, the Air Force in general is having problems finding enough qualified individuals to fill CDT positions for all programs. AF/TE will continue to work with the programs to help identify viable paths forward for compliance with this requirement and will continue to encourage personnel to apply to the T&E KLP Joint Qualification Board. Several continuing resolutions have caused increased uncertainty in RDT&E resource budgets and disruptions that have required significant rephasing of program funds. In FY 2016, Air Force T&E personnel leveraged the training courses available via the Air Force Institute of Technology and DAU to apply STAT and design of experiments to various test programs. AF/TE continues to raise awareness of these resources through informational memorandums to the centers and direct contact with Air Force test professionals at Air Force T&E policy summits. AF/TE continues to incorporate changes from the latest DoDI into a new release of Air DASD(DT&E) FY 2016 Annual Report 33

40 DoD Component Assessments Force Instruction (AFI) , Capabilities-Based Test and Evaluation, scheduled for release in early To support planning and strategic execution, Air Force T&E professionals have a robust test design process based on early test involvement and integrated test principles. AF/TE has noted an increase in clear and complete requirements documents; integrated acquisition strategies; up-front design of integrated testing in program strategies; and earlier identification of problems, both systemic and programmatic, allowing the problems to be addressed sooner. Through implementation of the Integrated Test Team (ITT) construct, acquisition programs are openly and regularly reviewed by the DT&E and OT&E communities. Although AF/TE has noticed an overall increase in clear and complete requirements documents, the rapidly evolving cyber requirements process has been challenging. No major changes occurred for test execution, data analysis, evaluation, and reporting activities in FY T&E execution is aided by ensuring that all T&E assets (e.g., funding, personnel, test ranges and facilities, test articles) are available at the proper time and place as the test program unfolds. Air Force testers, both DT&E and OT&E, are integrating their test efforts. Integrated testing is the best strategy for increasing test efficiency and effectiveness. The Air Force process for attracting, retaining, and rewarding T&E personnel remains largely unchanged. The Air Force has no defined T&E career field for the civilian or military workforce. Members must be assigned to test organizations to gain experience by performing test activities and to obtain DAWIA acquisition certification training. In FY 2013, the Air Force completed a review of all T&E-coded positions to ensure that all coded positions met established grade guidelines. Only 4 percent of the positions showed discrepancies with proper coding. These discrepancies were addressed and corrective actions taken. The Air Force considers the balance of Level I, II, and III positions within the T&E enterprise to match the duty responsibilities required. Although the DASD(DT&E) is concerned about the number of Level III T&E-coded positions within the Air Force and that the limited number could impact the ability to prepare CDTs for Air Force MDAPs and MAIS programs, at this time, the Air Force does not believe any further changes are required. The Air Force will continue to review and ensure that T&E positions are properly coded and at an appropriate certification level. DASD(DT&E) has recommended that the Air Force increase the number of CDT-coded positions to achieve the goal of having a CDT assigned to each of its MDAPs and MAIS programs. The Air Force implemented AFI , Management of Acquisition Key Leadership Positions (KLP), signed on October 19, 2015, which designates all CDT positions as KLPs within program structures. The Air Force is actively moving toward coding positions appropriately based on AFI and will continue to do so in FY 2017 and beyond. Although coding the positions appropriately is important, as stated earlier, the Air Force has few individuals qualified to meet the high standards defined for a CDT, and those who are qualified are provided an incentive to transition to program management rather than continue in T&E. Despite these qualification issues, the Air Force currently makes up 42 percent (25 out of 59) of T&E KLP Joint Qualification Board-qualified CDTs across DoD. 34 DASD(DT&E) FY 2016 Annual Report

41 DoD Component Assessments Cybersecurity T&E The Air Force Life Cycle Management Center (AFLCMC) Center Test Authority (AFLCMC/AZT (CTA)) has been actively involved in the Air Force Materiel Command cyber T&E summits at Wright-Patterson Air Force Base (AFB), Ohio, on a regular basis. The CTA is actively involved to help facilitate the exchange of information between policy makers and program offices. Information exchanged deals with but is not limited to current cyber T&E capabilities (DT/operational testing (OT) for traditional information technology (IT)/network systems, critical infrastructure, and weapons systems) to determine roles and responsibilities of different organizations, agree on a desired end state for future capabilities, and determine remaining gaps to achieve the desired Air Force end state. At the Air Force Test Center (AFTC), corresponding with an increase in aircraft and weapon system cyber T&E efforts, existing avionics (775th Test Squadron) and electronic warfare (771st Test Squadron) engineers at Edwards AFB, California, were detailed into supporting cyber test activities. These opportunities provided cyber test exposure and broadening to 10 experienced DT engineers. The 96th Test Wing and 412th Test Wing initiated a collaborative effort to establish a cyber T&E capability at Edwards AFB modeled after the Avionics Vulnerability Mitigation and Protection Lab at Wright-Patterson AFB. Approximately 6,000 square feet of previous F-22 laboratory space was allocated to support future cyber test, training, and tool development efforts while constituting the home location for the 46th Test Squadron s Edwards AFB operating location for aircraft and weapons cyber T&E. The Space and Missile Systems Center (SMC) Engineering Directorate (SMC/EN) is standing up a core working group, bringing together expertise among the various divisions in the SMC/EN directorate and external support organizations and directorates to specifically address cybersecurity T&E. Interoperability T&E AFLCMC/AZT (CTA) has established a center process guide for Test Early Strategy and Issues Sessions (T-ESIS) to assist program offices in developing early test strategy. The desired overall outcome of the T-ESIS is to develop an overarching program test concept/strategy that determines the program s ITT membership, T&E test types needed, and the selection of ITT subgroups. During this process, the CTA ensures that JITC is involved as a stakeholder with the program in standing up the ITT. Interoperability requirements from known/expected Net Ready KPPs or key system attributes (KSAs) are identified up front as part of early test strategy development. At the AFTC, the 775th Test Squadron is continuing an ongoing initiative to use Link 16 and variable message format data collected during interoperability events and from potential interoperable platforms to improve the fidelity of the simulated traffic used during developmental ground and flight test. The Distributed Ground Station-2, operated by the 548th Intelligence Group at Beale AFB, California, is processing, exploiting, disseminating, and serving as an end-to-end interoperability partner. SMC ensures that programs are interoperable through the involvement of JITC. DASD(DT&E) FY 2016 Annual Report 35

42 DoD Component Assessments Reliability T&E AFLCMC/AZT (CTA) helps disseminate best practice TEMP documentation for reliability, availability, and maintainability (RAM) testing, processes, or techniques and valid RGC formats. AFLCMC/EZ has established a center handbook process guide for reliability-centered maintenance to assist program offices in achieving inherent reliability and restore deteriorated reliability for DoD materiel, as well as to ensure that safety and performance characteristics are maintained. AFTC is assisting in development of two initiatives to update the Reliability, Availability, Maintainability, and Cost Rationale Report Manual and the DAU courseware for reliability engineering. The SMC Standard SMC-S-013, Reliability Program for Space Systems, is the standard used by SMC and its programs for ensuring that adequate reliability is built into the development program of the space system and is part of the acquisition process. This standard stresses design for reliability, which dramatically increases the chances that the system will meet its operational reliability requirements if reliability is addressed before the system design is finalized. T&E Capabilities AFLCMC/AZT (CTA) collects, disseminates, and posts listings of qualified Lead DT&E Organizations to support program needs for DT&E execution. The CTA also posts links to the Air Force T&E Organizations and Facilities Database for program planning and references to other Service resources and also provides references to capabilities available in international test facilities and ranges. The CTA provides capability resources to encourage programs to consider/use test capabilities of Air Force, MRTFB, and other Service Government test facilities. The AFTC categorizes T&E capabilities into seven primary mission areas and manages the health of those mission areas through Mission Area Panels (MAPs). The MAPs assist and advise AFTC leadership to enable them to effectively position the AFTC enterprise to support the test needs of current and future customers through implementation of Improvement and Modernization (I&M) programs. SMC/EN has a wide variety of T&E expertise in its directorate in addition to 17 specific specialty engineering disciplines, each with additional capability and T&E knowledge. In terms of SMC T&E execution, SMC industry partners and contractors operate anechoic chambers for electromagnetic interference/electromagnetic compatibility testing, acoustic and vibration tables for random vibration/launch survivability testing, and thermal vacuum chambers for space environmental effects. Additionally, the Aerospace Corporation (a Federally Funded Research and Development Center (FFRDC)) performs R&D into cutting-edge technologies and capabilities. T&E of Autonomous Systems AFLCMC/AZT (CTA) continues to provide test management support to AFLCMC/WI remotely piloted aircraft programs/systems especially in the integration testing of new sensors and other capabilities. The AFTC s 412th Operations Group stood up the new Emerging Technologies Combined Test Force (ET CTF) with a mission of providing agile, innovative flight test capabilities for emerging 36 DASD(DT&E) FY 2016 Annual Report

43 DoD Component Assessments technologies. The primary focus of the ET CTF at this time is in the testing of small unmanned aircraft systems and autonomy. The 412th Test Wing has supported the OSD Autonomy TEVV Working Group and the TRMC UAST Working Group. These efforts will inform requirements and needs for future infrastructure, capabilities, and methodologies for the testing of autonomy DASD(DT&E) Assessment of the Air Force Report Based upon the report submitted by the Air Force, the DASD(DT&E) assesses that the Air Force has adequate T&E organizations and capabilities to support the Air Force T&E mission. The Air Force has identified personnel assigned as the CDT for its MDAPs and MAIS programs with some vacancies. As stated in previous DT&E annual reports, DASD(DT&E) remains concerned that not all the CDT positions have been coded as T&E KLPs. The Air Force is lagging behind the other DoD Components when coding positions as KLPs (see Tables 3-3 and 3-4). In addition, several positions reported were noted to be coded in a career field other than T&E (i.e., engineering and program management), and assigned personnel do not have adequate T&E certification (Level III required). The Air Force has indicated in discussions that it is working to code all T&E KLP positions. DASD(DT&E) recommends that the Air Force take action in FY 2017 to increase the number of CDT-coded positions to achieve the goal of having a CDT assigned to each of its MDAPs and MAIS programs. This coding will be a great step forward to cover all of the Air Force MDAPs and MAIS programs. The Air Force has made an effort to encourage its personnel to apply to the T&E KLP Joint Qualification Board. DASD(DT&E) commends the Air Force on its efforts to increase the pool of highly qualified individuals. DASD(DT&E) remains concerned that the limited number of Level III personnel in Air Force T&E could impact the Air Force s ability to prepare future CDTs for Air Force MDAPs and MAIS programs. DASD(DT&E) recommends that the Air Force take actions to increase the number of Level III T&E-coded positions to enable an increase in the number of Level III T&E certified personnel, which in turn would increase the number of T&E personnel eligible to apply to the T&E KLP Joint Qualification Board. The Air Force continues to have a concern about multiple OSD oversight lists. In accordance with DoDI , DASD(DT&E) engages with MDAPs, MAIS programs, and USD(AT&L)-designated special interest programs, which make up the USD(AT&L) list DISA Report Summary The DISA report indicates that the overall state of personnel to conduct DT&E within DISA is adequate to support the DISA mission. DISA currently has an appropriately sized workforce and the personnel are sufficiently trained. DISA continues to support initiatives to improve T&E efficiencies, while increasing overall effectiveness. DISA is building stronger IT compliance testing programs that will correct problems earlier and reduce the cost and schedule impacts of legacy approaches. DISA continued its efforts to DASD(DT&E) FY 2016 Annual Report 37

44 DoD Component Assessments evolve its T&E laboratory environment, ensuring that test tools, reference implementations, and test infrastructure are in place to support rigorous T&E of programs, applications, and services. DISA is composed of nearly 6,000 civilian employees; more than 1,500 active duty military personnel from the Army, Air Force, Navy, and Marine Corps; and approximately 7,500 defense contractors. DISA provides, operates, and ensures command and control (C2) and informationsharing capabilities and a globally accessible enterprise information infrastructure in direct support of joint Warfighters, national-level leaders, and other mission and coalition partners across the full spectrum of military operations. The DISA T&E workforce is composed of engineers, computer scientists, IT specialists, and operations research professionals. For the DT&E workforce, the personnel are primarily spread across DISA PMOs with a portion of them assigned under JITC, which is aligned organizationally within the DISA Development and Business Center and is the Joint Interoperability Certifier of DoD IT/national security systems (NSS). JITC is the only non-service Operational Test Agency and performs OT&E execution and cybersecurity assessments for DISA and other external customers. JITC provides DT&E services to DISA programs when required; however, many DISA programs have test managers within their PMOs who are responsible for testing. DISA MAIS programs establish dedicated DT&E teams, allowing JITC to focus on its role as the independent test agent for interoperability evaluations, OT&E, and cybersecurity assessments. Within DISA, a broad range of IT exists that T&E must support. Cloud services, whether Government or public, are being integrated onto a single converged IT infrastructure. This integration necessitates more stringent cybersecurity and interoperability evaluations. Likewise, mobility and enterprise unified capabilities (UC) are adding new services/capabilities to the distant end user, and cyber defense initiatives have resulted in a new DoD cyber C2 framework. These new concepts have required DISA T&E to evolve its methods for conducting T&E without increasing resources or time to test. DISA T&E continued its efforts to evolve the DoD information network (DoDIN) laboratory environment, ensuring that test tools, reference implementations, and test infrastructure are in place to support rigorous T&E of programs, applications, and services. Efforts included standing up on-premise and off-premise Infrastructure as a Service capabilities, implementing lab automation, and employing virtualization through software-defined networking. DISA T&E is evolving this environment to serve as a federated infrastructure that aids development and ensures that DISA rigorously tests, evaluates, and certifies enterprise solutions before they are fielded. The JITC Commander serves as the T&E Executive for DISA providing guidance and comments on all matters within DISA that require a T&E review. In support of early acquisition phases, JITC retains the strategic T&E expertise and resources to help with any DISA RFPs and source selection activities. As part of DISA acquisition procedures, any program that is in the process of defining requirements or developing acquisition artifacts, measures, or parameters must coordinate its documentation through JITC. DISA uses IPTs in support of T&E for programs, leveraging the joint testing and user communities to help develop and implement its testing artifacts. Most DISA acquisition programs use this forum to address T&E data analysis, problem solving, test planning, test execution, and reporting. 38 DASD(DT&E) FY 2016 Annual Report

45 DoD Component Assessments DISA PMOs are structured to effectively support DT&E efforts surrounding program test activities. To facilitate these efforts, each PMO has a lead test integrator whose primary responsibility is to ensure that the established testing processes are executed as planned. The PMO test lead serves as a single point of contact with the background, knowledge, and authority to direct all testing activities. PMOs for MAIS programs also generally provide a test staff to perform independent evaluations and review and provide technical comments on all software documentation. PMOs often consult with JITC personnel to execute tests for smaller projects that do not have sufficient test expertise or to be enlisted as the test manager for all DT efforts. In FY 2016, JITC developed an Early Test Involvement (ETI) program that enables JITC to be a part of the DISA requirements analysis process before a decision is made to move forward with an acquisition or development effort. The outcomes of the ETI process are an initial T&E risk assessment and a T&E management approach that allow DISA to scope proper levels of T&E early. In addition, JITC instituted a T&E scorecard approach as a means for reporting on DISA project status from an OT&E perspective. JITC developed a T&E Scorecard Guidebook that provides detailed descriptions of areas of evaluation (AoEs) with associated metrics and measures that are used to evaluate project capabilities. In FY 2016, JITC began hosting workshops and developed training modules to further define these AoEs. The T&E Scorecard Guidebook also assists DISA PMOs to better scope T&E, develop the evaluation approach, and provide a standard means to report testing status throughout the project life cycle. Cybersecurity T&E JITC cybersecurity service areas include RMF assessments, vulnerability and risk assessments, intrusion detection systems and security tools assessment, cybersecurity OT&E support, and major combatant command exercise assessments. JITC assists Government and non-government organizations with their security certification and accreditation-related initiatives for full compliance with legislative requirements and established guidelines. In addition, the JITC cybersecurity technical framework provides assessment support inclusive of Defense in Depth and DoDIN operations. Developmental and laboratory testing of joint regional security stacks (JRSS) and Joint Information Environment cyber capabilities continued in FY 2016 at the DISA Fort Meade, Maryland, location. In July 2016, JITC completed a successful JRSS computer network defense lab-based exercise environment to determine the effectiveness of JRSS version 1.0. In FY 2016, JITC measured the Joint Force Headquarters (JFHQ) DoDIN mission thread effectiveness in a live operational environment to identify performance trends and characterize the JFHQ-DoDIN capabilities and limitations to successfully execute mission-essential functions and mission-essential tasks. The JITC Cybersecurity Assessment Team worked closely with personnel from the U.S. Strategic Command, U.S. Transportation Command, and U.S. Cyber Command to develop an assessment plan that meets DOT&E requirements and to give the combatant commander a measure of operational risk posed by the cybersecurity posture of the systems and networks deployed in support of the combatant commander s assigned missions. In FY 2016, the JITC Cybersecurity Assessment Team provided assessments for four major Tier I exercises as well as planning for future exercise assessments. JITC also continued to DASD(DT&E) FY 2016 Annual Report 39

46 DoD Component Assessments conduct cybersecurity tests and assessments of commercial products for accreditation and entry on the UC Approved Products List. Interoperability T&E DISA increased test rigor and reduced test costs and time of interoperability evaluations for several programs by expanding the use of its automated interoperability data collection and analysis capability, the Joint Analysis Net-centric Evaluation Testing Toolkit (JANETT). DISA initiated an effort to provision JANETT as a service available via the Web so PMs can leverage the tool on their own earlier in the acquisition life cycle, which is reducing risk of interoperability issues during OT or while in operations. DISA expanded the use of its Automated Test Case Generator (ATCGen), an automated Link 16 standards conformance test tool, to increase test rigor and reduce PM test costs and schedules. JITC added automated test sets to ATCGen to address complex Mode 5 identification, friend or foe; ballistic missile defense (BMD); and net-enabled weapons system test requirements. JITC made the ATCGen available 24x7 as a Web service so PMs could test at their convenience without any interaction with JITC personnel. These enhancements in automation reduced the analysis time for a Link 16 standards conformance test from weeks to seconds and increased test rigor, which is reducing sortie and live-fire requirements during follow-on OT. DISA continued to enhance the technical expertise of its Government workforce for UC, tactical communications systems, C2 systems, and multiple platforms to reduce dependence on contractor labor and the time to get contracts in place for planning, executing, and reporting on testing efforts. This effort makes DISA testers much more responsive to PMs and vendors and is saving between 2 and 6 months start-up time depending on the complexity and number of customers involved in the effort. Reliability T&E DISA employs two basic strategies to evaluate reliability for hardware systems under test: direct measurement during test events and continuous monitoring before and after fielding. Direct measurement during test events normally provides insufficient data to achieve statistical confidence because of the relatively short duration of observation periods. However, for the DoD Teleport program, there were multiple test events and lengthy observation periods, so this strategy was acceptable. The continuous monitoring approach utilizes data reported operationally and collected over longer periods of time. This approach was employed for SECRET Internet Protocol Router Network (SIPRNET) token reliability on the Public Key Infrastructure (PKI) program. T&E Capabilities JITC is the major element of the DISA MRTFB, DoD s only non-service and IT-focused MRTFB. JITC global reach extends to the entire spectrum of DoD, Federal Government, private industry, and allies in support of C2, intelligence, and defense reform initiatives. As an MRTFB, JITC can engage directly with vendors to obtain critical pre-acquisition test results. JITC also supports non-dod organizations such as the National Geospatial-Intelligence Agency, National Security Agency, and Federal Emergency Management Agency. 40 DASD(DT&E) FY 2016 Annual Report

47 DoD Component Assessments JITC emulates IT/NSS operational architectures in its test facilities, ensuring that interoperability issues around the globe can be reconstructed and addressed remotely. JITC facilities are strategically located at Fort Huachuca, Arizona; Indian Head, Maryland; and Fort Meade, Maryland. The diverse capabilities of each respective location allow the Services to have access to a dynamic environment for laboratory tests and on-site field evaluations DASD(DT&E) Assessment of the DISA Report Based upon the report submitted by DISA and subsequent discussions, the DASD(DT&E) assesses that DISA has adequate T&E organizations and capabilities to support the DISA T&E mission. DISA has identified CDTs for its two MAIS programs (Global Combat Support System Joint and Teleport) and reported on the KLPs in its organization. The CDTs are occupying properly coded KLPs and have Level III certification. For MAIS programs, DISA PMOs are structured to effectively support DT&E efforts surrounding program test activities. DASD(DT&E) recommends that DISA consider the pool of qualified CDTs when filling future T&E KLPs MDA Report Summary MDA reported an optimum balance of T&E expertise in the workforce with the right skill sets, education levels, and technical experience to conduct a successful T&E program. During FY 2016, the MDA Director for Test maintained this balance through implementation of MDA Engineering and Support Services (MiDAESS) contracts and by taking full advantage of the Missile Defense Career Development Program (MDCDP). Vacated MDA civilian T&E positions were reengineered and realigned to address high-priority requirements. MDA has no MRTFB T&E personnel. The MDA T&E program functionally aligns a highly technical and qualified workforce composed of employees from multiple sensor, shooter, and C2 program offices and various support functions across MDA to execute an increasingly complex ground test, flight test, war games, and exercises program. The MDA Director for Test serves as the Test Functional Manager (TFM) to coordinate all activities within the Ballistic Missile Defense System (BMDS) Test Functional Area (TFA). The MDA T&E workforce consists of civilian and military acquisition-coded T&E personnel, other career field civilian personnel who support T&E activities, MiDAESS contractor support personnel, Joint National Integration Center Research and Development Contract personnel, and FFRDC personnel. All civilian, military, and contractor positions in the TFA are documented in the MDA manpower tool. The TFM determines Government manpower and support contractor requirements, approves all hiring action, executes a standardized Government civilian hiring process to recruit quality personnel in a timely manner, and ensures that support personnel matrixed to the TFA are meeting expectations. In FY 2016, MDA planned, programmed, budgeted, staffed, and managed a comprehensive BMDS test program to characterize BMD capabilities and support fielding of an integrated and DASD(DT&E) FY 2016 Annual Report 41

48 DoD Component Assessments effective capability to the Warfighter. MDA is sufficiently organized and resourced to support all phases of flight and ground testing as defined in the test policy, joint engineering and test Integrated Master Test Plan development instruction, flight test concept of operations (CONOPS), and ground test CONOPS. The MDA T&E workforce consists of civilian and military personnel assigned to acquisitioncoded T&E positions, civilian and military personnel from other career fields who support T&E activities, support contractors, prime contractors, FFRDC personnel, and University Affiliated Research Center (UARC) personnel. This combination of personnel provides an optimum balance of T&E workforce expertise and ensures that MDA has the right skill sets, education levels, and technical experience to plan and execute a successful T&E program. The MDA Directorate for Test and the TFA are adequately organized, funded, and staffed to support all phases of testing. Because of the complex nature of BMDS testing, MDA uses a phased process to integrate Directorate for Test, Directorate for Engineering, element program executives/directors/managers, and other organizational and functional managers to establish a single T&E workforce executing the full life cycle of any given test event. Although the Directorate for Test and Directorate for Engineering share responsibilities as the offices of primary responsibility across test phases, the result is a single, integrated MDA workforce that works together, independent of organizational boundaries, to successfully execute the BMDS test program. The current alignment of the test evaluation and analysis functions within the MDA Directorate for Test and Directorate for Engineering organizations meets Title 10, U.S.C., requirements. Within MDA, functional managers are responsible for defining DAWIA career field certification requirements for their respective workforce. The Director for Test, as the functional manager for the test workforce, establishes T&E certification requirements for core and matrix test positions. The Director for Engineering, as the engineering workforce functional manager, establishes engineering certification requirements for core and matrix engineering positions. The MDA Director is responsible for all BMDS testing; the Director for Engineering is responsible for defining BMDS test program technical objectives and analyzing test results; and the Director for Test is responsible for the planning, design, and execution of cost-effective testing to meet program objectives. MDA conducts these engineering and test activities in an integrated fashion, in support of BMDS requirements verification, performance assessment, and M&S verification, validation, and accreditation. The test program is then based on planning inputs from the Director for Engineering, including BMDS requirements and technical objectives, target requirements, and scenario definitions. The Director for Engineering draws on the expertise of the Director for Test to design test events and analyzes the test results to provide data for BMDS verification and assessment. Throughout the BMDS test process, the Director for Engineering, Director for Test, and program elements work together to ensure that individuals performing essential T&E and engineering functions are appropriately skilled, experienced, and coded for a particular position. In previous reports to Congress, DASD(DT&E) expressed concern regarding the proper coding of 29 MDA civilian (DAWIA: non-t&e) positions assigned to SE but performing T&E functions. These 29 positions are further broken down into seven DAWIA engineering-coded MDCDP positions that convert to T&E upon graduation and 22 DAWIA engineering-coded civilians. The engineering 42 DASD(DT&E) FY 2016 Annual Report

49 DoD Component Assessments DAWIA coding of the seven MDCDP positions is directly linked to MDA s funding source for the MDCDP. Therefore, the DASD(DT&E) concern is with the remaining 22 engineering-coded civilians. DASD(DT&E) and MDA have agreed to work together to resolve this concern. MDA will review the positions in the SE organization that are currently performing the analysis, assessment, evaluation, and reporting of test data/results and recode those positions to T&E if the majority of duties and responsibilities are T&E functions. DASD(DT&E) will investigate the possibility of coding positions in more than one career field. MDA has concerns about the potential consequences of dual coding. For example, a new hire would be required to certify in both engineering and T&E within a 2-year period a very difficult task. Similarly, backfilling vacant dual-coded positions may prove difficult because of the additional certification requirement. MDA will conduct a detailed review of the 22 positions in the SE organization that are currently performing the analysis, assessment, evaluation, and reporting of test data/results to determine whether the majority of duties and responsibilities are T&E or engineering functions. Once completed, MDA will recode positions if required. MDA will complete the detailed review no later than May Cybersecurity T&E MDA has a robust cybersecurity program to manage its RDT&E resources for all test venues. A trained and certified cybersecurity workforce is now assigned to manage, monitor, and audit the cybersecurity of Directorate for Test enclaves and systems. The Director for Test has worked aggressively in 2016 to achieve DoD CIO guidance, MDA CIO guidance, and Directorate for Test Information System Security Manager guidance. The MDA Director for Test has also staffed a Cybersecurity Test Team (CTT) that is beginning to plan and coordinate a cybersecurity developmental and operational test program for MDA. The CTT will support BMDS cybersecurity assessment by planning and coordinating cybersecurity experiments and cyber-focused activity as part of the existing ground test program to identify vulnerabilities and the potential impact on BMDS mission execution. MDA programs will leverage test results to develop and execute cybersecurity remediation plans. The CTT is responding to new systems requirements, operational feedback, and new legislation requirements for cybersecurity testing to document and execute a comprehensive BMDS cybersecurity test plan. Interoperability T&E MDA conducted several test events to demonstrate interoperability. MDA executed Flight Test Operational-02 Event 2a; Flight Test Operational-02 Event 1a; Ground Test, Integrated (GTI)-06 Part 2 HWIL and Ground Test, Distributed-06 Part 2 field testing; GTI-ISR (Israeli) (16) test event; developmental HWIL interoperability testing activities with the U.S. Army Integrated Air and Missile Defense Battle Command System; and SBIRS Increment 2/Tri-wise test campaign in support of the SBIRS Increment 2 capability upgrade decision to the BMDS operational capacity baseline. The Command and Control, Battle Management, and Communications (C2BMC) system integrated U.S. missile defense elements participating in two NATO test events, Ensemble Test 4 DASD(DT&E) FY 2016 Annual Report 43

50 DoD Component Assessments and Steadfast Alliance 2016, in preparation for declaration of NATO BMD IOC. C2BMC successfully demonstrated IAMD interoperability between U.S. Aegis BMD, NATO C2, and ships from eight allied nations during At-Sea Demonstration 2015 in October 2015 at the United Kingdom Hebrides range in Scotland. Reliability T&E MDA issued Directive (Reliability, Availability, and Maintainability (RAM)) in November 2015 to assist in standardization of the execution of reliability engineering activities such as reliability growth testing (RGT) across BMDS subsystems. The directive requires the development of RAM strategies for each BMDS subsystem to influence system design, achieve program R&M requirements, and decrease the logistics burden and cost over the life cycle of the programs. The RAM strategies will be integrated into the SE process, documented in a Government RAM Program Plan, and reflected in acquisition strategies and contract requirements. RGCs will be developed that identify the amount of testing required to verify program/product reliability requirements. The RGCs will contain intermediate goals to track progress toward demonstrating reliability requirements. The test hardware and range support equipment required to conduct reliability growth and demonstration testing will be identified and documented. MDA continued the implementation of HALT in accordance with MDA Policy Memorandum #77. Applications include the design and qualification efforts for the Long-Range Discrimination Radar and the Ground-Based Midcourse Defense redesigned kill vehicle. MDA completed analysis of the RGT on Configuration 2 of the Terminal High-Altitude Area Defense weapon system. The RGT demonstrated a reliability improvement of approximately 63 percent that resulted from corrective actions that were developed, qualified, and implemented to address known hardware and software failure modes. The chartered Standard Missile (SM)-3 Reliability Working Group established quarterly assessment reporting of the SM-3 Blocks IA and IB reliability. The assessment utilizes a Bayesian statistics model to assess the results from ground and flight test data from SM-2, SM-3, and SM-6. The results from the assessments are being used to support flight test readiness executive reviews, BMDS Element/Component Characterizations for Analysis, fleet operational assessments, and the Block IB full-rate production (FRP) decision. T&E Capabilities Although flight tests have become more sophisticated and complex, the test resource budget has decreased by 36 percent since FY MDA eliminated two airborne sensor platforms, the Widebody Airborne Sensor Platform and High-Altitude Observatory (HALO) III aircraft, and the Mobile Launch Platform used to launch targets at sea. The airborne sensors fleet replacement is not scheduled to begin until FY The age of the HALO I, II, and IV aircraft has increased the cost and availability of critical spares and maintenance facilities. These aircraft collect flight test optical data and provide critical data on flight test anomalies. The declining test resource budgets result in deferred maintenance for critical test assets and infrastructure except for safety-related maintenance. At the same time, increasing cybersecurity risk posed to MDA test infrastructure forced the use of a significant portion of the test budget to fund ongoing cybersecurity enhancements. 44 DASD(DT&E) FY 2016 Annual Report

51 DoD Component Assessments To provide the operational realism required to execute Flight Test Operational-02 Event 2, the MDA Director for Test established a range capability in and around Wake Island. This range capability was established because the target trajectory, flight test safety, and debris mitigation requirements at existing ranges were too restrictive to meet the operational realism required for this test. T&E of Autonomous Systems MDA has no autonomous systems DASD(DT&E) Assessment of the MDA Report Based upon the report submitted by MDA and subsequent discussions, the DASD(DT&E) assesses that MDA has adequate personnel supporting T&E, but not all personnel are coded T&E, and MDA has adequate capabilities to support the MDA T&E mission. MDA aligns test analysis and evaluation functions within the Directorate for Engineering and codes these positions as engineering. MDA feels that the Directorate for Engineering works collaboratively with the Directorate for Test to leverage expertise in defining technical objectives and analyzing test results. Although DASD(DT&E) supports the integration between engineering and T&E, DASD(DT&E) continues to recommend that positions performing T&E functions be coded T&E. DASD(DT&E) does not object to the organizational construct of MDA but believes that individuals performing T&E duties should be coded T&E. DASD(DT&E) believes that an individual performing the duties associated with T&E (as described in the T&E Workforce Competency Model) should be coded as T&E regardless of where the individual is aligned organizationally. DASD(DT&E) and MDA continued to work together to resolve this concern. As stated in the DT&E FY 2015 Annual Report, DASD(DT&E) recommends that MDA continue reviewing the positions in the SE organization that are currently performing the analysis, assessment, evaluation, and reporting of test data/results and recode those positions to T&E if the majority of duties and responsibilities are T&E functions. DASD(DT&E) will continue to investigate the possibility of coding positions in more than one career field. MDA is the Lead DT&E Organization for the BMDS program. DASD(DT&E) plans to continue to monitor and review the ability of an organization that is part of a program office to perform the statutory responsibilities of a Lead DT&E Organization. 3.3 T&E Acquisition Workforce T&E Workforce In accordance with DoDI , the DASD(DT&E) is the functional leader for the T&E career field in the acquisition workforce. This section provides a global perspective of the entire DoD T&E workforce, including DT, OT, Government, contractor, acquisition, and non-acquisition. The entire T&E workforce includes personnel supporting all aspects of the T&E mission beyond the acquisition-specific matter. These personnel provide critical expertise in support of the DT&E mission and the success of T&E across DoD but are not part of the acquisition workforce. DASD(DT&E) FY 2016 Annual Report 45

52 DoD Component Assessments Over the last 8 years, the DASD(DT&E) has requested data on the entire T&E workforce. As noted in previous reports, limitations to the data exist because the DoD Components used manual methods rather than automated systems to collect the data and the data were not all-inclusive. Over the years, the DoD Components have improved their manual processes to provide DASD(DT&E) with T&E workforce data that better estimate the entire T&E workforce. The T&E workforce data categories are as follows: Military and Civilians o T&E Coded o Acquisition Coded Non-T&E o Non-Acquisition Coded Additional T&E Support o Support Contractors o FFRDC/UARC o Developer T&E Support T&E Workforce Based on DoD Component Reports Figure 3-2 shows the composition of the T&E workforce based on the latest data from the DoD Component reports. As in past years, the category of civilian acquisition coded non-t&e personnel has the highest percentage at 43 percent, and support contractors ranks next highest at 15 percent, down 8 percent from last fiscal year. The civilian and military T&E-coded personnel percentages remained unchanged at 10 percent and 3 percent, respectively. The civilian non-acquisition T&E personnel (11%) is a small subset of the MRTFB workforce that provides support to the programs and range support during testing. 46 DASD(DT&E) FY 2016 Annual Report

53 DoD Component Assessments Developer T&E Support FFRDC / UARC 5% 0% Support Contr 15% Civ T&E Coded 10% Mil Non Acq Coded 5% Mil Non T&E 8% Civ Acq Coded Non-T&E 43% Mil T&E Coded 3% Civ Non-Acq T&E 11% (0% indicates a number less than 1%) Figure 3-2. FY 2016 T&E Personnel Breakdown DASD(DT&E) FY 2016 Annual Report 47

54 DoD Component Assessments Figure 3-3 displays the composition of the T&E workforce by DoD Component, based on data from the DoD Component reports. (0% indicates a number less than 1%) Figure 3-3. FY 2016 T&E Personnel Breakdown by DoD Components Acquisition T&E Workforce A subset of the entire T&E workforce is the acquisition T&E workforce. Minimal changes occurred in the acquisition T&E workforce over the past 5 years. Table 3-2 shows the acquisition T&E workforce comparison between FY 2015 and FY During FY 2016, the acquisition T&E workforce had an overall increase of 46 T&E positions. The Army showed a decrease in T&E-coded positions, whereas the DON and the Air Force showed an increase. T&E workforce data, extracted from the AT&L Workforce Data Mart system, are consistent with data provided in the DoD Component reports; however, some minor differences exist between the DoD Component data and the data in the AT&L Workforce Data Mart system. The average age of the T&E workforce has decreased from 42.5 years old in FY 2015 to 42.0 years old in FY DASD(DT&E) FY 2016 Annual Report

55 DoD Component Assessments DoD Component Table 3-2. Acquisition T&E Workforce Comparison, FY 2015 vs. FY 2016 FY 2015 FY 2016 Civilian Military Total Civilian Military Total Difference Army 1, ,956 1, , DON 2, ,328 2, , Air Force 1,781 1,246 3,027 1,841 1,241 3, Fourth Estate* TOTAL 6,971 1,721 8,692 6,981 1,757 8, *Fourth Estate refers to DoD organizations, other than the Military Services, having DoD manpower resources. Military personnel assigned to the Fourth Estate organizations are tracked by the Services Key Leadership Positions (CDTs) Background. In accordance with sections 139b and 1706 of Title 10, U.S.C., the Secretary of Defense shall require that each MDAP and MAIS program be supported by a CDT. The CDT is responsible for the following: Coordinating the planning, management, and oversight of all DT&E activities for the program. Maintaining insight into contractor activities under the program and overseeing the T&E activities of other participating Government activities under the program. Helping PMs make technically informed, objective judgments about contractor DT&E results under the program. In addition, USD(AT&L) policy designates the CDT as a mandatory KLP for each MDAP and MAIS program, and the CDT must be designated in the T&E career field. The DoD Components are working to code CDT positions as T&E KLPs for MDAPs and MAIS programs and are filling those positions with qualified individuals. In FY 2016, DASD(DT&E) and the DoD Components continued to focus on supporting implementation of the USD(AT&L) KLP memorandum. Table 3-3 shows the total number of MDAPs and MAIS programs, as reported by the DoD Components, and number of CDTs identified by name. From FY 2015 to FY 2016, the Army had a reduction in MDAPs and MAIS programs and also reduced the number of programs that are not supported by a CDT. Additionally, one Army vacancy reflected does not require that a CDT be aligned to the program because the program is in FRP. The DON also had a reduction in MDAPs and MAIS programs while maintaining the lowest percentage of CDT vacancies among the Military Departments. All of the DON programs except one are supported by a CDT. In addition, the DON reported three pre-mdaps that have a CDT identified by name. Although the Air Force still has the highest number of vacancies, it has increased its number of filled CDT slots by 70 percent (24 to 41) from FY 2015 to FY 2016 and has made significant progress in filling CDT positions. One Air Force Program, the Evolved Expendable Launch Vehicle, procures services rather than systems and does not require a CDT. The Fourth Estate did not experience any changes in its MDAPs and MAIS programs from FY 2015 to FY DASD(DT&E) FY 2016 Annual Report 49

56 DoD Component Assessments DoD Component Table 3-3. MDAPs/MAIS Programs and CDTs in FY 2015 vs. FY 2016 MDAPs/ MAIS Programs FY 2015 FY 2016 CDTs Identified by Name CDTs Vacant MDAPs/ MAIS Programs CDTs Identified by Name CDTs Vacant Army * DON Air Force ** Fourth Estate TOTAL *One program is not required to have a CDT identified; the program is in FRP. **One program is not required to have a CDT identified; the program procures services, not systems. Table 3-4 shows the comparison of the total number of MDAPs and MAIS programs, as reported by the DoD Components, and CDT positions coded as T&E KLPs from FY 2015 to FY Overall, the number of MDAPs and MAIS programs across the DoD Components increased by 10. The DoD Components continued to code CDT positions as T&E KLPs with 66 percent of the 129 MDAPs and MAIS programs having CDT positions that are coded as T&E KLPs. The Army experienced a reduction in MDAPs and MAIS programs from FY 2015 and reduced the number of CDT positions that are not coded as T&E KLPs. The DON maintains the highest number of MDAPs and MAIS programs and CDT positions that are coded as T&E KLPs. The DON reduced the number of CDT positions not coded as T&E KLPs from 10 in FY 2015 to 2 in FY In addition, the DON reported three pre-mdaps that have CDT positions coded as T&E KLPs. The Air Force increased its number of MDAPs and MAIS programs by 19 and continues to have the highest number of CDT positions not coded as T&E KLPs. The Fourth Estate did not experience any change from FY 2015 in its number of programs, and each program has a CDT identified by name that is coded as a T&E KLP. DoD Component Table 3-4. MDAPs/MAIS Programs and CDT T&E KLPs in FY 2015 vs. FY 2016 MDAPs/ MAIS Programs FY 2015 FY 2016 CDT Positions Coded as T&E KLPs CDT Positions Not Coded as T&E KLPs MDAPs/ MAIS Programs CDT Positions Coded as T&E KLPs CDT Positions Not Coded as T&E KLPs Army DON Air Force Fourth Estate TOTAL DASD(DT&E) FY 2016 Annual Report

57 DoD Component Assessments Of the 86 CDT positions that are coded as T&E KLPs, shown in Table 3-4, 71 (or 83%) are filled and identified by name by the DoD Components. Next Steps. DASD(DT&E) will continue to work with the DoD Components as they progress in designating and coding CDTs as T&E KLPs for MDAPs and MAIS programs and tracking qualification of incumbent KLPs. DASD(DT&E) will update requirements and training curriculum to ensure that CDTs are properly qualified. Future annual reports will document progress, as needed. DASD(DT&E) FY 2016 Annual Report 51

58 This page intentionally blank. 52 DASD(DT&E) FY 2016 Annual Report

59 4 ADDITIONAL REPORTING REQUIREMENTS The FY 2013 NDAA, signed on January 2, 2013, includes additional requirements for the DT&E annual report to Congress. The FY 2013 NDAA requires a separate section that addresses the adequacy of resources available to the DASD(DT&E) and the Lead DT&E Organizations of the Military Departments to carry out their responsibilities. 4.1 Adequacy of Resources for DASD(DT&E) DASD(DT&E) resources addressed are the FY 2016 budget and associated staff allocated to carry out assigned responsibilities. The FY 2016 budget, shown in Table 4-1, provides funding for the responsibilities prescribed by law and assigned in DoDI , Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)). Table 4-1. DASD(DT&E) FY 2016 Budget ($K) Program Element FY 2016 President s Budget FY 2016 Appropriation D8Z $17,371 $21,371 As of February 2017, DASD(DT&E) executes its statutory responsibilities with a professional staff of 10 organic Government civilian and military personnel. Since the FY 2015 report, one of the Government civilian positions was eliminated. Table 4-2 provides the DASD(DT&E) Government workforce and contractor support. Organic staff of the DASD(DT&E) office consists of the DASD(DT&E), one SES Principal Deputy, one Military Staff director, five senior civilian (GS-15 level) Deputy Directors, and two civilian staff specialists. The DASD(DT&E) augments its Government staff with personnel detailed from the TRMC. These personnel include three Military Service members and three civilians to provide additional Government representation in program engagements. At the current staffing levels, DASD(DT&E) remains selective in its level of engagement with MDAPs, MAIS programs, and USD(AT&L)-designated special interest programs. DASD(DT&E) assesses DT&E activities to the level of available resources to inform decision makers. DASD(DT&E) Workforce Staffing (Government and Contractor) Table 4-2. DASD(DT&E) Workforce and Contractor Support Organic TRMC Detailee Total Filled Vacant Government Civilian Military Contractor/FFRDC Support Total DASD(DT&E) FY 2016 Annual Report 53

60 Additional Reporting Requirements 4.2 Adequacy of Resources for DoD Component Lead DT&E Organizations Lead DT&E Organizations are responsible for the following: Providing technical expertise on T&E issues to the CDT for the program. Conducting DT&E activities for the program, as directed by the CDT. Assisting the CDT in providing oversight of contractors under the program and in reaching technically informed, objective judgments about contractor DT&E results under the program. DASD(DT&E) monitors and reviews the DT&E activities of the MDAPs (including the activities of the CDTs and Lead DT&E Organizations) Process to Assess the Adequacy of Resources for DoD Component Lead DT&E Organizations The DoD Components reported on their Lead DT&E Organizations and associated programs (see Table 4-3). To assess the adequacy of resources available to the Lead DT&E Organizations, DASD(DT&E) requested that the program offices address the following items: Provide the name of the Government agency serving as the Lead DT&E Organization. Describe the T&E expertise and capabilities (ranges, instrumentation, etc.) needed to support the program in FY 2016 and beyond. Describe current gaps in the Lead DT&E Organization and in any other participating test organizations supporting the Lead DT&E Organization. Describe perceived future gaps in the Lead DT&E Organization and in any other participating test organizations supporting the Lead DT&E Organization. Provide feedback regarding the Lead DT&E Organization and its future ability to support the program office. DASD(DT&E) also requested that the Lead DT&E Organizations address the following items: Provide the unit identification code (UIC) and location of the Lead DT&E Organization. Describe the DT&E activities that have been directed by the CDT and how the Lead DT&E Organization interacts with the CDT on DT&E activities. Describe the test organizations providing significant support to the Lead DT&E Organization. Provide the UICs and locations of the test organizations providing significant support. Describe contractors DT&E capabilities that the Lead DT&E Organization is using to carry out its mission. Describe the Lead DT&E Organization role in assisting the CDT in providing oversight of contractors under the program and in reaching technically informed, objective judgments about contractor DT&E results. 54 DASD(DT&E) FY 2016 Annual Report

61 Additional Reporting Requirements Provide details of any high-demand skills and expertise that the Lead DT&E Organization provides to programs. Describe current gaps in skills and expertise and the Lead DT&E Organization actions/plans to fill the gaps. Describe perceived future gaps in skills and expertise and the Lead DT&E Organization actions/ plans to fill the gaps. Table 4-3. List of Lead DT&E Organizations and Programs Lead DT&E Program Name Organization ARMY ATEC AFED Army IAMD IFPC2-I Joint Air-to-Ground Missile (JAGM) M109A7 Family of Vehicles (FoV) Self-Propelled Howitzer (SPH) and Carrier, Ammunition, Tracked (CAT) Vehicle ATEC C4ISRED Joint Tactical Radio System (JTRS) Handheld, Manpack, and Small Form Fit (HMS) Manpack (MP) Radio JTRS HMS Rifleman Radio (RR) Mid-Tier Networking Vehicular Radio (MNVR) (AN/VRC-118(v)1) ATEC MSED Abrams M1A2 System Enhancement Package Version 3 (SEPv3) ECP 1a Bradley Family of Vehicles (BFoV) ECP 2 JLTV Stryker FoV ECP AMSAA ACWA NAVY NAWCAD HX-21 CH-53K Heavy-Lift Replacement Helicopter NAWCAD VX-20 MQ-4C Triton Unmanned Aircraft System (UAS) Multi-Mission Maritime Aircraft (P-8A Poseidon) NAWCAD VX-23 F-35 Lightning II NSWC PHD Littoral Combat Ship (LCS) and Mission Packages (MPs) U.S. Navy IAMD ZUMWALT-Class Destroyer (DDG 1000) NUWC NPT VIRGINIA-Class Submarine PMS 378T GERALD R. FORD Class Nuclear Aircraft Carrier (CVN 78) PMS 397 COLUMBIA-Class Submarine SSC LANT DCGS-N Increment 2 SSP Trident II Life Extension (D5LE) AIR FORCE 96th Test Wing AOC-WS Increment 10.2 B61-12 LEP TKA CRH Family of Advanced Beyond Line-of-Sight Terminals (FAB-T) JMS Increment 2 DASD(DT&E) FY 2016 Annual Report 55

62 Additional Reporting Requirements Lead DT&E Organization 412th Test Wing SMC/GPEV SMC/RSE MDA Directorate for Test Program Name Small Diameter Bomb Increment II (SDB II) F-22 Increment 3.2B Modernization F-35 Lightning II KC-46A Tanker Modernization GPS Enterprise SBIRS MDA BMDS Assessment of Adequacy of Resources for DoD Component Lead DT&E Organizations The DoD Components reported that the resources for the Lead DT&E Organizations are adequate to support near-term priorities and identified some concerns. The DoD Components are implementing different constructs to meet the statutory requirement that each MDAP be supported by a governmental test agency, serving as Lead DT&E Organization for the program. The Lead DT&E Organizations for all Army MDAPs and MAIS programs are within ATEC AEC, whose mission includes DT&E and OT&E activities. AMSAA serves as the Lead DT&E Organization for the DoD ACWA program. The Navy Lead DT&E Organizations include program offices, warfare centers, and PEOs. The Air Force uses test wings and SMC to perform the Lead DT&E Organization duties. The MDA Directorate for Test is the Lead DT&E Organization for the BMDS program, and DISA did not provide information on the Lead DT&E Organization because DISA does not have any MDAPs. Since the statutory requirement began in FY 2012, DASD(DT&E) has been reviewing the constructs annually and will report out in future reports as these models evolve and acquisition outcomes are realized. DASD(DT&E) continues to engage with the Lead DT&E Organizations, address their concerns, and monitor the DT&E capabilities needed by AEC and the ability of an organization that is part of a program office to perform the statutory responsibilities of a Lead DT&E Organization. Tables 4-4 through 4-7 provide the assessments of the adequacy of resources for the Lead DT&E Organizations to carry out their responsibilities Army Lead DT&E Organizations The Army Lead DT&E Organizations within ATEC AEC include the Aviation-Fires Evaluation Directorate (AFED), the C4ISR Evaluation Directorate (C4ISRED), and the Mounted Systems Evaluation Directorate (MSED). The primary focus of AEC is to plan, support, conduct, and provide independent evaluations, assessments, and experiments in order to provide essential information to decision makers. The AEC mission includes DT&E and OT&E activities. DASD(DT&E) continues to monitor the DT&E capabilities needed by AEC to perform the activities of a Lead DT&E Organization and will report out in future reports, if needed. AMSAA serves as the Lead DT&E Organization for the DoD ACWA program. 56 DASD(DT&E) FY 2016 Annual Report

63 Additional Reporting Requirements Table 4-4. Assessment of Adequacy of Resources for Army Lead DT&E Organizations Lead DT&E Supported Organization Programs Assessment ATEC AFED Army IAMD DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC AFED are adequate to support near-term priorities. The PMO identified a concern about future hiring limitations at test ranges (see below). The 2017 NDAA authorized the temporary (2017 and 2018) direct hiring authority for major range and test facilities. DASD(DT&E) and TRMC will support the MRTFBs in implementing the direct hiring authority, which will provide the MRTFBs with the flexibility to address the PMO s concerns. The PM also identified that a large amount of data will be collected and transferred to off-site evaluation centers. ATEC AFED is aware of the challenge and will need to coordinate with White Sands Missile Range (WSMR), New Mexico, to ensure the timely transfer of data for timely analysis. Lead DT&E Organization: ATEC AFED identified no gaps but acknowledged the challenges of supporting multiple programs at the range concurrently and the need to ensure the timely transfer of data and timely analysis. PMO: The PMO identified no gaps. The PMO did express concerns about the ability of the range to hire personnel with the expertise and experience needed to support future testing and the ability of the range to transfer large amounts of data for analysis. ATEC AFED IFPC2-I DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC AFED are adequate to support near-term priorities. The current and future gaps (identified below) are being addressed. DASD(DT&E) will monitor progress. Lead DT&E Organization: ATEC AFED recognized a capability gap in the inability of current instrumentation to meet IFPC2-I s target service rate. ATEC plans to use M&S to evaluate this requirement. ATEC AFED is aware of the PMO s perceived future gap (identified below). PMO: The PMO identified two gaps. A current gap is the limitation of current instrumentation to support the full verification of IFPC2-I s target service rate requirement. A perceived future gap is the lack of Operational Test Command (OTC) personnel, which may limit OTC effectiveness in supporting major programs simultaneously; currently, the same OTC staff is assigned to support IFPC2-I and Army IAMD, and both programs have multiple OT events planned in the near future. ATEC AFED JAGM DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC AFED are adequate to support near-term priorities. No current gaps have been identified. DASD(DT&E) will monitor the PM s potential future gap (identified below). Lead DT&E Organization: ATEC AFED identified no gaps. PMO: The PM identified a potential future gap. New rocket motor solutions are in development that will provide extended-range capability to JAGM and Hellfire missiles. JAGM has an unfunded requirement to achieve a range of 16 kilometers in the future. The current Army test ranges may not provide the ability to test these extended-range missiles over the entire engagement envelope or in operationally realistic conditions. DASD(DT&E) FY 2016 Annual Report 57

64 Additional Reporting Requirements Lead DT&E Organization ATEC AFED ATEC C4ISRED ATEC C4ISRED ATEC C4ISRED Supported Programs M109A7 FoV SPH and CAT Vehicle JTRS HMS MP Radio JTRS HMS RR MNVR (AN/VRC- 118(v)1) Assessment DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC AFED are adequate to support near-term priorities. No current gaps have been identified. DASD(DT&E) will monitor the PM s potential future gaps (identified below). Lead DT&E Organization: ATEC AFED identified no gaps. If future testing requires the development of new standardized test procedures, ATEC, as part of its annual standardization program, will then develop new procedures or revise existing procedures to meet the requirement. PMO: The PM identified potential future gaps: the lack of a standard test operating procedure for new technologies such as cybersecurity testing, active protection systems, and other inbound communication/network and survivability-related technologies; the lack of resources and process for updating test operations procedures and international test operations procedures; and the lack of resources to update instrumentation. DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC C4ISRED are adequate to support near-term priorities. No gaps have been identified. DASD(DT&E) recognizes the challenges noted by the PM and will monitor progress. Lead DT&E Organization: ATEC C4ISRED identified no gaps. PMO: The PM noted the need to work closely with ATEC on the capability to appropriately test the system against its cyber requirements, focus on additional analytical techniques for processing data from the field into quick reports, and ensure additional emphasis on maintaining subject matter expertise on radio testing while evolving for the future. DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC C4ISRED are adequate to support near-term priorities. No gaps have been identified. DASD(DT&E) recognizes the challenges noted by the PM and will monitor progress. Lead DT&E Organization: ATEC C4ISRED identified no gaps. PMO: The PM noted the need to work closely with ATEC on the capability to appropriately test the system against its cyber requirements, focus on additional analytical techniques for processing data from the field into quick reports, and ensure additional emphasis on maintaining subject matter expertise on radio testing while evolving for the future. DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC C4ISRED are adequate to support near-term priorities. No gaps have been identified. DASD(DT&E) recognizes the challenges noted by the PM and will monitor progress. Lead DT&E Organization: ATEC C4ISRED identified no gaps. PMO: The PM noted the current need for ATEC to better integrate SoS testing. The PM also noted that resources are not always available in the given timeline or location to support individual program testing. A persistent network test bed is needed to support tests that require a small number of radio resources so that the larger networks can be released to support the larger tests. The PM noted that there are emerging cyber requirements and threats that will need to be assessed appropriately with rapid turnarounds on data to aid fielding. Software updates will most likely occur on a recurring basis with the need for future regression testing. Future testing on alternative Army platforms will also be required need to ensure asset availability, coordination, and integration are available and testable. 58 DASD(DT&E) FY 2016 Annual Report

65 Additional Reporting Requirements Lead DT&E Organization ATEC MSED Supported Programs Abrams M1A2 SEPv3 ECP 1a Assessment DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC MSED are adequate to support near-term priorities. No current gaps have been identified. DASD(DT&E) recognizes the personnel challenges identified below as a potential future gap and will monitor progress. Lead DT&E Organization: ATEC MSED identified a potential future gap in the loss of qualified test site personnel because of budget cuts and retirement attrition. The loss of positions across the command makes it more difficult to backfill retiring personnel with other experienced personnel. ATEC is working hard to hire, train, and develop personnel who can fill these positions as hiring authority is granted. PMO: The PMO identified the retiring SMEs at ATEC as a potential future gap. ATEC MSED BFoV ECP 2 DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC MSED are adequate to support near-term priorities. DASD(DT&E) recognizes the identified gaps and will monitor progress. Lead DT&E Organization: ATEC MSED identified the lack of expertise in database management for large data sets as a current gap. Future gaps include the loss of qualified test site personnel because of budget cuts and retirement attrition and the lack of standardized test procedures for cybersecurity testing and active protection systems. PMO: The PMO identified the lack of standardized test procedures for cybersecurity testing and active protection systems as a gap. ATEC MSED JLTV DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC MSED are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: ATEC MSED identified no gaps. PMO: The PM identified no gaps. ATEC MSED Stryker FoV ECP DASD(DT&E): DASD(DT&E) assesses that the resources for ATEC MSED are adequate to support near-term priorities. DASD(DT&E) recognizes the gaps identified by ATEC MSED and the PMO and will monitor progress. Lead DT&E Organization: ATEC MSED identified no current gaps. In case a backload of work does occur, the AEC Test Manager and ATEC G-9 can work with the test teams at primary and reinforcing test centers to mitigate the temporary gaps in personnel and facilities and to manage surge work and complete planned testing in a timely manner. As a perceived future gap, ATEC MSED identified the possibility of losing qualified test site personnel because of budget cuts and retirement attrition. The loss of positions across the command makes it more difficult to backfill retiring personnel with other experienced personnel. ATEC is working hard to hire, train, and develop personnel who can fill these positions as hiring authority is granted. PMO: The PMO stated that staffing levels at ATEC may need to be increased to support current and future major programs. It is anticipated that Stryker will have major high-profile efforts concurrently in test across multiple ATEC sites. The PMO also stated that significant testing is projected to occur at the Electronic Proving Ground (EPG), Fort Huachuca, Arizona. Currently, there is only one test director supporting Stryker testing. It is desirable and recommended that additional manpower be considered at EPG to support future Stryker workloads. DASD(DT&E) FY 2016 Annual Report 59

66 Additional Reporting Requirements Lead DT&E Supported Organization Programs Assessment AMSAA ACWA DASD(DT&E): DASD(DT&E) assesses that the resources for AMSAA are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: AMSAA identified no gaps. PMO: The PM identified no gaps Navy Lead DT&E Organizations The Navy Lead DT&E Organizations include program offices, warfare centers, and PEOs. For the programs reviewed in this report, the Navy Lead DT&E Organizations include the Naval Air Warfare Center, Aircraft Division (NAWCAD), Naval Rotary-Wing Aircraft Test and Evaluation Squadron Two One (HX-21); NAWCAD, Air Test and Evaluation Squadron Twenty (VX-20); NAWCAD, Air Test and Evaluation Squadron Twenty-Three (VX-23); Naval Surface Warfare Center (NSWC), Port Hueneme Division (PHD); Naval Undersea Warfare Center (NUWC), Newport (NPT); PEO for Aircraft Carriers (PMS 378T); COLUMBIA-Class Submarine Program Office (PMS 397); SPAWAR Systems Center Atlantic (SSC LANT); and Strategic Systems Programs (SSP). Table 4-5. Assessment of Adequacy of Resources for Navy Lead DT&E Organizations Lead DT&E Organization NAWCAD HX-21 NAWCAD VX-20 Supported Programs CH-53K MQ-4C Triton UAS Assessment DASD(DT&E): DASD(DT&E) assesses that the resources for NAWCAD HX-21 are adequate to support near-term priorities. DASD(DT&E) will monitor the gaps identified below by NAWCAD HX-21. Lead DT&E Organization: NAWCAD HX-21 is actively seeking to fill three additional avionics positions and is coordinating with NAVAIR T&E Competency to increase the size of the test team over the course of the next year to address the expected future workload. PMO: The PMO identified no gaps. DASD(DT&E): DASD(DT&E) assesses that the resources for NAWCAD VX-20 are adequate to support near-term priorities. DASD(DT&E) will monitor the concern identified below. Lead DT&E Organization: NAWCAD VX-20 reported that because of competing needs from other air programs, Test Pilot School (TPS)/naval flight officer (NFO) manning is being closely monitored and managed with VX-20 in support of flight testing needs. Full-time equivalent support is also being challenged in very specialized disciplines such as flutter, loads, cybersecurity, and test directing. Aircrew staffing needs have been discussed as part of the T&E WIPT, and requirements have been briefed to the NAVAIR manpower board annually. PMO: The PMO also reported that TPS/NFO manning is being closely monitored and managed with VX-20 in support of flight testing needs. 60 DASD(DT&E) FY 2016 Annual Report

67 Additional Reporting Requirements Lead DT&E Organization NAWCAD VX-20 NAWCAD VX-23 Supported Programs P-8A Poseidon F-35 Lightning II Assessment DASD(DT&E): DASD(DT&E) assesses that the resources for NAWCAD VX-20 are adequate to support near-term priorities. DASD(DT&E) will monitor the concerns identified below by NAWCAD VX-20. Lead DT&E Organization: NAWCAD VX-20 reported that reductions in TPS-trained NFOs and reductions in enlisted aircrew staffing are being closely monitored to enable test execution. NAWCAD VX-20 is working to staff cybersecurity personnel and improve labs and infrastructure. NAVAIR should have the capability to organically store and process P-8 airworthiness test data for future increments. PMO: The PMO identified no gaps, but reductions in TPS-trained NFOs and reductions in enlisted aircrew staffing are being closely monitored to address any risks to program execution. DASD(DT&E): DASD(DT&E) assesses that the resources for NAWCAD VX-23 are adequate to support near-term priorities. Although no gaps have been identified by VX-23, the TRMC and DASD(DT&E) have been leading a Fifth-Generation Aerial Threat Emulation (5GATE) Shortfall Analysis of Alternatives activity to find materiel solutions to fill known test capability gaps in 5GATE capabilities for both DT and OT. Lead DT&E Organization: NAWCAD VX-23 identified no gaps. NAWCAD VX-23 and the Joint Program Office (JPO) are actively working on the size and scope of the DT aircraft fleet and test team staffing. JPO: The JPO identified no current or future perceived gaps. The F-35 program, in the midst of a transformation lasting 3 to 5 years, will work to understand, define, and address perceived future gaps. The JPO and the Lead DT&E Organizations are actively working on the size and scope of the DT aircraft fleet and test team staffing. The F-35 test force needs capability and capacity to support the F-35 program s transition through system development and demonstration (SDD) flight test planned to the end of CY 2017, IOT&E execution in FY 2018, and post-sdd/follow-on modernization DT activities (including but not limited to the Block 4 program). Activities such as post-sdd fleet capacity studies, DT aircraft viability contract actions, and revised program introduction documents are under way to understand, define, and address future needs. NSWC PHD LCS and MPs DASD(DT&E): DASD(DT&E) assesses that the resources for NSWC PHD are adequate to support near-term priorities. No gaps have been identified. NSWC PHD NSWC PHD U.S. Navy IAMD ZUMWALT- Class Destroyer (DDG 1000) Lead DT&E Organization: NSWC PHD identified no gaps. PMO: The PMO identified no gaps. DASD(DT&E): DASD(DT&E) assesses that the resources for NSWC PHD are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: NSWC PHD identified no current gap. NSWC PHD is actively pursuing additional DT personnel in preparation for an expected increase in the level of effort and number of events in the upcoming DT schedule. PMO: The PMO identified no gaps. DASD(DT&E): DASD(DT&E) assesses that the resources for NSWC PHD are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: NSWC PHD identified no current gap. NSWC PHD is actively pursuing additional DT personnel in preparation for an expected increase in the level of effort and number of events in the upcoming DASD(DT&E) FY 2016 Annual Report 61

68 Additional Reporting Requirements Lead DT&E Organization NUWC NPT Supported Programs VIRGINIA- Class Submarine DT schedule. Assessment PMO: The PMO identified no gaps. DASD(DT&E): DASD(DT&E) assesses that the resources for NUWC NPT are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: NUWC NPT identified no gaps. PMO: The PMO identified no gaps. PMS 378T CVN 78 DASD(DT&E): DASD(DT&E) assesses that the resources for PMS 378T are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: PMS 378T identified no gaps. PMO: The PMO identified no gaps. PMS 397 SSC LANT COLUMBIA- Class Submarine DCGS-N Increment 2 DASD(DT&E): DASD(DT&E) assesses that the resources for PMS 397 are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: PMS 397 identified no gaps. PMO: The PMO identified no gaps. DASD(DT&E): DASD(DT&E) assesses that the resources for SSC LANT are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: SSC LANT identified no gaps. PMO: The PMO identified no gaps. SSP Trident II D5LE DASD(DT&E): DASD(DT&E) assesses that the resources for SSP are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: SSP identified no gaps. PMO: The PMO identified no gaps Air Force Lead DT&E Organizations The Air Force uses test wings and SMC to perform the Lead DT&E Organization duties. The Air Force Lead DT&E Organizations include the 96th Test Wing; 412th Test Wing; GPS Directorate s Systems Engineering Division, SMC (SMC/GPEV); and Remote Sensing Division, SMC (SMC/RSE). Table 4-6. Assessment of Adequacy of Resources for Air Force Lead DT&E Organizations Lead DT&E Organization 96th Test Wing Supported Programs AOC-WS Increment 10.2 Assessment DASD(DT&E): DASD(DT&E) assesses that the resources for the 96th Test Wing are adequate to support near-term priorities. DASD(DT&E) will monitor the gaps identified below. Lead DT&E Organization: The 96th Test Wing stated that a current and future gap is the limited number of personnel available to support concurrent test events for several programs. The 96th Test Wing planned for manpower turnovers with targeted hires and aggressive training and will continue this practice into the future. The rate of personnel growth is dependent on funding. PMO: The PMO identified a potential future gap in the limited number of 46th Test Squadron personnel available to support increased test demands. 62 DASD(DT&E) FY 2016 Annual Report

69 Additional Reporting Requirements Lead DT&E Organization 96th Test Wing 96th Test Wing 96th Test Wing 96th Test Wing 96th Test Wing 412th Test Wing Supported Programs B61-12 LEP TKA CRH FAB-T JMS Increment 2 SDB II F-22 Increment 3.2B Modernization Assessment DASD(DT&E): DASD(DT&E) assesses that the resources for the 96th Test Wing are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: The 96th Test Wing identified no gaps. PMO: The PMO identified no gaps. DASD(DT&E): DASD(DT&E) assesses that the resources for the 96th Test Wing are adequate to support near-term priorities. DASD(DT&E) will monitor the gap identified below. Lead DT&E Organization: The 96th Test Wing identified no gaps. PMO: The PMO identified cybersecurity testing as a current gap. DASD(DT&E): DASD(DT&E) assesses that the resources for the 96th Test Wing are adequate to support near-term priorities. DASD(DT&E) will monitor the gap identified below. Lead DT&E Organization: The 96th Test Wing identified no gaps. PMO: The PMO identified cybersecurity testing as a current gap. DASD(DT&E): DASD(DT&E) assesses that the resources for the 96th Test Wing are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: The 96th Test Wing identified no gaps. PMO: The PMO identified no gaps. DASD(DT&E): DASD(DT&E) assesses that the resources for the 96th Test Wing are adequate to support near-term priorities. DASD(DT&E) will monitor the gaps identified below. Lead DT&E Organization: The 96th Test Wing identified a current gap in control room qualified flight test engineers. This gap is being filled through Government support contractors. PMO: The PMO identified the expected shortage of support aircraft as a potential future gap. DASD(DT&E): DASD(DT&E) assesses that the resources for the 412th Test Wing are adequate to support near-term priorities. DASD(DT&E) will monitor the gaps identified below. Lead DT&E Organization: The 412th Test Wing identified as a current gap the shortage of control room qualified flight test engineers. The gap is being filled through Government support contractors. PMO: The PMO identified the following currents gaps: The ability of the Lead DT&E Organization to generate F-16 support targets to accomplish the two-versus-many requirements is difficult. The KC-46 has taken some of the limited AFTC F-16s to Seattle, Washington, for support. JSF (both OT and DT) requires a lot of target support. Increment 3.2B has significant target support requirements. AFTC has implemented Raptor Days to minimize the impact. Currently, all Navy ranges have a restriction on the AIM-120 Flight Termination Systems, which essentially prohibits F-22 live-fire testing of AIM-120 missiles on those ranges. Limited range and drone availability is impacting AIM-9X live fires. The lack of availability of the Navy ranges has forced the F-22 to accomplish live fires at WSMR and the Utah Test and Training Range and requires waiting for range time when the test program is ready to execute live-fire DASD(DT&E) FY 2016 Annual Report 63

70 Additional Reporting Requirements Lead DT&E Organization 412th Test Wing 412th Test Wing Supported Programs F-35 Lightning II KC-46A Tanker Modernization Assessment testing. The PMO noted that the expected shortage of support aircraft will continue to be a problem for F-22 testing in the future. DASD(DT&E): DASD(DT&E) assesses that the resources for the 412th Test Wing are adequate to support near-term priorities. Although no gaps have been identified by the 412th Test Wing, the TRMC and DASD(DT&E) have been leading a 5GATE Shortfall Analysis of Alternatives activity to find materiel solutions to fill known test capability gaps in 5GATE capabilities for both DT and OT. Lead DT&E Organization: The 412th Test Wing identified no gaps. JPO: The JPO identified no gaps. DASD(DT&E): DASD(DT&E) assesses that the resources for the 412th Test Wing are adequate to support near-term priorities. DASD(DT&E) will monitor the gaps identified below. Lead DT&E Organization: The 412th Test Wing identified no gaps. PMO: The PMO identified engineering manpower to develop test plans, execute tests, analyze data, and generate reports as a current and future gap. SMC/GPEV GPS Enterprise DASD(DT&E): The Air Force has yet to increase the number of skilled test personnel necessary to execute T&E plans as identified in the recent Nunn- McCurdy recertification to Congress. Lead DT&E Organization: SMC/GPEV identified no gaps. PMO: The PMO identified no gaps. SMC/RSE SBIRS DASD(DT&E): DASD(DT&E) assesses that the resources for SMC/RSE are adequate to support near-term priorities. No gaps have been identified. Lead DT&E Organization: SMC/RSE identified no gaps. PMO: The PMO identified no gaps MDA Lead DT&E Organization MDA is the Lead DT&E Organization for the BMDS program. MDA has assigned an organization, MDA Directorate for Test, within the agency to act as the Lead DT&E Organization. Table 4-7. Assessment of Adequacy of Resources for MDA Lead DT&E Organization Lead DT&E Organization MDA Directorate for Test Supported Program BMDS Assessment DASD(DT&E): DASD(DT&E) assesses that the resources for MDA are adequate to support near-term priorities. No gaps have been identified. DASD(DT&E) plans to continue to monitor and review the ability of an organization that is part of a program office to perform the statutory responsibilities of a Lead DT&E Organization. Lead DT&E Organization: The MDA Directorate for Test identified no gaps. PMO: The PMO identified no gaps. 64 DASD(DT&E) FY 2016 Annual Report

71 5 DOD TEST RESOURCE MANAGEMENT CENTER In FY 2016, the TRMC provided advocacy, oversight, and guidance for all matters pertaining to assessment of and strategic planning for DoD T&E resources. These responsibilities included annual certification of the Service and Defense Agency T&E budgets and wide-ranging studies on topics such as fifth-generation threat requirements; a hypersonics business case; T&E of autonomous systems; costs associated with long-range, open-ocean strategic weapons tests; and test range capabilities to support testing against emerging advanced threats. In addition, the TRMC oversees management of the CTEIP, T&E/S&T Program, JMETC Program, and NCR. In March 2016, the Director, TRMC was designated as the DoD Executive Agent (EA) for Cyber Test Ranges. 5.1 TEMP Review The TRMC reviewed acquisition program TEMPs as needed to support DASD(DT&E). In FY 2016, the TRMC reviewed 13 TEMPs to assess the adequacy of test resources documented in the TEMP, including test infrastructure, distributed testing, interoperability, and cybersecurity. The TRMC provided feedback, via DASD(DT&E), to the program offices on ways to address test limitations and improve program awareness of DoD test capabilities. Similar to FY 2015, the TRMC identified several issues for further investigation including the following: test target and threat systems cost, availability, and fidelity in threat representations; and limitations to support full power testing of jamming systems. To address these issues, the TRMC initiated the Fifth-Generation Aerial Target Joint Service Tiger Team, continued investments in threat-representative target technologies via S&T programs, and continued engagement with range sustainability stakeholders to monitor and help mitigate encroachment and environmental regulation impacts. 5.2 Strategic T&E Roadmaps For the first time, the TRMC incorporated roadmaps and test capability assessments into the Strategic Plan for DoD T&E Resources, outlining plans for enhancement of key test capability areas with DoD-wide applicability and impact across multiple acquisition programs. Cyberspace T&E Infrastructure Roadmap. Lays out the vision for six key enabling capabilities to support acquisition testing; research, development, and S&T initiatives; training; and combatant command exercises. Radio Frequency (RF) Spectrum for T&E Common Operating Picture. Outlines spectrum T&E needs in concert with the DoD Electromagnetic Spectrum (EMS) Strategy and the associated DoD EMS Strategy Roadmap and Action Plan (RM&AP). Electronic Warfare (EW) T&E Infrastructure Improvements Roadmap. Outlines the approach to address DoD high-priority EW test capability needs through balanced investments across HWIL, installed system test facilities (ISTFs), and OARs. DASD(DT&E) FY 2016 Annual Report 65

72 DoD TRMC Hypersonics T&E Resource Investment Roadmap. Outlines needed enhancements to existing ground test facilities, flight test ranges, instrumentation, M&S tools, test methodologies, and workforce to support ongoing hypersonics weapon acquisition programs and future test requirements. Infrared Countermeasures (IRCM) T&E Resource Investment Roadmap. Focuses on developing test capabilities to support T&E of aircraft survivability equipment involving IRCM systems for defeating electro-optical/infrared (IR) guided missiles and hostile fire from unguided munitions. Nuclear Survivability Assessment. Outlines plans for seven test capability shortfalls identified in the Chemical, Biological, Radiological, and Nuclear Survivability Oversight Group s Nuclear Survivability Test Capability Roadmap. Targets Assessment. Outlines plans for target development, procurement, and sustainment. 5.3 Test Infrastructure Studies Fifth-Generation Threat Requirements Study As directed by the USD(AT&L), the TRMC completed a Joint Rapid Analysis Team study to determine requirements for the development of an emulation of fifth-generation threat aircraft. In this study report, the TRMC delivered unanimous consensus across the Services and DOT&E on Fifth-Generation Aerial Threat Emulation (5GATE) requirements. In July 2016, the USD(AT&L) directed, and the TRMC and DASD(DT&E) have been leading, a 5GATE Shortfall Analysis of Alternatives activity to find materiel solutions to fill known capability gaps in 5GATE capabilities for both DT and OT. This report will be complete in March 2017 to inform the FY 2019 budget process Hypersonics Business Case Report In the follow-up to an OSTP/Secretary of Defense/Administrator of NASA study directed by the FY 2013 NDAA, the TRMC led development of a DoD report and plan on the requirements and proposed investments to cost-effectively meet DoD needs through The 2015 report and plan provided an effective business case for the successful request of $350 million over the Future Years Defense Program (FY 2017 FY 2022) to develop a suite of infrastructure for hypersonic weapon system R&D Range Capability for Emerging Advanced Threats In response to a query posed by the House Armed Services Committee (HASC) on the FY 2017 NDAA, the TRMC initiated an assessment of the technologies, gaps, footprint, and workforce needs anticipated for fifth- and sixth-generation aircraft and armaments and an assessment and recommendation of investment alternatives to address those shortfalls. Results of this assessment will be briefed to the HASC in March DASD(DT&E) FY 2016 Annual Report

73 DoD TRMC T&E of Autonomous Systems This study will organize and standardize current and future autonomous system T&E methods, capabilities, and infrastructure and will address T&E challenges unique to testing autonomous systems, such as measuring trust Military Construction As directed by House Report on the FY 2016 NDAA, the TRMC conducted a comprehensive assessment of MRTFB-only military construction needs and investments. The assessment provided an estimated cost to replace or bring to code deficient structures, as well as a plan for ensuring sufficient capacity for all MRTFB facilities to support current and projected future operations. The final report was submitted to the congressional defense subcommittees in April Congressional Briefing on Prioritization of Joint Test Activities The HASC expressed concern about DoD policies and processes for scheduling T&E resources for joint and multi-service acquisition programs and R&D projects (HASC H.R. 4909). The HASC directed the Director, TRMC to provide the committee with a briefing addressing the underlying concerns, which includes recommendations for process improvements Reimbursement Process for the MRTFB In Senate Report S on the FY 2017 NDAA, the Senate Armed Services Committee (SASC) directed the Secretary of Defense to submit a report on how the reimbursement process for the MRTFB relates to operational unit payment procedures. The committee is concerned that a number of optimal and potentially lower cost training opportunities are declined by operational training units because of the difficulty of locating funds to reimburse MRTFBs. The report is due to the SASC and HASC by February 1, Lease Versus Buy of DoD Test Capability Senate Report accompanying H.R on the FY 2016 NDAA contains an Item of Special Interest entitled Acquisition of Commercial Test, Research, and Measurement Capability. The committee language states that the USD(AT&L) should conduct a review of the acquisition practices for acquiring COTS research, test, and measurement equipment and capabilities and report to the congressional defense committees not later than 120 days after the date of enactment of this Act. The USD(AT&L) found that there is no prohibition for the lease of test equipment by DoD Components. The DoD test community has available and has utilized a variety of options through which it can lease commercially available test equipment whenever leasing rather than purchasing provides the best value to the Government. DASD(DT&E) FY 2016 Annual Report 67

74 DoD TRMC Space Threat Environment In 2014, the President and national security team tasked DoD to increase investment in space resilience and capabilities and improve DoD space governance so as to preserve these investments over time. Informed by this guidance, DoD conducted a Space Strategic Portfolio Review, which concluded that DoD needs to more fully address space as a potential domain of conflict rather than as a sanctuary, and to shift its view of space from an enabling function to a warfighting mission area. The DOT&E provided guidance on threat representation in the OT&E of space systems: to identify and address the resource and infrastructure limitations that currently constrain our ability to conduct adequate testing of space systems. In response, the TRMC is conducting a space threats representation study to guide the space system acquisition community by identifying threats to space systems that DoD is facing and will face in the future; to provide the program offices and the developmental and operational test communities with the descriptions of facilities and locations of threat representations for T&E; and to provide an investment roadmap to address gaps in those threat representations Study of National Security Space and Missile Test Ranges and Infrastructure Congress directed the Office of Cost Assessment and Program Evaluation (CAPE) to conduct an evaluation of space and missile test ranges and infrastructure, supporting a USD(AT&L) briefing to the HASC. The TRMC supported CAPE in the collection of data and the conduct of a business case analysis to determine whether the test goals can be accomplished while achieving meaningful savings. The study was limited to assessment of telemetry collection and flight safety, which are mandatory and are common requirements of all flight tests. CAPE concluded that the business case for a space-based or UAV-based range does not exist at this time if used solely for the purposes of telemetry collection for hypersonics. Technology was not sufficiently mature to present alternatives for a space-based range during the course of the CAPE study. Further work is required to develop operating concepts and to assess requirements and the number of flight tests required. Technology investments enable more efficient test operations. The TRMC is investing in S&T that will improve flexibility and reduce complexity in space and missile tests. As these technologies mature and information becomes available to support the business cases, space-based or UAV-based ranges may become viable Cyber Study In 2015, the TRMC conducted a Cyberspace T&E Infrastructure (CT&EI) Capabilities-Based Assessment (CBA) in support of a CAPE-directed Office of the USD(AT&L) Cyber Ranges Evaluation of Alternatives. This study directly supported a POM 17 issue resulting in a significant increase in TRMC funding to address identified gaps. The CBA further informed the FY 2016 DASD(DT&E)-chartered Institute for Defense Analyses DoD Cybersecurity T&E Capabilities and Gaps Study to characterize the capability and capacity of the DoD cybersecurity DT&E enterprise and related gaps and recommendations. Once designated as the EA for Cyber Test Ranges in March 2016, the TRMC chartered the RAND Corporation, an FFRDC, to conduct a study to identify potential requirements for federated and integrated cyber test range infrastructure and to suggest technical and operational standards for that 68 DASD(DT&E) FY 2016 Annual Report

75 DoD TRMC infrastructure to enable effective and efficient testing across DoD s cyber capabilities. The TRMC also formed a formal Cyber Test Range Requirements Working Group with Service representatives to validate those requirements to implement the vision of a federated CT&EI for testing the cybersecurity posture of DoD systems Test and Training EMS Usage Data Collection and Reporting Study (T2 EMS Usage Study) The TRMC, in conjunction with the DoD CIO, the Defense Spectrum Organization, and the Under Secretary of Defense for Personnel and Readiness (USD(P&R)) cosponsored a 9-month T2 EMS Usage Study in direct support of Recommendation 21 of the DoD EMS Strategy and RM&AP, which calls for DoD to collect spectrum usage data at 50 percent of test and training activities by A tri-service team (led by the Army) assessed current spectrum collection and reporting systems and policies, and proposed selected frequencies and a methodology for initiating a pilot data collection and reporting effort starting in The TRMC and other cosponsors of the study are assessing the study recommendations to initiate a pilot effort on western test and training ranges to begin in T&E/S&T Program The T&E/S&T Program develops test technologies to keep pace with evolving weapons technologies. Funded within the Advanced Technology Development Budget Activity, the T&E/S&T Program is critical to ensuring that DoD has the ability to adequately test advanced systems that will be fielded in the future. T&E/S&T Program technology development projects typically begin at Technology Readiness Level (TRL) 3 and mature to TRL 6; deliverables include test technology prototypes and demonstrations in relevant test environments. The T&E/S&T Program addresses long-term gaps in the T&E infrastructure, as well as risk reduction for the development of test capabilities by CTEIP, JMETC, NCR, and DoD Component I&M efforts. The TRMC centrally manages the T&E/S&T Program. The program employs a decentralized execution process through eight Test Technology Areas, each of which is led by an Executing Agent from one of the Services and based at a test organization in the field. Moreover, each Executing Agent leads a working group composed of representatives from the DoD T&E and S&T communities, with expertise related to the respective test technology. The eight Test Technology Areas are C4I and Software-Intensive Systems Test, Cyberspace Test Technology, Directed Energy Test, EW Test, High-Speed Systems Test, UAS Test, Advanced Instrumentation Systems Technology, and Spectrum-Efficient Technology. The T&E/S&T Program also advances OSD STEM initiatives for the T&E community by involving academic institutions in projects initiated by response to broad agency announcements and by supporting intern activities within the TRMC and at DoD test ranges and facilities. Significant ongoing technology developments are described in the following paragraphs. Improving Hypersonic Propulsion Systems Testing. The High-Speed Systems Test Technology Area is developing a prototype hypersonic aeropropulsion clean air test bed to replicate a realistic, variable Mach flight profile in a wind tunnel at the required temperatures and DASD(DT&E) FY 2016 Annual Report 69

76 DoD TRMC energies. The project is in the second phase of development. This phase is demonstrating the capability to mix clean air from different sources and then deliver that air into a test engine with the energy that the system would experience traveling at speeds between Mach 4.5 and 7.5. This technology will advance DoD efforts to reduce developmental and acquisition risks of high-speed strike weapons by adequately testing scramjet engine performance and operability in ground test facilities. Improving Torpedo Performance Testing. The C4I and Software-Intensive Systems Test Technology Area effort is developing prototype algorithms for the U.S. Navy Weapons Analysis Facility to predict torpedo performance. This project addresses propagation models, which are computationally too slow to support real-time simulations of torpedoes in environments in which acoustic propagation is range dependent, and the lack of methodologies and acceptability criteria to support verification, validation, and accreditation of the torpedo simulation. Improving IR Systems Testing. The EW Test Technology Area is developing a realistic, highresolution, IR two-color scene projector capable of emulating hot objects rapidly traversing a realistic background. This technology will provide the ability to project the IR scene of an incoming threat into an aircraft sensor and enable realistic dynamic testing of two-color missile warning systems and directed IRCM systems. This development is complemented by development of a high-frame-rate, large-format, broadband scene projector that will have broad applicability to IRCM system testing as well as testing of advanced IR guided munitions seekers. Expanding the Test Opportunities for HELs. The Directed Energy Test Technology Area developed a prototype light detection and ranging system to characterize the atmosphere on HEL slant propagation paths. This system will provide range-resolved refractive turbulence profiles, water vapor density, and extinction profiles due to aerosols. This technology will provide the ability to understand how atmospheric effects distort HEL beam propagation. The project developed land- and ship-based prototypes. Neutron Dense Plasma Focus. The Directed Energy Test Technology Area is developing an enhanced capacitor bank and tube prototype in support of ultrashort neutron pulse test capabilities and replacement of the White Sands Missile Range (WSMR), New Mexico, Fast Burst Reactor. This technology will provide the ability for higher fluence output to support nuclear survivability testing and reduce the overhead costs of using real nuclear material for testing. Improving Extended-Range Weapons Testing. The Spectrum-Efficient Test Technology Area is prototyping a beam-forming, phased-array telemetry antenna system suitable for mounting on an unmanned aircraft to support over-the-horizon test operations. Designed to support extendedrange tests, this prototype will augment the sea-based telemetry network and reduce the risk of telemetry dropouts in key phases of large-footprint weapons tests across the oceans. Improving Sanitization of Cyber Test Environments. The Cyberspace Test Technology Area effort developed an automated sanitization framework of cyber-range components. This trusted, consistent sanitization approach will enable specialized assets to be shared among user communities that require access at varying levels of security without risk of compromising classified data or artifacts at the NCR. Some of the sanitization modules will be used in the NCR, and the sanitization framework is being implemented at the Joint Warfare Analysis Center in Dahlgren, Virginia. 70 DASD(DT&E) FY 2016 Annual Report

77 DoD TRMC Improving Behavior Prediction for Autonomous System Testing. The UAS Test Technology Area is developing a stress-testing tool for UAS software that reveals behavior performance failures within the system. This tool will improve test safety by automatically learning rules for activating vulnerabilities deep within a UAS network, thereby increasing the thoroughness and efficiency of vulnerability detection. During development, the software test tool was used to support the testing of an autonomous system technology demonstrator. When development is complete, the prototype will transition into Government system integration laboratories to support the testing of next-generation autonomous systems. Improving Live-Fire Measurements for Arena Testing. The Advanced Instrumentation Systems Technology area is developing a passive imaging technology to derive size, shape, mass, drag coefficients, and velocity vectors for individual fragments to quickly characterize fragments during warhead testing. This technology will sense fragments at high spatial resolution and fast temporal resolution to improve warhead characterization, which is critical to the development of next-generation weapons. Moreover, the system will significantly reduce labor and material costs for test setup and fragment recovery associated with traditional arena testing. All of the above Test Technology Areas are described more fully in a separate T&E/S&T Program Annual Report, which is provided to stakeholders and other interested parties. 5.5 Central Test and Evaluation Investment Program (CTEIP) CTEIP provides an enterprise approach for DoD investments in T&E capabilities that meet multi- Service and Defense Agency test requirements. The major portion of CTEIP funding is devoted to JIM projects that address critical, leading-edge capabilities needed to support the T&E of increasingly complex and sophisticated weapons, sensor, and C2 systems. JIM projects are nominated by T&E EAs on behalf of their respective Service or Defense Agency. CTEIP also funds Resource Enhancement Program (REP) projects, which address high-priority, near-term OT needs. REP projects are nominated by the Service or Defense Agency OT commands and are approved and prioritized by the DOT&E. Additionally, CTEIP funds threat simulator development efforts through the Threat Systems Program (TSP). In total, CTEIP funds 60 to 70 projects per year ranging from studies of test technologies to full-scale developments. During FY 2016, 15 JIM, REP, and TSP projects were successfully completed, and 46 projects continued in execution. A complete review of all 2016 CTEIP projects will be published in the 2016 CTEIP Annual Report. Projects that will be initiated during FY 2017 are currently being evaluated. The following is a list of the major enterprise investment areas. EW Testing. Several OSD studies have identified gaps in DoD s ability to test EW capabilities. Programs of record need to verify and validate U.S. and allied EW system performance against dense and diverse RF threat system environments. In response to this need, CTEIP established the Electronic Warfare Infrastructure Improvement Project (EWIIP) as a portfolio that will develop advanced ISTF and OAR threat simulation capabilities. The major OAR components of EWIIP are the OSD-led Radar Signal Emulator (RSE) project, which develops 16 high-power, reprogrammable, relocatable RF emitters, and the Navy-led Closed-Loop Passive Electronically Scanned Array (PESA) Simulator (CLPS) project, which fields two relocatable, closed-loop DASD(DT&E) FY 2016 Annual Report 71

78 DoD TRMC surface-to-air missile simulators. RSEs are now beginning to be integrated onto the Nevada Test and Training Range (NTTR) and are expected to complete production in March CLPS units are in development and are expected to be integrated onto the NTTR in late spring The major ISTF EWIIP projects are the Next-Generation Electronic Warfare Environment Generator (NEWEG) project and the Advanced Dynamic Transmit Array (ADTRA) project. The NEWEG project develops a high-fidelity EW environment generator that enhances three Navy and Air Force ISTFs and establishes commonality among DoD RF stimulators. The initial instantiation of NEWEG is expected to be fielded by December The ADTRA project develops a free-space RF transmission capability that will serve as the amplification/antenna subsystem for the next-generation signal generator at the Benefield Anechoic Facility (BAF). The ADTRA system is forecasted for fielding in September 2020 and will be interoperable with the Combat Electromagnetic Environment Simulator (CEESIM), the NEWEG, and the CEESIM I&M Program Life Cycle Extension. Other key EW investments include the REP-funded Digital Integrated Air Defense System (DIADS) Weapons Control and DIADS Sensor Reactivity Upgrade both completed during FY 2016 and the TSP-funded Integrated Air Defense System for OT. In addition, CTEIP completed the B-2 Defensive Management System (DMS) Infrastructure upgrade project at the BAF. The B-2 DMS upgrade included infrastructure upgrades and threat signal development that support the first DT of the B-2 in an ISTF environment. Net-Centric and Cyber Warfare Testing. Cyber/net-centric operations are a critical enabler for operations in air, land, maritime, and space domains. During FY 2016, CTEIP continued execution of JIM projects supporting critical cyber and net-centric enablers. The Network- Centric Weapons T&E Environment project is developing a distributed capability to assess netcentric weapons SoS performance (e.g., SDB II). The Cyber Test Analysis and Simulation Environment project will expand cybersecurity testing analysis capabilities and M&S tools. The Multi-Level Secure Joint/Coalition Network Environment project will provide a persistent, multi-level secure data management capability on the JMETC network for the DoD RDT&E community. Survivability and Vulnerability Testing. CTEIP continues to modernize DoD T&E capabilities to protect DoD s strategic warfare systems from the damaging effects of electromagnetic pulse (EMP) and high-power microwave (HPM) threats. The ongoing Vertical EMP Simulator JIM project is developing a vertical EMP test capability at two test facilities and a narrowband HPM test capability for conducting aircraft intersystem electromagnetic vulnerability testing. The vertical EMP capability at the WSMR facility achieved IOC in December CTEIP also started the Pulsed Neutron Environment project to assess the vulnerability of strategic weapons components to pulsed neutrons experienced during nuclear explosions. The system will use a new low-enriched uranium design, which increases neutron density (fluence), and will lower security costs of current aging capabilities at WSMR. CTEIP has also started the Radar Cross Section (RCS) Range Relevance project to improve RCS measurement capability and capacity at the Air Force National Radar Cross Section Test Facility, Hollman AFB, New Mexico, and the Navy s Atlantic Test Range, Naval Air Station Patuxent River, Maryland. These upgrades will increase spectral coverage and improve system sensitivity, dynamic range, and clutter management to improve DoD s capability to measure system RCS performance and vulnerability in developmental and operational environments. 72 DASD(DT&E) FY 2016 Annual Report

79 DoD TRMC Hypersonic Systems Testing. A number of recent congressional and DoD-directed studies have identified shortfalls in ground and open-air capabilities to test the new class of hypersonic systems, which includes hypersonic cruise missiles and boost-glide vehicles. These studies have resulted in a DoD-approved multiyear roadmap of investments that will be executed under CTEIP starting in FY Major investments include (1) a new facility to ground test the air vehicle and engine aeropropulsion performance between 4 and 7.5 Mach in a clean air environment, in support of the T&E/S&T phase II effort to prototype the capability discussed in section 5.4; (2) an upgraded H2 hypersonic test facility at the Arnold Engineering Development Center (AEDC) to enable materials testing of larger leading-edge components (e.g., nose cones, wing elements) in representative hypersonic environments; (3) an upgrade of DoD s Tunnel 9 high Mach facility in White Oak, Maryland, to provide aerodynamic and aerothermal data at hypersonic speeds up to Mach 18; (4) a suite of improvements to the Holloman AFB High-Speed Test Track and AEDC s G-Range to test erosion performance of leading components in dust, rain, and snow/ice; and (5) a new laser-based atmospheric conditions measurement system to reduce flight test evaluation uncertainty and improve launch and recovery operations. Spectrum-Efficient Telemetry. The availability of RF spectrum to support DoD test requirements is becoming more restrictive as systems under test require more bandwidth because of increased test data requirements. As a result, CTEIP is improving DoD telemetry systems to add flexibility for real-time management of test data and instrumentation during missions as well as to use the newly available C-band frequencies. The ongoing integrated Network Enhanced Telemetry JIM project will enhance current one-way serial streaming telemetry with a two-way C-band network radio capability that provides real-time management of aircraft test data and instrumentation. Similarly, the Commercial Derived Aircraft-Based Instrumentation Telemetry System project will provide long-range, airborne range control; range safety and flight termination services; and improved airborne telemetry for open-ocean testing worldwide. Spectrum efficiency is achieved through the use of phased-array antennas providing a five-fold increase in the number of systems tracked. High-Accuracy Time-Space-Position Information (TSPI). The accuracy of advanced guidance and navigation systems in high-performance aircraft and advanced precision munitions has equaled or surpassed the TSPI capability of current test instrumentation. CTEIP is improving DoD ability to more accurately measure a test item s location and phenomenology while in flight. The Common Range Integrated Instrumentation System JIM project, which will achieve IOC at ranges in FY 2017/FY 2018, will replace the aging Advanced Range Data System and provide ranges with the capability to collect highly accurate TSPI (i.e., less than 1 meter). The Advanced Range Tracking and Imaging System JIM project will improve optical tracking capability to observe and record performance (including TSPI) of aircraft or surface-launched missiles and munitions. Aircraft Survivability. The sophistication and technology of surface-to-air missiles and air defense weapons, as well as ground fire systems, continue to be a significant threat to aircraft. The ongoing Joint Distributed IRCM Ground-Test System (JDIGS) JIM project enables highfidelity, low-cost ground testing of installed missile warning systems and IRCM systems. The test capabilities already delivered by JDIGS have supported Navy ISTF missile warning systems and control processor testing and supported Air Force IRCM testing. Additionally, the completed Multi-Spectral Sea and Land Target Simulator (MSALTS) REP project provided portable, mobile open-air missile plume simulators to test IRCM systems against land- and sea- DASD(DT&E) FY 2016 Annual Report 73

80 DoD TRMC based threats. The follow-on MSALTS Ultraviolet Emitter Enhancement project further improves missile plume emulation. The ongoing Joint Standard Instrumentation Suite REP project measures and collects signature, TSPI, and related data of threat missile and hostile fire munitions firings. Unmanned Autonomous System (UAS) Testing. In 2015, CTEIP completed the Unmanned Aircraft Systems Mission Environment project that provided test capability for testing and evaluating unmanned aircraft systems in ISTF and open-air environments. In 2016, CTEIP initiated the Swarm Autonomy and Scoring project that upgrades the Navy s existing High-Speed Maneuverable Surface Targets (HSMSTs) with semiautonomous control, develops a real-time casualty assessment capability for 30-millimeter and 57-millimeter naval guns against HSMST swarms, and provides improved UAS overhead scoring capabilities for gun- and aerial-delivered weapons impact accuracy and weapons effects testing against representative surface swarming threats. Threat Simulators. Under the REP and TSP projects, CTEIP has a number of ongoing threat simulator projects to provide the threat realism needed for U.S. systems to test against. For example, REP is developing the following threat simulators: (1) the Airborne Early-Warning Interoperability Simulator to emulate an operationally representative target environment in ISTFs; (2) the Boosted Zombie Target, which develops an economical two-stage target for Patriot Advanced Capability (PAC)-3 missile testing; (3) the Advanced Mine Simulation System Upgrade that measures the susceptibility of U.S. Navy platforms against simulated mine threats and adds new mine triggering emulations; and (4) the Submarine-Launched Modular 3-inch Device, which develops threat surrogate submarine-launched countermeasures. The TSP is completing the Advanced Threat Signal Injection Jammer project, which upgrades injection jammer assets to support increased waveform generation and upper band capabilities. TSP is also developing the Airborne Jammer Standoff Jamming system that provides maximum GPS jamming power on the system under test for extended periods of time. 5.6 Joint Mission Environment Test Capability (JMETC) Program The JMETC Program continued to serve as the DoD corporate infrastructure for linking distributed facilities and enabling users to test and evaluate systems and SoS warfighting capabilities in a joint context while realizing significant reduction in cost, schedule, and risk. Currently in the tenth year since inception, the JMETC Program has provided the T&E community with an infrastructure that supports testing across the full spectrum of the acquisition life cycle with a focus on distributed interoperability and cybersecurity test and training. In addition to network connectivity, JMETC also provides tools, services, and subject matter expertise to aid in the planning and execution of distributed test and training events. Some of the key FY 2016 JMETC achievements are summarized below. Expanded the distributed infrastructure over the JMETC Secret Network (JSN) and the JMETC Multiple Independent Levels of Security (MILS) Network (JMN) to 115 functional and planned sites at Government and industry locations. 74 DASD(DT&E) FY 2016 Annual Report

81 DoD TRMC Supported 70 distributed LVC events, primarily focused on interoperability and cybersecurity testing, training, and experimentation. Continued to oversee the JSN to support distributed LVC testing at the Secret classification level with a focus on interoperability T&E. JMETC continued to provide network infrastructure; distributed test planning, execution, and analysis tools; remote collaboration services; and distributed testing expertise to numerous events conducted on the JSN. Also, during FY 2016, the monitoring and troubleshooting equipment located at each JSN site was upgraded to improve performance and enhance its security posture. Continued the growth and maturity of the JMN, providing an infrastructure capable of supporting the concurrent execution of multiple events at varying classifications up to and including Top Secret (TS)/Sensitive Compartmented Information (SCI)/Special Access Program/Special Access Required, with a focus on the support of interoperability T&E at higher classifications and cybersecurity test and training requirements. JMN connectivity was expanded in FY 2016 to 14 new locations; the infrastructure supported 10 concurrent events per week on average, with a peak of 14 events. User demand continues to grow, with 23 events already requested for execution in FY 2017 and a forecast of significantly more. Continued development and deployment of Regional Service Delivery Points (RSDPs), which provide increased, low-cost, cyber range capacity for the execution of cyber test and training activities in virtualized, high-fidelity representations of cyber-contested environments. In FY 2016, significant improvements were made to the RSDP tool suite to increase efficiencies during environment deployments and sanitization, the fidelity of red-blue-gray environments, traffic generation, and data collection. Additionally, two more RSDPs were fielded, bringing the total to four with one more procured and awaiting final facility accreditation for installation in early FY Led and participated in the execution of several cyber tabletop events focused on identifying potential vulnerabilities using a lightweight, low-cost, intellectually intensive exercise to explore the effects of adversarial cyberspace operations on a system, SoS, or family of systems during the execution of a specific mission. This methodology was successfully applied to multiple acquisition programs in FY 2016 and is now also being applied to identify potential vulnerabilities in T&E infrastructure at MRTFBs under a pilot project led by JMETC. 5.7 EA for Cyber Test Ranges The Deputy Secretary of Defense designated the Director, TRMC as the EA for Cyber Test Ranges on March 8, In that new capacity, the TRMC assumed oversight of designated cyber test infrastructure and responsibility for related architectures, standards, priorities, and investments to meet DoD requirements. The Deputy Secretary of Defense designated the Secretary of the Army as the EA for Cyber Training Ranges. In this capacity, the Secretary of the Army assumed oversight of designated cyber training infrastructure and responsibility for related architectures, standards, priorities, and investments. The respective EAs for Cyber Test and Training Ranges will work closely together to garner efficiencies and ensure consistent standards. EA roles and responsibilities will be further developed and promulgated in a forthcoming DoD Directive. DASD(DT&E) FY 2016 Annual Report 75

82 5.8 National Cyber Range (NCR) DoD TRMC In FY 2016, the NCR supported 58 events for MDAPs, training, and operational exercises, as shown in Figure 5-1. This number represents a significant increase over FY 2015, in part because the NCR was offline for 5 weeks to support assessment and accreditation (A&A) and infrastructure recapitalization. The NCR is unique in that it can simultaneously execute up to four independent tests at differing classifications ranging from unclassified to TS/SCI on cryptographically isolated test beds. The NCR is fully accredited to operate at the TS/SCI level based upon Intelligence Community Directive 503 certification and accreditation requirements. In June 2016, the NCR added two additional test beds to support operational contingencies and a large-scale cyber exercise, increasing the number of concurrently operating test beds to six. The NCR has the ability to deploy cyberspace environments with the scale and fidelity to realistically portray cyberspace threats (e.g., malware, distributed denial-of-service attacks, and cross-site scripting) and support a full spectrum of cyberspace testing, operations, and training events. Throughout 2016, the NCR consistently provided the highest caliber of support and satisfied a wide range of customer requirements. Services provided to NCR customers included architectural analysis, product evaluations, system and target emulation, risk reduction activities, R&D testing, SoS testing, security architecture evaluation, HWIL testing, and malware and forensic analysis. NCR FY 2016 Events 3.6% 14.5% 30.9% MDAP Cybersecurity DT&E % Mission Rehersal - 9.1% Training/Exercises % 41.8% 9.1% Cyberspace Capability OT&E - 3.6% Cyberspace Capability DT&E % Figure 5-1. NCR FY 2016 Events In FY 2016, in support of training exercises, the NCR demonstrated the capability to rapidly configure complex network topologies varying in scale from hundreds to tens of thousands of highfidelity computing assets, interconnecting more than 50 logical ranges, emulating hundreds of enclaves in support of thousands of users. The diverse cyberspace environments emulated dozens of operating system variants, unique wireless assets and computing hardware, and representative websites. The NCR provided realistic emulations of Nonsecure Internet Protocol Router Network (NIPRNET) and SIPRNET enclaves hosting Web and servers and clients. It also provided high-fidelity representations of public internet infrastructure with thousands of websites. In addition 76 DASD(DT&E) FY 2016 Annual Report

83 DoD TRMC to generating robust cyberspace environments, the NCR provides a world-class Cybersecurity Evaluation Team (CET) to assist customers in planning and executing cybersecurity T&E and training events. The CET works closely with customers to understand findings and alternative mitigation strategies. In turn, NCR customer knowledge is greatly enhanced as knowledge is shared by the CET. The NCR is a dual-use capability that supports operational, training, and acquisition customers. Recently conducted CBAs predict that customer requirements will rapidly exceed current NCR capacity. DoD is developing a plan to increase capacity to support acquisition (test), training, and operations cyber range capability to satisfy increasing demand. Following the currently ongoing recapitalization and A&A, the NCR will be able to support up to eight test beds, with computing assets able to scale up to approximately 250,000 lightweight virtual machines with storage capacity to approximately 600 terabytes. By 2019, the TRMC anticipates expanding capacity to provision about 40 test beds capable of supporting 400 to 500 events per year distributed across four or more locations. 5.9 Range Sustainability Encroachment issues continue to slowly erode the capabilities of test range infrastructure and consume resources. In 2016, the TRMC conducted the fourth biennial Test Infrastructure Encroachment Survey. Data from the survey, annual Infrastructure Range Reviews, and field visits to the ranges reaffirmed that the top three factors affecting testing are adjacent land use, airspace limitations, and spectrum encroachment Adjacent Land Use The majority of adjacent land use challenges stem from energy development. In 2016, there was a 35 percent increase in the number of energy projects that TRMC reviewed. In addition, the TRMC participated in the DoD mission compatibility assessment of the Bureau of Ocean Energy Management s Outer Continental Shelf Oil and Gas Leasing Draft Proposed Program; provided functional oversight in defense against a proposed wind farm near the NTTR; and is currently examining Western U.S. transmission line projects and potential wind development off the California coast all of which threaten to impact DoD test capabilities. To avoid areas of potential conflict with military operations and readiness and to help energy developers early in their siting decision process, the TRMC worked with the DoD Siting Clearinghouse to publish Risk of Adverse Impact on Military Operations and Readiness Area maps for areas around the NTTR, Naval Air Weapons Station China Lake, and Edwards AFB. A growing adjacent land use concern is foreign encroachment in which foreign entities acquire access or land control, or otherwise conduct business near test ranges, increasing the potential for persistent surveillance of sensitive DoD test activities. To address this concern, the TRMC is supporting a USD(P&R) legislative initiative aimed at identifying foreign transactions with other Federal landholding agencies and increasing involvement in the DoD process for participation in the Committee on Foreign Investment in the United States. DASD(DT&E) FY 2016 Annual Report 77

84 DoD TRMC Airspace Limitations Airspace limitations are being exacerbated by the proliferation of small unmanned aircraft systems (suas). These suas can impact DoD s ability to use already constrained test airspace by posing safety and security concerns to test activities. As a result, the TRMC is actively involved in DoDwide and interagency forums addressing policies that will assist in managing the threat posed by suas Spectrum Encroachment The recent sell-off of the megahertz band to the International Mobile Telecommunications services, as a result of the Advanced Wireless Services-3 auction, has created additional challenges for T&E spectrum operations. Through its ongoing spectrum stewardship program, the TRMC remains actively engaged with the DoD CIO in numerous activities to ensure access to the RF spectrum at DoD test ranges. Elements of TRMC s spectrum stewardship program include the RF Spectrum for T&E Common Operating Picture and the T2 EMS Usage Study, previously mentioned in this report. In addition, the TRMC continues to pursue 18 technology solutions that the Services need at test ranges to fully implement the use of three frequency bands ( megahertz, megahertz, and megahertz) allocated for aeronautical mobile telemetry services at the 2007 World Radiocommunication Conference. The TMRC also partners with the DoD CIO by participating in the international regulatory process to protect the bands used by DoD test ranges for weapon systems testing from spectrum encroachment caused by changes to international radio regulations. Additionally, the TRMC provides leadership in the DoD Spectrum Access Research and Development Program (SAR&DP), which the DoD CIO established with the Office of Management and Budget to advance R&D in support of DoD spectrum-sharing and relocation efforts using Spectrum Relocation Fund proceeds. The TRMC chaired the SAR&DP T&E Assessment Working Group, which evaluated and prioritized more than 40 spectrum-sharing project proposals that were submitted through the SAR&DP process MRTFB T&E Workforce MRTFB T&E Workforce Assessment The TRMC is assessing a second consecutive year of MRTFB T&E workforce data. Although trends are not yet visible, the TRMC has identified the following as pervasive challenges across the MRTFB: time to hire, attrition rates, aging workforce, and future skills needs. Conversely, through the Infrastructure Readiness Review process, the TRMC identified several best practices that individual MRTFB activities have developed to address these challenges. 78 DASD(DT&E) FY 2016 Annual Report

85 DoD TRMC Workforce Outreach The TRMC continues to interact with undergraduate and graduate students pursuing technical T&Erelated degrees at historically black colleges and universities and minority-serving institutions and to provide the students with exposure to career opportunities in T&E. Additionally, in 2015, the TRMC initiated an Adopt-a-School volunteer program with Barcroft Elementary School in Arlington, Virginia. The TRMC and DASD(DT&E) volunteers served as mentors to improve boys capabilities and confidence in reading through the Cool Boys Book Club and provided creative and fun ways for girls to learn about technology and gain confidence through the Girls for Engineering, Math, and Science program. To facilitate the TRMC priority of being a strong advocate for the T&E workforce, the TRMC will strengthen its workforce initiatives through community outreach and by collaborating with the Assistant Secretary of Defense for Research and Engineering on workforce initiatives Budget Certification and Trends The TRMC was established in the FY 2003 NDAA to review and provide oversight of proposed Department of Defense budgets and expenditures for the test and evaluation facilities and resources of the Major Range and Test Facility Base of the Department of Defense; and all other test and evaluation facilities and resources. In FY 2016, the TRMC produced a Budget Certification Report (BCR) containing the Director s analysis of the major FY 2017 T&E budget submissions as well as the Director s certification that the budgets are adequate to support the MRTFB mission. The TRMC also began analysis of the major FY 2018 budget submissions. The BCR satisfied the reporting requirements of section 196(e)(2) of Title 10, U.S.C., for assessment of the MRTFB and designated non-mrtfb T&E capabilities. Analysis of the major FY 2017 T&E budget submissions, Budget Exhibit data submitted by the Services, and documentation provided by the Services in support of the President s Budget submissions highlighted the following T&E resource summary trends. Note that all funding analyses are in constant FY 2016 dollars. Because the FY 2003 NDAA designated FY 2006 as the year for full implementation of its provisions, FY 2006 serves as the point of departure for most of the summary analyses that follow, which cover the FY 2006 FY 2018 time frame, with few exceptions. The number of RDT&E programs conducting testing has remained high throughout the FY 2006 FY 2016 time frame with continued growth forecasted. Total spending to perform T&E at MRTFB activities peaked at around $4.3 billion in the FY 2008 FY 2011 time frame and has since declined. o Most of this decline has occurred in institutional funding for costs not directly charged to customers. This funding was relatively flat through FY 2012 but has declined consistently since then and is now about 20 percent below the FY 2006 level. o Customer testing increased about 10 percent from FY 2006 to the FY 2009 FY 2012 time frame but has since returned to roughly the FY 2006 level. DASD(DT&E) FY 2016 Annual Report 79

86 DoD TRMC The MRTFB workforce has shown a steady decline since FY If current trends continue, projections show that the MRTFB workforce will be almost 20 percent smaller in FY 2018 than in FY Among the Services, the overall reductions have been most pronounced in the Army and least pronounced in the Navy. Modernization funding, as measured by facility investment and military construction funding, has declined over this period. o Spending averaged more than $600 million per year though FY 2011; it has averaged less than 75 percent of that level since that time. o Military construction funding has shown the greatest decline. Construction funding for T&E facilities has almost disappeared since FY 2009, with only FY 2012 and FY 2017 showing substantial construction investment. The T&E enterprise continues to face budget, policy, environmental, modernization, workforce, and new test technology development challenges. These demands place intense pressure on the community to ensure that T&E capability is in place when and where it is needed. The TRMC has no evidence that the general downtrends in test resources have adversely affected necessary T&E, although a small number of anecdotal reports suggest that some programs could not be supported when they desired. However, the funding trends, especially in investment and construction accounts, are concerning because the ability of the MRTFB to support the testing of high-technology weapons of the future could be adversely affected. As the Services struggle with maintaining and modernizing existing capability, new T&E capability investment must continue in order to stay ahead of emerging technologies and the needs of acquisition programs International Facilities The TRMC recognizes that is it important to not only work within the Federal Government to meet the challenges of shrinking resources but also look beyond U.S. borders to solicit cooperation and exchanges with U.S. allies. In 2016, the TRMC participated in exchanges with Australia and the United Kingdom (UK). The TRMC held discussions with the Ministry of Defence in the UK to better understand its strategy of using a government-owned, contractor-operated scenario for management of its test ranges. The exchange was open and forthright, providing information on the positive and negative aspects of such an arrangement. Australian representatives visited the United States to study the TRMC and the role it plays in maintaining test capability, and the TRMC in turn learned more about the Australian ranges. Overall, the TRMC recognizes that solutions in the future may come from the global community, and it is exploring those possibilities. 80 DASD(DT&E) FY 2016 Annual Report

87 6 PROGRAM ENGAGEMENT AND ASSESSMENTS The FY 2016 Annual Report highlights the engagement activities and assessments of 40 programs (MDAPs, MAIS programs, USD(AT&L)-designated special interest programs) that have reached a significant milestone or had significant DT&E activities. Significant activities include DASD(DT&E) program assessments, first test flight, completed system integration lab testing, completed ground testing, and dedicated Government DT&E. For those programs that received a DASD(DT&E) program assessment during the fiscal year, a separate paragraph highlighting the findings and recommendations of that assessment is included. Of the programs monitored by DASD(DT&E) during FY 2016, including the 40 programs reported in this annual report, none requested a waiver or deviation from requirements in the TEMP in FY Assessments are as of the end of FY 2016 (September 30, 2016); however, some assessments may include information on program status beyond that date. DASD(DT&E) FY 2016 Annual Report 81

88 This page intentionally blank. 82 DASD(DT&E) FY 2016 Annual Report

89 6.1 DoD Programs This section includes summaries of the following 8 programs: Assembled Chemical Weapons Alternatives (ACWA) Ballistic Missile Defense System (BMDS) Defense Agencies Initiative (DAI) Increment 2 Department of Defense Healthcare Management System Modernization (DHMSM) F-35 Lightning II Joint Light Tactical Vehicle (JLTV) Key Management Infrastructure (KMI) Increment 2 Public Key Infrastructure (PKI) Increment 2 DASD(DT&E) FY 2016 Annual Report 83

90 Assembled Chemical Weapons Alternatives (ACWA) Executive Summary: The ACWA program is responsible for managing the destruction of the last U.S. chemical weapons stockpile in support of the congressionally mandated Chemical Demilitarization Program to eliminate all chemical warfare and related materiel. DoD selected two systems contractor (SC) teams to design, build, systemize, pilot test, operate, and close the two program destruction plants: the Bechtel Parsons Blue Grass joint venture for the Blue Grass Chemical Agent-Destruction Pilot Plant (BGCAPP) at the Blue Grass Army Depot, Kentucky, and the Bechtel Pueblo Team for the Pueblo Chemical Agent-Destruction Pilot Plant (PCAPP) at the Pueblo Chemical Depot, Colorado. The plants are first-of-a-kind facilities designed to destroy chemical agents using low-temperature, low-pressure neutralization processes. Both plants selected explosive destruction technology (EDT) systems to safely destroy mustard chemical munitions that cannot be easily processed through the main plants. Lead DT&E Organization: AMSAA Summary of FY 2016 DT&E Activities October 1, 2015 September 6, 2016, the PCAPP SC team completed systemization activities at the main plant to start the 155-millimeter mustard munitions campaign. October 1, 2015 September 6, 2016, AMSAA evaluated the PCAPP main plant site setup, systemization, and training activities in preparation for live-agent operations; verified the installation of the system per design; and verified that the equipment was operational and the system met the minimum acceptance criteria (MAC). October 1, 2015 September 6, 2016, the PCAPP SC team continued to conduct an operational readiness review (ORR). The ORR is a comprehensive assessment to verify personnel proficiency and that processes, procedures, support structures, and management oversight meet operational requirements and comply with Federal and Colorado State regulations to start OT pilot testing. October 1, 2015 September 30, 2016, the SC team conducted systemization activities at the BGCAPP main plant. July 19 22, 2016, the SC team conducted a BGCAPP laboratory information system (LIS) cybersecurity self-assessment of the LIS security controls to obtain the vulnerability and configuration compliance scan of the LIS network. Program office, DASD(DT&E), and AMSAA cybersecurity representatives participated. August 13 22, 2016, the PCAPP SC team conducted the main plant integrated operations demonstration (IOD) as a part of the ORR to demonstrate that the entire plant functions as designed and meets the established MAC. Program office, DASD(DT&E), and AMSAA representatives witnessed the demonstration. September 27, 2016, the SC team conducted the BGCAPP static detonation chamber (SDC) systemization DT readiness review to assess the project s readiness to begin SDC systemization DT. The SDC is BGCAPP s EDT system. The Chemical Demilitarization Test Integrated Product Team (TIPT) participated. 84 DASD(DT&E) FY 2016 Annual Report

91 DoD ACWA Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) provided four Defense Acquisition Executive Summary assessments to the USD(AT&L) in FY 2016; there were no potential or actual significant DT program issues to report. DASD(DT&E) approved the DT&E plan within the BGCAPP Test Concept Plan on January 11, 2016; this update addressed the BGCAPP EDT facility and enhanced cybersecurity T&E planning. The PCAPP main plant met all required entrance criteria to start agent destruction operations; the main plant started operations on September 7, The BGCAPP SDC is ready to enter systemization DT. The Chemical Demilitarization TIPT recommended that the PEO ACWA enter the DT systemization phase of the SDC T&E program. DASD(DT&E) Program Assessment DASD(DT&E) conducted a DT&E assessment of the PCAPP main plant in July 2016 to support the start of agent operations based upon contingent requirements. The summary of the DASD(DT&E) evaluation follows: o Performance. The PCAPP facility contractor conducted system and key support process demonstrations to verify that the plant equipment operates as designed in an operational environment. The demonstrations confirm that the operational characteristics of the facility meet design specifications and document the operational characteristics of the plant equipment. DASD(DT&E) supported the start of agent operations contingent upon the main plant meeting the IOD MAC, with all Category 1 findings closed out and a clear path forward to correct Category 2 findings that includes a resolution date, and verification that PCAPP closed out remaining Category 1 pre-operations survey findings. o Cybersecurity. Based on cybersecurity test results, DASD(DT&E) assessed that plant start-up using live chemical agents is low risk. DASD(DT&E) supported the start of agent operations contingent upon the ACWA program closing out high-priority vulnerabilities discovered during the cooperative vulnerability and penetration assessment and the cybersecurity adversarial assessment. o Interoperability. The main plant does not interact with the DoDIN; therefore, there is not a requirement to obtain a joint interoperability certification. DASD(DT&E) assessed that PCAPP met all contingent verification requirements, and the main plant started destruction operations on September 7, As of October 1, 2016, the main plant had destroyed 1, millimeter mustard munitions. Conclusion: PCAPP met all requirements to start live-agent 155-millimeter mustard munitions operations and is currently conducting destruction operations. The BGCAPP SDC is ready to enter systemization DT. DASD(DT&E) FY 2016 Annual Report 85

92 Ballistic Missile Defense System (BMDS) Executive Summary: The BMDS is intended to counter ballistic missiles of all ranges short, medium, intermediate, and intercontinental. The BMDS is an integrated, layered architecture that provides multiple opportunities to destroy missiles and their warheads before they can reach their targets. The system includes networked overhead persistent infrared sensors and ground- and seabased radars for target detection and tracking, and ground- and sea-based interceptor missiles for destroying ballistic missiles. These elements are coupled via a command and control, battle management, and communications (C2BMC) system that networks, integrates, and synchronizes missile defense systems operations, providing the Warfighter with the needed links between the sensors and weapon systems. MDA uses an incremental approach to delivering BMDS capabilities and is planning to deliver increments of BMDS capability annually or biannually that simultaneously meet capability needs for a Phased Adaptive Approach to regional defense and homeland defense. Each increment includes multiple BMDS capabilities that are new or improvements to existing capabilities. BMDS Increment 3 focused on near-term discrimination improvements for homeland defense and Increment 4 will continue to focus on homeland defense capability improvements through Increment 6, scheduled for completion in 2020, includes the Redesigned Kill Vehicle, Long-Range Discrimination Radar (LRDR), and discrimination improvements. The European Phased Adaptive Approach (EPAA) is the U.S. contribution to NATO BMDS, designed to protect U.S. deployed forces and NATO allies in Europe from ballistic missile attacks from the Middle East. EPAA was announced by the President in 2009 and Phase I was deployed in 2011, providing coverage of NATO territory in southeastern Europe with the sea-based Aegis weapon system and Standard Missile (SM)-3 Block IA. MDA declared EPAA Phase II (Increment 2) technical capability with the Aegis Ashore in Romania in December 2015, with the capability to launch SM-3 Block IA and IB variants. EPAA Phase III (Increment 5) is scheduled for completion in December 2018 and is planned to include the deployment of a second Aegis Ashore site in Poland, upgrade of the Aegis Ballistic Missile Defense (BMD) weapon system, and delivery of the SM-3 Block IIA. Additionally, MDA plans to procure an additional 35 SM-3 Block IB missiles in FY MDA has fielded a BMDS consisting of 34 ground-based interceptors for long-range homeland defense, with 44 interceptors expected to be available by the end of There are 33 Aegis warships capable of long-range surveillance, tracking, and BMD using deployed SM-2, SM-3, and SM-6 missiles. There are six active Terminal High Altitude Area Defense batteries, as well as ground- and sea-based radars deployed around the world. DASD(DT&E) is focused on ensuring that the DT&E planned and conducted will fully inform capability fielding and production decisions for future systems. 86 DASD(DT&E) FY 2016 Annual Report

93 DoD BMDS Lead DT&E Organization: MDA Directorate for Test Summary of FY 2016 DT&E Activities September 30 October 15, 2015, MDA conducted BMDS distributed ground testing (GTD-06 Part 1) involving Aegis Ashore in Romania, Aegis Baseline 9.B1 (ashore) and Aegis Baseline 9.C1 (at sea) and the Army Navy/Transportable Radar Surveillance (AN/TPY-2) forward-based mode radar, and the C2BMC system. October 20, 2015, MDA conducted Aegis BMD flight test ASD-15. USS ROSS, an Aegis BMD configured destroyer, tracked and successfully engaged a short-range ballistic missile using an SM-3 Block IA guided missile, while USS THE SULLIVANS simultaneously tracked and engaged an air-breathing threat using an SM-2 Block III missile. December 7, 2015, MDA conducted SM-3 Cooperative Development (SCD) controlled test vehicle (CTV) performance flight test SCD CTV-02, using Aegis Baseline 9.C2 and the SM-3 Block IIA missile. December 8, 2015, MDA conducted Aegis Ashore CTV performance flight test AA CTV-02, using Aegis Baseline 9.B1 and the SM-3 Block IB Threat Upgrade (TU) missile. January 28, 2016, MDA conducted ground-based midcourse defense (GMD) CTV performance flight test CTV-02+, using a Capability Enhancement (CE)-II Exoatmospheric Kill Vehicle (EKV) with alternate divert thrusters, the GMD fire control loop, and multiple radars. February 3, 2016, MDA attempted Aegis BMD CTV performance flight test SM CTV-01: USS PAUL HAMILTON, an Aegis BMD configured destroyer, with an SM-3 Block IB TU missile with the third-stage rocket motor (TSRM) ECP modification. March 29 May 16, 2016, MDA conducted BMDS integrated HWIL ground testing (GTI-06 Part 2) involving GMD discrimination improvements. May 9, 2016, DASD(DT&E) approved the BMDS Integrated Master Test Plan (IMTP) The IMTP documents the MDA plan for conducting BMDS flight test events and ground test campaigns. May 17, 2016, MDA conducted Aegis Sea-Based Terminal (SBT) flight test FTX-21. USS JOHN PAUL JONES, an Aegis Baseline 9.C1-configured destroyer, detected and tracked a medium-range ballistic missile (MRBM) Type III target. May 25 and 26, 2016, Aegis BMD conducted CTV-01a/02. USS HOPPER, an Aegis BMD configured destroyer, conducted two flight tests of the SM-3 Block IB TU missiles with the TSRM ECP modification. June 16 22, 2016, MDA conducted integrated ground testing (GTI-ISR) involving Israeli systems, operational BMDS elements, and C2BMC. September 19 30, 2016, MDA conducted distributed ground testing (GTD-06 Part 2) to assess near-term discrimination improvements for homeland defense. Summary of FY 2016 DT&E Engagement and Assessments Distributed ground testing (GTD-06 Part 1) provided data to assess capabilities in multiple combatant commands areas of responsibility (AORs), EPAA Phase 2, and cross-aor T&E. Scenarios addressed U.S. Central Command and U.S. European Command capability against operationally realistic threats to enhance theater/regional BMD, including Aegis Ashore in Romania. Aegis BMD flight test, ASD-15, successfully demonstrated maritime theater missile defense coalition sea-based defense against simultaneous antiship and ballistic missile threats with coordinated C2, while operating in a defined maritime theater of operations. DASD(DT&E) FY 2016 Annual Report 87

94 DoD BMDS MDA conducted CTV performance flight test CTV-02+ to evaluate the new CE-II EKV alternate divert thruster design and assess end-to-end radar discrimination techniques. One of four alternate divert thrusters fired nominally on its first commanded burn but failed to fire on subsequent commands. MDA convened an Anomaly Investigation Team that attributed the failure to the legacy EKV flight computer electronics. The other three alternate divert thrusters performed nominally for all commanded burns. The test used a target modified to enhance tracking (not threat representative) and test specific radar software builds. These aspects limit what can be learned concerning system responses to realistic targets. Neither the discrimination algorithms nor scenario-specific changes to the tracking radar were fully tested and evaluated in ground testing before the flight test. Aegis SBT flight test, FTX-21, successfully demonstrated the ability of an Aegis Baseline 9.C1 (BMD 5.0 Capability Upgrade) configured ship to detect and track an MRBM target within the Earth s atmosphere. HWIL ground testing (GTI-06 Part 2) provided data to assess near-term and enhanced homeland defense capabilities. The assessment crossed U.S. Northern Command and U.S. Pacific Command (USPACOM) operational mission space and included a cybersecurity T&E period. Aegis BMD used ground and flight test events to successfully demonstrate TSRM ECP performance. Two successful flight tests of the SM-3 Block IB TU missile against dynamic test targets assessed minimum and maximum inter-pulse delay conditions. During ground and flight testing, several other failures occurred that delayed the SM-3 Block IB TU full production decision. Of primary concern are the changes to the missile design that are not directly associated with the TU improvements. Since the introduction of the SM-3 Block IA missile, flight test results indicate a downward trend of estimated missile reliability. Cooperative integrated ground testing (GTI-ISR) with Israel successfully demonstrated U.S. interoperability with Israeli Arrow and David s Sling Weapon Systems. The assessment addresses interoperability with C2BMC and the capability to produce and maintain a common integrated air picture. Distributed ground testing (GTD-06 Part 2) provided data to assess near-term discrimination improvements for homeland defense, the defense of USPACOM, and increased engagement capability for launch on remote. Because MDA is not required to prepare acquisition documentation required by DoDI , MDA does not prepare a BMDS-level TEMP. In lieu of a TEMP, MDA documents the test program in an IMTP. DASD(DT&E) is working with MDA to expand the BMDS IMTP to document the overall BMDS evaluation strategy. DASD(DT&E) Program Assessment: DASD(DT&E) provided a DT&E assessment concerning the GMD CTV that assessed the alternate divert thruster design for the CE-II EKV and end-to-end discrimination. DASD(DT&E) provided classified recommendations to MDA following the test. Conclusion: DASD(DT&E) is working with MDA to implement the use of DEFs for new programs and redesign activities. At this point, the LRDR BMDS-level evaluation plans lack measureable parameters that would drive test design. DASD(DT&E) will work with MDA to develop plans that support informed development and fielding decisions. DASD(DT&E) also recommends that MDA develop a homeland defense evaluation strategy to ensure that the test strategy is aligned with the capability increment fielding decisions. 88 DASD(DT&E) FY 2016 Annual Report

95 Defense Agencies Initiative (DAI) Increment 2 Executive Summary: The DAI program modernizes the Defense Agencies financial management processes by streamlining financial management capabilities, addressing financial reporting material weaknesses, and supporting financial statement auditability for the majority of Defense Agencies and DoD Field Activities. The Defense Logistics Agency (DLA) is deploying Increment 2 in four releases. The program plans full deployment of DAI for the 4th quarter FY Lead DT&E Organization: JITC Summary of FY 2016 DT&E Activities March 14 March 25, 2016, the PMO conducted a Government system integration test (SIT) of Release 2.1 to verify key technical and functional system characteristics of time and labor modifications to address the needs of three Defense Agencies joining DAI in FY DAI DT&E technical personnel executed the SIT and were augmented by functional community users at their respective agency s discretion and availability. March 28 April 8, 2016, the PMO conducted SIT of Release 2.2 of core capabilities that provide interfaces to DAI that comply with the Purchase Request Data Standard and the Procurement Data Standard as well as a suite of standardized governance, risk, and compliance reports and tools. April 18 June 16, 2016, the PMO conducted a contractor/government Release 3 development integration test to validate that the configuration done by the business processes yields the desired outcomes and that the reports, interfaces, conversions, extensions, forms, and workflow developed for the release work as an integrated part of the solution and perform as expected in a production-like environment. Budget formulation and direct disbursing to the Department of the Treasury were new capabilities added in Release 3. April 25 May 12, 2016, the PMO conducted a user acceptance test (UAT) of Release 2.1 to assess system performance in a representative end-user environment. The UAT was facilitated by the DAI Test Team and was performed primarily by functional community users. May 13 June 3, 2016, the PMO conducted UAT of Release 2.2. June 20 July 28, 2016, the PMO conducted SIT of Release 3. August 1 September 18, 2016, the PMO conducted UAT of Release 3. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) provided four Defense Acquisition Executive Summary assessments to the USD(AT&L) in FY 2016; there were no potential or actual DT-related program problems to report. DASD(DT&E) participated in SIT and UAT readiness reviews and production readiness reviews (PRRs) for Releases 2.1, 2.2, and 3 to ensure that entrance criteria were met. No unresolved DASD(DT&E) FY 2016 Annual Report 89

96 DoD DAI Severity 1 or Severity 2 defects and no clusters of Severity 3 defects were present at PRR for any of the releases. DASD(DT&E) assisted the program in revising the acquisition and test strategy for the Increment 2, Release 3 update of the DAI TEMP including the DEF. DASD(DT&E) provided a DT&E program assessment in support of the Release 3 limited fielding decision (LFD) and Release 4 development decision. The DAI program did not request a waiver or deviation from requirements in the TEMP. DASD(DT&E) Program Assessment DASD(DT&E) conducted a DT&E assessment of DAI Increment 2 in September 2016 to support the Release 3 LFD and Release 4 development decision. A summary of the evaluation follows: o Test Planning and Test Resources. System DT is conducted at DAI and agency sites. The DT environment includes a scaled production-like representative operating system for DT events. DASD(DT&E) approved the DT&E plan within the Release 3 TEMP on June 14, The test strategy therein describes the program s approach for planning and conducting integration, user acceptance, performance, continuity of operations (COOP), cybersecurity, regression, and sustainment testing. The program did not provide an updated TEMP for Release 4 in time to support the Release 4 development decision. This condition was assessed as low risk because of the level of Release 4 DT planning in the approved Release 3 TEMP and the standard testing practices within the program. o Performance. The program met exit criteria for an LFD for Release 3. Threshold criteria were met for all three KPPs and all eight KSAs when an approved work-around for the Department of the Treasury direct disbursing KSA was considered. DASD(DT&E) assessed the program as low risk for having all business processes and functions verified before entering IOT&E in March o Interoperability. The program met net-ready KPP requirements during DAI Increment 2, Release 2 development. Release 3 demonstrated the capability to meet interface requirements, based on SIT and UAT results, with one exception. All Department of the Treasury system interface partner systems were not available in time for the Release 3 test. The program is deferring these tests until viable interfaces become available, anticipated for early 2nd quarter FY o Reliability. The program met Release 3 LFD criteria for no unresolved Severity 1 or Severity 2 defects at the conclusion of UAT and no clusters of Severity 3 defects in any single functional area. Reliability growth planning is accomplished by qualitatively and quantitatively monitoring system defects over time. o Cybersecurity. Category I and II findings in Releases 2 and 3 were closed or mitigated in accordance with the program s plan of action and milestones. A red team was not able to exploit DAI as an outsider or as an insider with user-level credentials to DAI using limited or moderate capabilities. DISA performs vulnerability scans monthly of the DAI operational, COOP, and test environments, and the DLA Computer Emergency Response Team regularly performs application scans. DT software assurance code scans began in the 3rd quarter FY The program has a current authorization to operate through July The PMO is transitioning from DIACAP to the RMF for DoD IT to manage cybersecurity risks. DASD(DT&E) assessed DAI s ability to meet technical requirements to be low risk. Conclusion: Release 3 DT results evaluated through SIT and UAT supported an assessment of low risk for the LFD. 90 DASD(DT&E) FY 2016 Annual Report

97 Department of Defense Healthcare Management System Modernization (DHMSM) Executive Summary: DHMSM is a highly tailored MAIS program to acquire and field a modernized configurable and scalable electronic health record (EHR) system. DHMSM will support the availability of longitudinal health records for more than 9.6 million DoD beneficiaries and more than 153,000 Military Health System (MHS) personnel globally. DoD procured a commercial best-ofsuite (BoS) EHR system, augmented by best-of-breed products for requirements unmet by the BoS, naming the EHR solution MHS GENESIS. DHMSM will focus on replacing DoD legacy healthcare systems for fixed-facility medical treatment facilities. DoD awarded a contract in August 2015 to the Leidos Partnership for Defense Health (LPDH), a multicontractor team of Leidos, Cerner, Accenture, Henry Schein, and 31 other companies. Lead DT&E Organization: DHMSM Program Office Summary of FY 2016 DT&E Activities October 1, 2015 September 30, 2016, the PMO developed the program s initial detailed test plan (DTP) to support the IOC authority to proceed. June 20 24, 2016, LPDH conducted an MHS GENESIS training overview to familiarize the Government test teams with the system under test and to set expectations about test methodology employed during contractor testing and the rules of engagement. June 28 29, 2016, the DHMSM PMO conducted the Systems Engineering Technical Review #2 In-Progress Review/Preliminary Test Readiness Review. The review focused on the LPDH team s readiness to start the contractor-executed configuration and integration test (CIT). July 18 29, 2016, the Defense Health Agency (DHA) Cyber Security Division (CSD), LPDH, and the PMO jointly conducted a risk assessment of commercial services shared with DoD at the Cerner Technology Center to gauge the degree of difference between commercial practice and DISA Security Technical Implementation Guides. July 21, 2016, the LPDH PM conducted the CIT contractor readiness review to determine readiness to start the contractor test; the LPDH team determined that the test constraints/ limitations and risks were acceptable, and the LPDH PM provided approval to start the test. July 25, 2016 ongoing, LPDH executed CIT activities to verify that the MHS GENESIS meets the DHMSM functional and nonfunctional requirements; CIT is a contractor-led/governmentwitnessed event. August 1 12, 2016, the DHA CSD conducted an independent verification and validation of DoDspecific infrastructure at the Cerner Technology Center, Kansas City, Missouri. November 10, 2016, the PM conducted a DT&E readiness review to determine program readiness to start DT&E; the PM determined that the test limitations and risks were acceptable and approved DT&E to start on November 14, Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) provided one Defense Acquisition Executive Summary assessment to the USD(AT&L) in FY 2016; key assessment points include the following: DASD(DT&E) FY 2016 Annual Report 91

98 DoD DHMSM o The schedule is very aggressive to meet the IOC; the PM appropriately identified schedule risk, developed risk mitigation measures, and is tracking the schedule in the program s risk registry. o To ensure the appropriate cybersecurity information technology environment for DHMSM operations, the program should consider vulnerabilities introduced by enabling and supporting systems and support activities, including third-party certification and assessment activities, logistics, maintenance activities, and all interoperable or ancillary equipment with which the system operates or interfaces. LPDH is behind schedule conducting CIT. As of October 1, 2016: o LPDH performed only 70 percent of the test cases planned for execution by October 1, o Only 66 percent of the executed test cases passed. o There were several Severity 1 software defects, primarily related to interfaces, and more than 100 Severity 2 defects, primarily related to interfaces and functional requirements. o CIT exit criteria specify that all test cases must be completed successfully and there must not be any Severity 1 or 2 software defects. The jointly conducted risk assessment of commercial services shared with DoD identified a much larger number of vulnerabilities than expected. o LPDH developed a Plan of Actions and Milestones (POA&M) addressing the highest vulnerabilities. o LPDH committed to completing all mitigation actions for the high-severity vulnerabilities by December 6, 2016; moderate severities by March 28, 2017; and low severities by June 30, The DHA CSD assessment of DoD-specific infrastructure at the Cerner Technology Center resulted in a greater than anticipated number of findings; most were related to commercial software patches that were not installed. o LPDH developed a POA&M with risk mitigations to address the issues. o Closing findings will require functionality testing. o The DHA Authorizing Official approved an interim authority to test for the network on September 30, The program developed a DTP to support the program s DT&E planned for the 1st quarter FY 2017; the program office placed the DTP under configuration control in September Considering the progress of CIT and the cybersecurity vulnerability mitigations, the USD(AT&L) approved a re-plan of the DHMSM schedule on October 12, o Initial MHS GENESIS deployment is now scheduled for Fairchild AFB, Washington, in February o The revised schedule allows more time to mitigate cybersecurity vulnerabilities, make improvements to interfaces, and test capabilities before initial deployment. o The modified schedule also deconflicts some of the concurrent testing, incorporating a sequencing of CIT, DT&E, and operational assessment test phases with limited overlap of CIT and DT&E. The DHMSM program did not request a waiver or deviation from requirements in the TEMP. Conclusion: DASD(DT&E) assessed the DT&E to support the initial limited fielding at Fairchild AFB as moderate risk, but the program is likely to meet schedule threshold requirements. 92 DASD(DT&E) FY 2016 Annual Report

99 F-35 Lightning II Executive Summary: The F-35 is in its final stages of the 8.5-year DT program with flight testing envisioned to complete in late 2017, but testing may extend further into Eighteen test aircraft are conducting DT at two test sites: six F-35A conventional takeoff and landing (CTOL) variants, two F-35B short takeoff and vertical landing (STOVL) variants, and one carrier variant (CV) at the Air Force Test Center located at Edwards AFB, California, and five F-35B STOVL variants and four F-35C CVs at the Naval Air Warfare Center, Patuxent River, Maryland. The program has completed nearly nine-tenths of F-35 DT, executing roughly 91 percent of the nearly 60,000 planned flight test points (TPs). The majority of these TPs were flown in the Block 3 flight envelope with the Block 3i/3F mission system configuration. Block 3F mission systems TP execution is behind the baseline schedule because of unplanned Block 3i/3F discoveries and correction of deficiencies required to support Air Force IOC and Marine Corps fielded aircraft. The remaining Block 3F mission systems flight testing will extend through 2017 and depending on discovery as well as TP fly rates, is likely to extend well into The mission systems flight testing delays are primarily because of the additional work required to mature Block 3i as well as a December 2016 delivery of the final flight test version of Block 3F, which was originally scheduled for March The DT integrated test force (ITF) is currently executing Block 3F testing in multiple increments to reduce risk before the start of IOT&E in 2018 at the earliest. The planned flight sciences TP execution is slightly behind the 2016 planned schedule though expected to complete by November The program completed a 1-month weapons delivery accuracy surge in August 2016, completing 12 of 15 planned live weapon test events. The program is progressing through its aircraft durability program having completed more than 14,000 hours of fatigue testing on all three variants in During this testing, structural failures in major bulkheads of the F-35B test article were determined to be primarily from anodizing and etching processes used in manufacturing. A new laser shock peening (LSP) program was established to increase the life of the redesigned bulkheads and is currently under development, verification, and qualification. The DASD(DT&E) still asserts that the F-35 program should procure another test article to fully certify the STOVL aircraft life given the number of configuration changes as a result of fatigue testing. Although nearly all of the major technical issues examined in separate DoD assessments have an identified way ahead, all remain behind in their originally projected verification through test. Lead DT&E Organization: 412th Test Wing; Air Test and Evaluation Squadron Twenty-Three (VX-23) DASD(DT&E) FY 2016 Annual Report 93

100 DoD F-35 Lightning II Summary of CY 2016 DT&E Activities During CY 2016, the test program accomplished the following: o February 25, 2016, the F-35 program completed aircraft logistics climatic chamber testing with a STOVL aircraft (BF-7). o April 26, 2016, the ITF completed Block 3iR6.21 testing in support of Block 3i fleet release for the Air Force IOC in July o May 23, 2016, the ITF initiated F-35A airborne gun testing. o June 1, 2016, the ITF completed F-35B/C KC-30 air refueling testing. o July 6, 2016, the ITF initiated first live gun pod firing ground testing. o July 18 August 17, 2016, the ITF conducted a 1-month weapons delivery accuracy testing surge, completing 12 of 15 live firing events. o August 3, 2016, the ITF completed F-35B/C KC-135 air refueling testing. o August 24, 2016, the ITF initiated chemical-biological testing with an F-35A (AF-4). o August 25, 2016, the ITF completed the final F-35C DT carrier testing (DT III) period aboard USS GEORGE WASHINGTON (CVN 73). o November 10, 2016, the ITF completed ground testing of the F-35B/C gun pod. o November 16, 2016, the ITF completed the final F-35B DT shipboard testing period aboard USS AMERICA (LHA-6) in conjunction with operation and fleet F-35Bs. For CY 2016, TPs required for re-fly, development, and discovery (RDD) have exceeded the 2016 planned TPs for mission systems and flight sciences. o Mission Systems 1,546 TPs planned; 1,784 TPs flown; 2,311 TPs approved and scheduled to be flown (49 percent increase in RDD planned for 2016). o Flight Sciences 449 TPs planned, 461 TPs flown, 614 TPs approved and scheduled to be flown (37 percent increase in RDD planned for 2016). Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) has been thoroughly engaged with the F-35 Program Office, the Services, and OSD staff in increasing insight into system maturity and test progress and ensuring that adequate test resources are planned to test F-35 against current and planned threats. DASD(DT&E) directly engaged in the STOVL and CV durability test article bulkhead failure analysis, test article repair, and LSP project development and certification. The program is meeting mission systems projected fly rates, but mission systems testing incrementally leading to the final Block 3F configuration is notably behind the baseline schedule because of unplanned Block 3i/3F discoveries, correction of deficiencies, and Block 3i/2B testing required to support Air Force IOC and Marine Corps fielded aircraft. Mission systems TP closure and capability verification are behind the planned schedule. Although flight test execution at the two primary test sites for 2016 flight sciences is below the planned fly rates, F-35A Block 3 flight envelope testing will complete in December 2016/early 2017, F-35C envelope testing will complete in the fall of 2017, and F-35B envelope testing will complete in late 2017/early Overall, roughly 91 percent of the nearly 60,000 planned flight TPs have been flown with the most complex TPs remaining; TPs in 2016 were mainly in the Block 3 envelope and Block3i/3F mission systems testing. Key factors impacting DT schedule were additional test requirements for Block 3i, Block 3F late incremental software deliveries to flight test, and an increase in the planned RDD TPs. For example, Block 3FR6 software is projected to be delivered more than 10 months late to flight test in December DASD(DT&E) estimates the completion of Block 3F mission systems flight test in the April to June 2018 time frame, 15 to 18 months behind the Block 3F baseline plan. Mission systems 94 DASD(DT&E) FY 2016 Annual Report

101 DoD F-35 Lightning II effectiveness and lethality testing is the critical path to completing Block 3F TPs. Risk to Block 3F DT execution is expected to rise in 2017 as the complexity of mission systems testing increases. Nearly all of the major technical issues examined in separate DoD assessments have an identified way ahead, though all remain behind in their originally projected verification through test. Full-scale ground durability testing is in various stages of completion with F-35A at 20,090 hours (second lifetime complete, continuing to a third lifetime at 24,000 hours); F-35B continuing testing at 15,145 hours (89.3 percent second lifetime complete); and F-35C at 16,000 hours (completed second lifetime) with structural deficiencies being uncovered; discovery rates to date are as follows: o CTOL 15 realized of 17 projected; 5 major versus 6 projected. o STOVL 35 realized of 17 projected; 11 major versus 6 projected. o CV 26 realized of 17 projected; 11 major versus 5 projected. The root cause of the F-35B test article s major bulkhead failures was determined to be primarily from anodizing and etching processes used in manufacturing; LSP has been identified to increase the life of the redesigned Bulkheads 496 and 472. LSP is still under development, verification, and qualification but has been authorized for Low-Rate Initial Production (LRIP) 11 production cut-in. Additional fixes have been identified for the other bulkheads. The F-35 program is conducting a STOVL durability gap analysis to determine the requirement to fully certify the STOVL aircraft to 8,000 hours lifetime. DASD(DT&E) s assessment is that the F-35 program will require another test article to fully certify the STOVL aircraft. Reliability: Performance across all variants is below the planned Operational Requirements Document RGCs established and expected for this phase of testing. Reliability has been a manageable factor for flight test execution, although continued below-growth-curve performance will affect testing as the program moves into more complex mission system events. The Autonomic Logistics Information System (ALIS) continues to be behind in development; ALIS did not meet the July 2016 Air Force IOC requirement and now is projected for an early 2017 delivery. ALIS 3.0 operational capability is medium to high risk to meet an early to mid-2018 delivery for IOT&E. DASD(DT&E) assesses the security cross-domain solutions planned to coordinate classified data flow through ALIS as medium to high risk. Stable ALIS operability with commensurate spare parts availability is a significant factor in maintaining efficient flight test execution. Mission Data File (MDF): The U.S. reprogramming lab continues to be behind schedule in developing an MDF to support IOT&E. The delay is mainly due to a lack of signal generators required to fully verify and validate the MDF. The F-35C stations 1/11 outer-wing substructure requires a redesign because of load exceedances during landings; an outer-wing redesigned substructure is currently being tested on an F-35C. The program conducted its 2016 DT&E activities in accordance with the approved TEMP; no waivers were requested. Conclusion: Block 3F mission effectiveness and lethality TP completion and weapons certification are the critical paths to completing F-35 DT. DASD(DT&E) is currently estimating an April to June 2018 completion of Block 3F DT&E. Nearly all of the major technical issues examined in separate DoD assessments have an identified way ahead, though all remain behind in their originally projected verification through test. DASD(DT&E) still asserts that the F-35 program will require another test article to fully certify the STOVL aircraft life. DASD(DT&E) FY 2016 Annual Report 95

102 Joint Light Tactical Vehicle (JLTV) Executive Summary: The JLTV family of vehicles (FoV) is expected to provide the joint Warfighter with a mobile, lightweight tactical vehicle capable of being transported by fixed- and rotary-wing aircraft, with increased force protection compared with the current up-armored high-mobility multipurpose wheeled vehicle. The FoV includes a two-seat variant (Utility/Shelter Carrier) and a four-seat variant (General Purpose (GP) and Close Combat Weapons Carrier). The GP can also be configured as a Heavy Guns Carrier. All configurations can add supplemental armor protection and special purpose kits. The JLTV should support rapid deployment and offensive operations; interoperate with other vehicle and weapon systems; provide increased force protection, reliability, maintainability, availability, payload, and fuel efficiency compared with current vehicles; and provide similar mobility, netcentricity, and transportability with reduced logistical footprint. Eight KPPs (mobility, transportability, net-ready (including cybersecurity), force protection, system survivability, payload, sustainment, and training) and five KSAs (operating and support cost, reliability, energy, interoperability, and unit cost) define the most significant user requirements for the system. Following completion of EMD in FY 2014, the Army selected Oshkosh Defense (Oshkosh) for production and completed a MS C review in August After the selection, one competitor filed a protest. After the Government Accountability Office denied Lockheed Martin s protest in December 2015, the program proceeded into low-rate initial production (LRIP) and Production and Deployment (P&D) planning. After an extensive Army-directed review and consideration of test scope reductions, OSD approved an updated JLTV TEMP in July 2016, and the JLTV Joint Program Office (JPO) proceeded with detailed planning. The Army received the first vehicles in September 2016 but then rejected deliveries in October 2016 to address suspension/frame production issues. Lead DT&E Organization: ATEC MSED Summary of FY 2016 DT&E Activities After EMD, before the LRIP contract award, Oshkosh completed additional design changes to reduce production costs, address requirement shortfalls, and consider requirements such as lowvelocity airdrop. Oshkosh continued to make design changes after protest withdrawal (December 2015) until the JPO accepted the baseline design. August 2015 July 2016, Oshkosh completed LRIP design changes and completed baseline review actions. In September 2015, the JPO briefed a contractor-proposed reliability specification assessed as a point estimate (rather than the recommended method of assessing the lower confidence bound of 96 DASD(DT&E) FY 2016 Annual Report

103 DoD JLTV a point estimate) included in the contract; the JPO also briefed a contract incentive to meet or exceed the new specification. From September 2015 through June 2016, the Army considered TEMP changes and reductions in test scope, with potential cost reductions of $59.6 million. After multiple deliberations about the impact of the cuts, the program and OSD agreed to test scope reductions that would result in saving $36 million. During the week of January 11, 2016, the JPO conducted the start-of-work meeting. The JPO announced that the engine supplier could supply approximately 2,000 engines (versus the planned production of more than 54,000) and the program would seek a replacement engine with similar form, fit, and function. On July 8, 2016, DASD(DT&E) approved the DT&E strategy within the TEMP. The approved TEMP reflects the agreed-upon test reductions, as well as descriptions of remaining testing that was not successfully completed during EMD: transportability testing and certification; cybersecurity testing; command, control, communications, computers, intelligence, surveillance, and reconnaissance (C4ISR) equipment integration; and corrosion testing. Beginning in August 2016, Oshkosh introduced additional design changes to the LRIP configuration and submitted ECPs to the program for approval. On September 22, 2016, the JPO conducted the developmental test readiness review (DTRR). The JPO identified 49 ECPs in that review (some are approved; others are awaiting review or consideration). On September 30, 2016, Oshkosh delivered the first seven test vehicles. On October 12, 2016, the JPO provided a briefing outlining the changes included with the new engine: the ECPs redesigned a myriad of systems including the cooling system, fuel system, lower frame, hydraulic system, brakes/suspension, C4I wiring, and software. The new vehicle configuration carries a new designation: JLTV A1. On October 31, 2016, the JPO announced that it would not accept any more vehicle deliveries until Oshkosh has corrected material compliance issues with the rear subframe assemblies. Summary of FY 2016 DT&E Engagement and Assessments Oshkosh conducted testing of its proposed design changes (after EMD but before LRIP contract award) but was not willing to provide this test data to the Government. The newly accepted reliability specification is numerically higher than the Army/Marine Corps user requirement. However, contrary to Army best practices for reliability, the specification is defined as a point estimate instead of applying confidence intervals. By defining the requirement this way, the contract effectively incentivizes frequent corrections that could result in significant configuration changes. This means the result may meet the contract specification but actually provide low confidence that the program is meeting the Army reliability requirement. The Army-directed review of the P&D test programs delayed TEMP approval and detailed test planning. In addition, the Army based its test scope reduction and associated costs savings on strong EMD performance; however, subsequent design changes (and a new variant using a new engine) drive the need to revisit the test program and make sure it is adequate to evaluate vehicle performance. o Transportability testing and certification, cyber testing, C4ISR integration, and corrosion testing were not adequately tested during EMD and represent some of the greatest risks in meeting mission requirements. The TEMP describes a C4ISR test plan and two cybersecurity DT&E test events: a cooperative vulnerability identification (CVI) and a combined CVI/cooperative vulnerability and penetration assessment (CVPA). The initial CVI, scheduled for late FY 2016, has evolved to an analytical DASD(DT&E) FY 2016 Annual Report 97

104 DoD JLTV exercise, rather than the more extensive cooperative test event described by the Cybersecurity T&E Guidebook. The combined CVI/CVPA should be addressed within the draft C4ISR test plan, but the relevant section is currently focused on routine assessment and authorization testing required under DoDI , Risk Management Framework (RMF) for DoD Information Technology (IT). At this time, the plan does not address in sufficient detail the test planning required for a successful and well-focused combined CVI/CVPA. The program should accelerate efforts on cybersecurity analysis, planning, and testing. Failure to do so could result in fielding a system that is susceptible to cyber intrusions. The DTRR in September 2016 included 20 pre test readiness review (TRR) items, of which 6 were Green (complete) and 14 were Yellow (incomplete, with a plan in place to meet the schedule), and 13 TRR checklist items with 2 rated as Green and 11 rated as Yellow. Some of these items are administrative in nature, whereas others (safety assessments, training packages, operator manuals) represent critical items that require significant effort before testing can begin. In addition, this planning has not kept pace with the ongoing design changes. During the DTRR, DASD(DT&E) highlighted the importance of considering design changes to ensure adequate testing. The program provided copies of approved ECPs, as well as the JLTV A1 changes. The JLTV configuration (even before the engine change in JLTV A1) has evolved away from the EMD configuration, and performance may not be consistent with EMD results. The production verification testing currently described in the TEMP will apply to the first (approximately) 2,000 vehicles but not to the JLTV A1. The JLTV A1 changes associated with the new engine have resulted in a significant system configuration change. Automotive performance and reliability performance could vary considerably between the EMD vehicle, the baseline design vehicle, the LRIP vehicle with the current engine (JLTV), and the LRIP/FRP vehicle with the new engine (JLTV A1). Current plans for engine testing, performance testing, and vehicle reliability testing are not complete as of the date of this report. JLTV A1 testing is also constrained by availability of LRIP A1 vehicles, which become available just before multi-service operational test and evaluation (MOT&E), and attempting to maintain the original program schedule without regard for the design changes. JLTV A1 used in MOT&E will have little production verification testing completed before MOT&E starts and risks failing performance measures in MOT&E. In particular, there will be little reliability testing that confirms that JLTV A1 meets user requirements before MOT&E. The JPO did not request a waiver or deviation from requirements in the TEMP. Conclusion: In FY 2016, at the Army s direction, the program updated the P&D test strategy based on vehicle performance during EMD in order to reduce test costs and leverage data from EMD testing. Since then, the nature and extent of the design changes planned for the vehicle warrant another review of the overall P&D test strategy to include reliability and cybersecurity testing. JLTV A1 testing should be rescoped to include performance and reliability testing that provides confidence that the production configuration vehicle meets the Army s requirement. The program should not proceed into MOT&E until adequate testing confirms LRIP and JLTV A1 vehicle performance. The program should not advance into FRP until adequate testing confirms JLTV A1 performance and capability. 98 DASD(DT&E) FY 2016 Annual Report

105 Key Management Infrastructure (KMI) Increment 2 Executive Summary: The KMI program is a vital element of the DoD defense-in-depth strategy, adopted to ensure the security posture for the DoDIN by providing transparent cryptographic capabilities consistent with operational imperatives and mission environments. The program completed Spiral 1 in 2012, after emerging from a cost- and schedule-driven critical change review in May Spiral 2 is in development and is employing agile software development methods to expand KMI capabilities and services to all clients of the legacy Electronic Key Management System. DASD(DT&E) approved an addendum to the Increment 2 TEMP covering Spiral 2 activities through the full deployment decision, currently scheduled for March The USD(AT&L) approved fielding of Spin 1 capabilities in December 2015, 17 months later than planned. Spin 2 testing also incurred delays and is projected to get a fielding decision in September Because the program can achieve its KPPs when Spin 3 is complete, Spin 4 capabilities have been moved into a subsequent increment so that Increment 2 can be completed without incurring another critical change. Lead DT&E Organization: JITC Summary of FY 2016 DT&E Activities April 5 9, 2016, the developmental contractor conducted DT&E 1 testing of Spiral 2, Spin 2 capabilities. April 21 26, 2016, the developmental contractor conducted a DT&E 1 retest to verify fixes of discrepancies noted during DT&E 1. July 11 22, 2016, JITC conducted DT&E 2 testing of Spiral 2, Spin 2 capabilities at multiple sites with Service user participants. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) supported multiple executive management reviews called by the DoD CIO to review technical progress. The test community conducted a complete review and rework of the remainder of the Spiral 2 schedule to include complementary events and ensure Service support for testing of new development and operations and maintenance updates. DASD(DT&E) provided four Defense Acquisition Executive Summary assessments to the USD(AT&L) in FY The KMI program did not request a waiver or deviation from requirements in the TEMP. DASD(DT&E) Program Assessment DASD(DT&E) provided a DT&E program assessment of Spin 1 capabilities in support of the Spin 1 software fielding decision in December A summary of the evaluation follows: DASD(DT&E) FY 2016 Annual Report 99

106 DoD KMI o Performance. During Spiral 1, KMI met five of the seven KPPs and is scheduled to meet the remaining two through capability deliveries planned in Spiral 2. o Reliability. Token reliability was identified as an issue during Spiral 1, and the program office implemented failure mode analysis to identify fixes to address the issue. Taking into consideration the three recent test events in which no token failures were recorded, DASD(DT&E) projects that the token will meet the reliability requirement once sufficient operational time has been accrued. o Interoperability. During Spiral 2, Spin 1 DT, JITC characterized information exchanges observed over the critical network interfaces (SIPRNET, NIPRNET, public switched telephone network, and Integrated Services Digital Network). JITC determined that KMI met accuracy and availability criteria for the required information exchanges based on operational assessment (OA) results. o Cybersecurity Testing. No high-priority security issues were identified during the Spiral 2, Spin 1 DT or OA events. Conclusion: The new schedule, although aggressive, provides the best opportunity for the program office to deliver needed capabilities to the user. An additional increment will be needed to deliver long-term sustainment capabilities that were part of Spin DASD(DT&E) FY 2016 Annual Report

107 Public Key Infrastructure (PKI) Increment 2 Executive Summary: The goal of the DoD-wide infrastructure is to provide PKI for the generation, production, distribution, control, revocation, recovery, and tracking of public key certificates and their corresponding private keys and enabling commercial off-the-shelf and Government off-the-shelf applications that provide information assurance and e-business capabilities. PKI will issue and manage electronic/digital identities and associated credentials and key materials for users, applications, servers, and network components. Increment 2 initiatives include the use of a hardware token on the SIPRNET as well as the NIPRNET Enterprise Alternate Token. The program completed testing for the Token Management System Release 3.0 and received a suitable and effective rating from the operational test authority, with limitations noted for some discrepancies during testing. Testing of Releases 4.1 and 4.2 progressed more slowly than planned because of discrepancies found in contractor and Government testing. As each test event slipped, the cascading effect raised the risk of achieving the full deployment decision (FDD). The TEMP Addendum for Spiral 3 was approved in February 2016 after 3 years of effort. The program office proposed foregoing DT&E IIA for Release 4.3 in order to include the capabilities in the limited user testing (LUT) for Releases 4.1 and 4.2 that started on July 18, DASD(DT&E) concurred in the proposal because of the minimal risk associated with Release 4.3 capabilities. The TEMP Addendum for Spiral 4 is in staffing. Lead DT&E Organization: DoD PKI PMO Summary of FY 2016 DT&E Activities December 8 11, 2015, the PMO and Service users conducted a DT&E IIA test of Releases 4.1 and 4.2 capabilities at multiple sites, hosted on JITC PKI laboratory and DISA operations and maintenance laboratory servers. January 26, 2016, the PMO demonstrated fixes to discrepancies found during DT&E IIA in a regression test. February 2 5, 2016, the PMO and Service users conducted a DT&E IIB test of Releases 4.1 and 4.2 capabilities hosted on DISA operational servers. March 29 April 1, 2016, the PMO demonstrated fixes to discrepancies identified during DT&E IIB in a follow-on regression test. Summary of FY 2016 DT&E Engagement and Assessments The TEMP Addendum for Spiral 3 was completed in 2016 and approved by DASD(DT&E) on February 22, The Spiral 4 TEMP Addendum is in staffing and projected for completion before integrated testing of initial Spiral 4 capabilities. DASD(DT&E) has remained engaged with the PMO through regular WIPTs, TEMP development sessions, and test observations. DASD(DT&E) FY 2016 Annual Report 101

108 DoD PKI DASD(DT&E) assessment of Releases 4.1, 4.2, and 4.3 DT results was provided to the PEO as part of the determination to proceed with a LUT of those capabilities conducted by JITC and overseen by DOT&E. DASD(DT&E) provided four Defense Acquisition Executive Summary assessments to the USD(AT&L) in FY 2016; there were no potential or actual program problems to report. The DoD CIO convened a quarterly program review to discuss non-person entity (NPE) requirements with Service CIOs and better understand the PMO s interpretation of the scope of the requirement and the possible technical solutions based on that scope. DASD(DT&E) contributed historical perspective on the origins of requirements and the policy implications inherent in the scope determination. The program office proposed foregoing DT&E IIA for Release 4.3 in order to include the capabilities in the LUT for Releases 4.1 and 4.2 that started on July 18, DASD(DT&E) concurred in the proposal because of the minimal risk associated with Release 4.3 capabilities. The PKI program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The PMO built an aggressive schedule to complete all necessary test events in time for the projected FDD. Recent slips to Releases 5 and 6 DT eliminated any slack that had been available and increased risk to on-time completion of follow-on operational test and evaluation scheduled for April Clarification of the scope of NPE device types and numbers to adequately bound OT requirements is necessary to properly prepare the Spiral 4 TEMP Addendum and deliver capabilities that the Services and Defense Agencies need to protect network hardware. 102 DASD(DT&E) FY 2016 Annual Report

109 6.2 Army Programs This section includes summaries of the following 10 programs: Abrams M1A2 System Enhancement Package Version 3 (SEPv3) Engineering Change Proposal (ECP) 1a Army Integrated Air and Missile Defense (IAMD) Bradley Family of Vehicles (BFoV) Engineering Change Proposal (ECP) 2 Indirect Fire Protection Capability Increment 2 Intercept (IFPC2-I) Joint Air-to-Ground Missile (JAGM) Joint Tactical Radio System (JTRS) Handheld, Manpack, and Small Form Fit (HMS) Manpack (MP) Radio Joint Tactical Radio System (JTRS) Handheld, Manpack, and Small Form Fit (HMS) Rifleman Radio (RR) M109A7 Family of Vehicles (FoV) Self-Propelled Howitzer (SPH) and Carrier, Ammunition, Tracked (CAT) Vehicle Mid-Tier Networking Vehicular Radio (MNVR) (AN/VRC-118(v)1) Stryker Family of Vehicles (FoV) Engineering Change Proposal (ECP) DASD(DT&E) FY 2016 Annual Report 103

110 Abrams M1A2 System Enhancement Package Version 3 (SEPv3) Engineering Change Proposal (ECP) 1a Executive Summary: The Abrams M1A2 SEPv3 ECP 1a main battle tank (MBT) is a full-tracked, low-profile land combat assault weapon system possessing significant survivability, shoot-on-the-move fire power, joint interoperability (for the exchange of tactical and support information), and a high degree of maneuverability and tactical agility. The crew has the capability to engage the full spectrum of enemy ground targets with a variety of accurate point and area fire weapons in urban and open terrain, as well as to defend against helicopter threats. The primary mission area for the Abrams M1A2 SEPv3 ECP 1a MBT is force application-engagement. The Abrams M1A2 SEPv3 ECP 1a MBT will be used as the principal weapon system of the U.S. Army s armored brigade combat team and will enable the joint force to maneuver to dismantle an adversary s system of offense and defense, assist in preempting an adversary s freedom of action, defeat or destroy the threat, and protect joint and coalition forces. The Abrams M1A2 SEPv3 ECP 1a MBT must be able to defeat/suppress enemy tanks, reconnaissance vehicles, infantry fighting vehicles, armored personnel carriers, anti-tank guns, guided missile launchers (ground and vehicle mounted), bunkers, dismounted infantry, and helicopters. The Abrams M1A2 SEPv3 ECP 1a program will maintain combat effectiveness of the Abrams tank fleet through 2050 and integrate the following technologies: Joint Tactical Radio System (JTRS); Joint Battle Command Platform; battery monitoring system (BMS); improved generator; upgraded slip ring assembly (USRA); enhanced hull power distribution unit (ehpdu) and common remote switching modules; line-replaceable modules; ammunition data link; integration kit for counter remote-controlled improvised explosive device electronic warfare Duke V3; next evolution armor upgrade; high-definition displays; mine-blast survivability improvements, including a new lightweight underbody armor plate; and an auxiliary power unit (APU). The Abrams MBT is an ACAT 1D program in the acquisition life cycle Operations and Support phase. The Army Acquisition Executive approved and directed the PEO Ground Combat Systems to execute the Abrams ECP efforts. The acquisition strategy relies on three separate production decision points (PDPs) to assess system performance and determine readiness for production. The test program is structured to inform each PDP. PDP 1, executed in April 2016, authorized the initial long-lead material purchase for 45 tanks. This smaller purchase limits risk as the program is still in the early stages of testing. Original equipment manufacturer (OEM) contractor test data as well as emerging production proveout test (PPT) data supported PDP 1. PDP 2, planned for March 2017, authorizes long-lead material buys for 180 tanks. PDP 3, planned for March 2020, will support the final long-lead material decision and a decision to upgrade the remaining 1,386 tanks. Lead DT&E Organization: ATEC MSED 104 DASD(DT&E) FY 2016 Annual Report

111 Army Abrams M1A2 SEPv3 ECP 1a Summary of FY 2016 DT&E Activities The Government phase of the PPT began in October OEM component qualification testing of the 20 line-replaceable units (LRUs) selected by the program is scheduled for completion in December Summary of FY 2016 DT&E Engagement and Assessments The program is executing a medium-risk DT&E schedule driven by highly concurrent execution of subsystem qualification testing and system-level PPT. Originally planned for completion on or about June 2015 and before PPT, qualification testing on three LRUs continued concurrently with PPT. Qualification testing of the APU was completed in February Testing of the USRA was completed in June Qualification testing of the improved generator is scheduled for completion in December If component qualification testing requires design changes to the LRUs, it could impact PPT. Issues identified during qualification testing and contractor system-level testing have driven the need to conduct an unplanned (from the TEMP) retrofit of the PPT test vehicles. The retrofit consists of modifications to the ehpdu electromagnetic interference wiring harness and firmware, the USRA, the APU exhaust door, the improved generator, and the BMS software, and an upgrade to the vehicle software (version 5.2). The impact of these retrofits on PPT data collected to date is still to be determined. RAM testing is ongoing. As of September 2016, of the 16,000 miles planned for RAM testing, 2,390 miles (15 percent) have been completed, reviewed, and scored. It is very early in testing to evaluate the RAM performance of the vehicle system. However, the interim results show a system-level reliability of 419 mean miles between combat mission failure (MMBCMF) and 56 mean miles between system failure (MMBSF). These results compare with Operational Requirements Document requirements of 320 MMBCMF and 101 MMBSF and SEPv2 test results of 246 MMBCMF and 63 MMBSF. Emerging results of automotive performance testing at Aberdeen and Yuma Test Centers acceleration, maximum and minimum speed, braking, grades, and slopes appear to be favorable with little impact due to the increased vehicle weight. The Abrams M1A2 SEPv3 ECP 1a program did not request a waiver or deviation from requirements in the TEMP. DASD(DT&E) Program Assessment In April 2016, DASD(DT&E) assessed system performance and published a DT&E assessment to support the PDP 1 decision. The summary of that evaluation follows: o Performance. As anticipated in the TEMP, and in development of the program test and acquisition strategies, most of the evaluation criteria could not be assessed at the time of PDP 1. However, observations could be made from contractor testing data, results of component qualification testing, and the limited data emerging from Government testing. Emerging results of automotive performance testing at Aberdeen and Yuma Test Centers acceleration, maximum and minimum speed, braking, grades, and slopes appeared to be favorable with little impact due to the increased vehicle weight. o Reliability. RAM testing was ongoing. Results of the first scoring conference in March 2016 were favorable. Although it was still very early in Government RAM testing, 598 miles of the 16,000 miles planned (or 3.1 percent of the total miles) were executed. RAM scoring of the 507 Test Incident Reports generated by testing resulted in emerging point estimates of 267 MMBCMF and 127 MMBSF. o Cybersecurity. Cybersecurity testing is planned for January DASD(DT&E) FY 2016 Annual Report 105

112 Army Abrams M1A2 SEPv3 ECP 1a DASD(DT&E) supported the program proceeding to PDP 1 and procuring vehicles to support continued DT&E and follow-on operational test and evaluation. The program had conducted rigorous contractor and component-level qualification testing to mitigate the risk of proceeding to PDP 1 before completion of Government system-level testing. Going forward, DASD(DT&E) recommended that the program review the detailed PPT schedule to ensure synchronization with the data requirements documented in the Evaluation Framework Matrix for PDP 2. Conclusion: The PM is effectively managing a DT&E strategy that is medium risk because of concurrency in test execution. DT scope is in accordance with the approved TEMP, although the DT schedule has experienced minor delays. 106 DASD(DT&E) FY 2016 Annual Report

113 Army Integrated Air and Missile Defense (IAMD) Executive Summary: The Army IAMD system provides the overarching capability to integrate air defense artillery mission command nodes, sensors, and launchers on an Army IAMD network to perform network-centric operational capabilities that individual element systems cannot achieve independently. The Army IAMD system will provide the common mission command capabilities for operations through the IAMD Battle Command System (IBCS). Army air and missile defense (AMD) provides area defense against tactical ballistic missiles, rockets, artillery, mortars, air-breathing targets, cruise missiles (CMs), and unmanned aerial systems. Program components include the engagement operations center (EOC), integrated fire control network (IFCN), IFCN relays, IBCS software, and sensor/shooter adaption kits. Associated AMD components are Patriot (Phased Array Tracking Radar to Intercept of Target) launchers, radars and radar interface units, and Sentinel radars. In addition, Indirect Fire Protection Capability will be included in the future. The program entered the EMD phase in December In 2012, the Army restructured the program because of cancellation of the Surface-Launched Advanced Medium-Range Air-to-Air Missile program, elimination of the Joint Land Attack Cruise Missile Defense Elevated Netted Sensor System program, and the decision not to procure the Medium Extended Air Defense System and to expand the Army IAMD battle command capabilities to the Army AMD command level. The program MS C was originally scheduled for the 1st quarter FY 2015 and is now tentatively scheduled for the 4th quarter FY The USD(AT&L) Defense Acquisition Executive will make the decision to proceed into the Production and Deployment phase. Lead DT&E Organization: ATEC AFED Summary of FY 2016 DT&E Activities November 12, 2015, the Army IAMD program office conducted the second flight test to demonstrate integration of IAMD components onto the IFCN. February 2016, the Army IAMD program office conducted DT&E search and track (S&T) missions to demonstrate simulated engagements. March 26 May 6, 2016, ATEC conducted a limited user test (LUT) to assess hardware and software maturity, operational performance, and the ability of Army IAMD to support the AMD mission without degrading current fielded sensor and weapon capabilities. As part of the LUT, the program conducted its third flight test to demonstrate near-simultaneous dual engagements against surrogate CM and tactical ballistic missile targets. Summary of FY 2016 DT&E Engagement and Assessments The Army IAMD program successfully demonstrated integration of AMD components through a flight test consisting of a battalion EOC, battery EOC, a Patriot radar interface unit and radar, an DASD(DT&E) FY 2016 Annual Report 107

114 Army Army IAMD IFCN relay connected to two adapted Patriot Advanced Capability (PAC)-3 launchers, and two IFCN relays connected to Sentinel radars onto an IFCN. The components combined to detect, track, engage, launch, and intercept a CM target with a PAC-3 missile. The Army IAMD program successfully demonstrated integration of AMD components through a flight test consisting of a battalion EOC with a Patriot radar, a non-collocated battery EOC, a remote Patriot radar interface unit and radar connected to an IFCN relay, a remote IFCN relay connected to two Patriot launchers equipped with PAC-3 missiles, a remote IFCN relay connected to two Patriot launchers equipped with Guidance Enhanced Missile Tactical (GEM-T) missiles, and two remote Sentinel radars connected to IFCN relays, all operating on the IFCN. The components combined to detect, track, engage, launch, and intercept a tactical ballistic missile target with a PAC-3 missile and a CM target with a GEM-T missile. The Army IAMD program accomplished DT&E S&T missions. The S&T missions provide an opportunity to assess capability in benign and airborne electronic attack waveform environments and perform nearly simultaneous simulated PAC-2 and PAC-3 engagements against air-breathing threat and CM surrogates. The DT&E S&T missions were not sufficiently robust to identify the inherent software instability, and the Army s decision to conduct the LUT almost immediately following the DT&E S&T missions prevented the Army from incorporating corrective actions for the failures identified. DASD(DT&E) recommends that for future DT&E, sufficient time be allotted to incorporate corrective actions before OT&E. In January 2017, DASD(DT&E) provided emerging results to the Army Acquisition Executive: o EMD testing shows that Army IAMD has demonstrated a limited capability to engage tactical ballistic missiles, CMs, and air-breathing threats with a mix of weapons using fused organic and nonorganic sensor data within a low-intensity environment. LUT results confirm that system performance is significantly short of threshold requirements as Army IAMD lacks the system maturity necessary to reliably perform most force operations and engagement operations tasks. The Army IAMD system has two key and related reliability requirements, and both estimates remain significantly lower than required and planned. The Army IAMD system requirement is to remain operational 90 percent of the time throughout a 72-hour mission. The current ATEC estimate is that the system has a 6 percent probability of remaining operational throughout a 72-hour mission with a reliability growth potential of 71 percent. The Army IAMD system requirement, as stated in the draft Capability Production Document, is to maintain a mean time between system abort of 205 hours. The current post-lut estimate is 21.8 hours with a reliability growth potential of 71.4 hours. o As stated in the DT&E FY 2015 Annual Report, DASD(DT&E) continues to recommend that program milestones be delayed until the program develops higher component maturity and program reliability requirements are achievable. o The LUT was to provide much of the foundational information for a knowledgeable MS C decision. However, immature hardware and software resulted in reduced scope and lessened operational complexity, which limits the ability to assess Army IAMD performance. DASD(DT&E) believes it is likely that there will be negative performance effects when the number of nodes and the geographic span of control are increased to operationally realistic levels. o Cybersecurity T&E activities are incomplete. Two of the three cybersecurity T&E activities were canceled because of software immaturity and lack of ability to accomplish Soldier training. System immaturity and instability made Soldier training against a cyber threat impractical. DASD(DT&E) believes that the current cybersecurity status is insufficient for MS C. 108 DASD(DT&E) FY 2016 Annual Report

115 Army Army IAMD o Army IAMD is subject to Army and joint interoperability certification. However, IBCS software supported only 15 of the 33 Link 16 message sets during the LUT. The lack of Link 16 integration is a major limitation and resulted in limited interoperability with joint external interfaces. Improvements have been made, but Army IAMD interoperability with Link 16 remains incomplete, and significant deficiencies remain. o The Army did not use the contractor and Government software integration laboratory (G-SIL) facilities as planned to find discrepancies, incorporate corrective actions, and mature IBCS software before integration at White Sands Missile Range (WSMR), New Mexico. The Army delivered the IBCS software near simultaneously from the contractor system integration lab (SIL) to the G-SIL, Tobin Wells system integration lab (T-SIL), and WSMR for range integration and test, rather than passing the software progressively using the planned test-fix-test strategy. This methodology did not allow for correlation between G-SIL, T-SIL, and WSMR T&E results. Extensive DT&E needs to occur as the IBCS software incorporates corrective actions and adds the functionality to provide full capability. DASD(DT&E) recommends that the Army exercise a disciplined and rigorous progressive test strategy to serially evaluate software in the contractor SIL, G-SIL, T-SIL, and WSMR to achieve correlation and maintain configuration control among the testing results. o DASD(DT&E) recommends that the Army PEO Missiles and Space develop a long-term strategy to identify SoS integration requirements and evaluation requirements for the sensors, weapon, IBCS, and IFCN components of the AMD portfolio. o The program canceled or postponed significant TEMP DT&E activities. Because of software development delays and the desire to avoid slipping the MS C decision date, the program canceled S&T missions and then partially added them back by postponing a flight test until after the MS C decision. The lack of details identifying when future capability will be available prevents developing an adequate T&E program to support the program through IOC. The Army IAMD program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The Army IAMD configuration baseline is immature and so far lacks the capability need identified in the Acquisition Strategy and CDD. The program has not met the exit criteria for EMD and is far from demonstrating the acceptable level of performance, reliability, cybersecurity, and interoperability necessary. DASD(DT&E) FY 2016 Annual Report 109

116 Bradley Family of Vehicles (BFoV) Engineering Change Proposal (ECP) 2 Executive Summary: The BFoV consists of the Bradley M2A4, which will fill the combat role of the Bradley infantry fighting vehicle, cavalry fighting vehicle, and engineer Bradley variant; and the M7A4, which will fill the combat role of the A3 Bradley fire support team with the fire support sensor system mission package. Each variant performs missions within the armored brigade combat team (ABCT) as part of a combined arms battalion or an armored reconnaissance squadron. The vehicles operate in task-organized formations within the ABCT but can also operate independently. The ABCT mission is to fight and win engagements and battles in support of operational and strategic national objectives. Bradley ECP 2 builds on ECP 1, which was a suspension upgrade program, beginning in FY ECP 1 and ECP 2 kits installed together will result in the conversion of existing Bradley A3 versions into A4 versions. The Army s current plan is to convert all Bradley A3s to A4s. ECP 2 provides the additional power/drivetrain/cooling/electronics upgrades needed to host and effectively integrate existing and planned Army mission command systems. The Army Acquisition Executive approved and directed the PEO Ground Combat Systems to execute the Bradley ECP efforts. The acquisition strategy relies on multiple production decision in-process reviews (IPRs), with the first IPR scheduled for June 2017 (74 vehicles), followed by nine more IPRs (180 vehicles each) through the 3rd quarter FY Lead DT&E Organization: ATEC MSED Summary of FY 2016 DT&E Activities The original equipment manufacturer (OEM) conducted contractor DT from FY 2015 through May Hardware and software design issues identified during OEM DT necessitated a modification and extension of the OEM DT phase and a 6-month slip to the start of Government DT (from January 2016 to June 2016). In May 2016, the Army conducted a developmental test readiness review (DTRR) to assess readiness to begin Government DT. This review resulted in the need to conduct a follow-on DTRR in June In July 2016, the USD(AT&L) approved an update to the DT&E strategy within the TEMP. The program updated the DT&E strategy to account for schedule delays primarily driven by the OEM DT schedule slip and an update to the reliability and maintainability test strategy. In August 2016, the contractor completed line-replaceable units qualification testing. Summary of FY 2016 DT&E Engagement and Assessments The program is executing a medium-risk DT&E schedule driven by highly concurrent test activities. During development of the DT&E strategy, DASD(DT&E) raised concerns about the 110 DASD(DT&E) FY 2016 Annual Report

117 Army BFoV ECP 2 Army s desire to limit tropics natural environment testing to test chambers. In response, and as documented in the TEMP, the program will conduct a review after completion of DT to determine the need for tropical testing in the natural environment. A Government DTRR conducted in May 2016 resulted in a program decision to hold a follow-on DTRR approximately 4 weeks later. This decision was driven by the need to complete analysis and corrective actions on hardware and software issues identified during the contractor testing phase as well as the need for additional time to complete detailed Government test planning. DASD(DT&E) concurred in the program s decision as it would reduce risk to the execution of Government DT&E. The program successfully completed the follow-on DTRR in June All stakeholders concurred that all significant issues identified during the previous DTRR had been addressed and the program was ready to begin Government testing. The BFoV program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The program is effectively managing a DT&E strategy that is medium risk because of concurrency in test execution. The DT schedule has experienced significant delays necessitating an update to the original DT strategy. DASD(DT&E) concurred in the program s decision to delay the start of the Government phase of DT&E because of delays in OEM DT, component qualification testing, correction of deficiencies identified by OEM testing, and incomplete Government test planning. DASD(DT&E) FY 2016 Annual Report 111

118 Indirect Fire Protection Capability Increment 2 Intercept (IFPC2-I) Executive Summary: The IFPC2-I system is a mobile, ground-based weapon system intended to defeat unmanned aerial systems (UAS); cruise missiles (CMs); and rocket, artillery, and mortar projectiles. The system provides 360-degree protection with the ability to engage simultaneous threats arriving from different azimuths. IFPC2-I is an SoS consisting of the following subsystems: existing fire-control sensor (AN/MPQ-64 Sentinel radar), existing interceptor (AIM-9X-2 missile), Army IAMD Battle Command System (IBCS) (in development), and the new (in development) multi-mission launcher (MML). The new MML is developed on a modified existing platform, the Medium Tactical Vehicle 10-ton dump truck (M1157). The IBCS includes the engagement operations center (EOC); all hardware, firmware, and software necessary to provide the primary functions of mission command and communications; search, acquisition, and tracking; target discrimination; and weapon launch, interceptor guidance/control, and target kill assessment. The IFPC2-I capability will be developed in three blocks, with each block a separate acquisition program. Block 1 will use the AIM-9X-2 interceptor, Sentinel radar, MML, and IBCS to support the counter-uas and CM defense missions. The IFPC2-I Block 1 mission is to defend military and geopolitical fixed assets, such as airbases, ports, and mission command nodes, and semi-fixed assets, such as choke points, forward operating bases, forward area refueling/rearming points, and refuel-on-the-move points, against UAS and CMs. The program entered the Technology Maturation and Risk Reduction (TMRR) phase on March 24, During this phase, initial design and all required technical gates, including a preliminary design review, were completed. After establishing an IFPC2-I Integration Lab and developing two MML prototypes, the program demonstrated initial integration in the lab, reducing technical risk. The TMRR phase culminated in an Engineering Demonstration (ED) live fire demonstrating system Technology Readiness Level 6 and meeting the MS A and Acquisition Decision Memorandum (ADM) exit criteria. Lead DT&E Organization: ATEC AFED Summary of FY 2016 DT&E Activities March April 2016, the program conducted an ED at White Sands Missile Range, New Mexico. As the capstone event for TMRR, the objective was to demonstrate all five ADM criteria. o Scenario 1: Demonstration of the baseline interceptor (AIM-9X Block II) in platoon organic configuration against a UAS. o Scenario 2: Demonstration of the baseline interceptor (AIM-9X Block II) in platoon nonorganic configuration against a CM. o Longbow Scenarios: Demonstration of compatibility with multiple interceptors including the Longbow Hellfire, Miniature Hit-to-Kill, and Tamir (Israeli) missiles. 112 DASD(DT&E) FY 2016 Annual Report

119 Army IFPC2-I Summary of FY 2016 DT&E Engagement and Assessments In November 2016, DASD(DT&E) published a DT&E assessment to support the MS B decision scheduled for the 1st quarter FY Based on the TMRR demonstrations results and program planning at the time of the assessment, DASD(DT&E) recommended that the program proceed to MS B and EMD as planned but identified two risks that need to be monitored as the program moves into EMD. o IFPC2-I relies on the Army IBCS as its mission command component. At this point in the IAMD development, the IBCS has not yet satisfied performance and reliability requirements. This dependency introduces high risk into the IFPC2-I development and performance. o The IFPC2-I cybersecurity posture is anchored in the cybersecurity posture of the legacy systems (e.g., Sentinel, IAMD) that comprise the overall system. These legacy systems could introduce vulnerabilities that could compromise or degrade IFPC2-I performance in a cybercontested environment. The IFPC2-I program did not request a waiver or deviation from requirements in the TEMP. DASD(DT&E) Program Assessment In November 2016, DASD(DT&E) assessed system performance, reliability, interoperability, and cybersecurity and published a DT&E assessment to support the MS B decision scheduled for the 1st quarter FY The results of the assessment are summarized below. o Performance. Performance during TMRR is characterized by the successful intercept of two different target classes in two different scenarios and the successful launch of multiple interceptors meeting the MS A and ADM exit criteria. CDD performance requirements were not assessed during the TMRR phase; instead, performance was assessed against five exit criteria outlined in the ADM. Test results show that all five exit criteria were met during the TMRR ED. EMD will evaluate performance in more challenging environments and will test components not evaluated during TMRR including the AIM-9X Missile Data Link and the Small Footprint Radio. o Reliability. RAM was not characterized during TMRR because of significant configuration differences between the TMRR prototypes and the EMD configurations. Post-MS B, RAM will be characterized as an SoS during reliability growth tests and missile flight tests. The system shall achieve a 66 percent probability of completing a 24-hour mission. Evaluating reliability as an SoS is the proper approach but places the program at risk given reliability growth problems associated with IAMD. o Interoperability. Interoperability is characterized by demonstrating proof-of-principle integration with (1) IAMD via the integrated fire control network and (2) multiple interceptors via weapons interface controllers. EMD interface control documentation is established. EMD integration testing will leverage software-/hardware-in-the-loop environments and live search/track tests to evaluate system interfaces before missile flight tests. IFPC2-I will not require joint interoperability certification and does not possess a Net Ready KPP. IFPC2-I will require Army interoperability certification after MS C. o Cybersecurity. The cybersecurity posture of the IFPC2-I program is characterized by the wide breadth of its cyber-attack surface, which is dispersed across multiple systems on the battlefield. A cyber vulnerability to any one of the IFPC2-I components is a risk to all and therefore each component must carefully and independently protect the integrity and availability of its own data as well as the information it exchanges between components. The program has planned a comprehensive set of cybersecurity tests during EMD to include RMF security assessments and cooperative and adversarial tests of the individual components (e.g., MML and the SoS (including the IAMD/EOC, AIM-9X, MML, and Sentinel radar)). The DASD(DT&E) FY 2016 Annual Report 113

120 Army IFPC2-I MML has undergone a basic software assurance test during TMRR and the resultant findings, though numerous, are typical of a development program at MS B. However, the broader cybersecurity posture of the SoS is of concern. The IAMD and the Sentinel radar introduce vulnerabilities that could compromise or degrade IFPC2-I operational missions in a cybercontested environment. Conclusion: Based upon the demonstrations conducted and a review and analysis of the documentation for the IFPC2-I, MML, Sentinel, AIM-9X, and IAMD programs conducted to date, DASD(DT&E) recommended that the program proceed to MS B and EMD as planned. Dependency on IAMD places the IFPC2-I SoS at risk for achieving performance and reliability. Vulnerabilities of the subsystems increase the cybersecurity risk to the overall SoS. 114 DASD(DT&E) FY 2016 Annual Report

121 Joint Air-to-Ground Missile (JAGM) Executive Summary: The JAGM system is a U.S. Army-led, joint interest, ACAT 1D program with common requirements shared with the U.S. Navy (USN) and U.S. Marine Corps (USMC). The JAGM is the next-generation air-to-surface standoff missile designed to replace Hellfire Laser; Hellfire Longbow; and air-launched, tube-launched, optically tracked, wire-guided (TOW) missiles, addressing an impending Army missile inventory expiration. The system will initially be integrated by the U.S. Army onto Apache (AH-64E) helicopters, and then later by the USN/ USMC onto Viper (AH-1Z) helicopters. JAGM will fill a critical gap in capability to engage targets in adverse weather and countermeasure environments, equipping the Warfighter with a fire-andforget capability in day and night battlefield limited-visibility conditions from significant standoff ranges during full-spectrum operations. The JAGM program is divided into three increments. Increment 1 incorporates a dual-mode seeker utilizing a semi-active laser and millimeter-wave radar on an existing Hellfire Romeo back end (motor, control actuators, and warhead) to engage light vehicles, light and heavy armored vehicles, maritime targets, and soft targets out to a range of 8 kilometers, providing point designation, active fire-and-forget, and laser-cueing engagement modes. Each mode is designed to allow the missile to lock-on before launch or lock-on after launch and allow the operator to command low-, direct-, or high-attack missile trajectories before launch. The Army was granted MS B approval in the 4th quarter FY 2015, authorizing entry into EMD. The JAGM PM owns the Interface Control Documents, placing the burden of system integration for threshold platforms on the developer. JAGM will be demonstrated in test from ground-launched and instrumented test aircraft. Because of late maturation of AH-64E system software, DT will not be able to demonstrate full system integration before MS C. The Army intends to support the MS C decision with a limited user test (LUT) to show functionality on AH-64E, using interim software in lieu of fully integrated systems software. To meet the Acquisition Program Baseline target, JAGM will have to complete 38 DT/integrated test shots and 10 LUT shots within 9 months. Lead DT&E Organization: ATEC AFED Summary of FY 2016 DT&E Activities March 28, 2016, the CDR concluded, with the assessment that the design was sufficiently mature to enter the system capability and manufacturing process demonstration phase. May 26, 2016, the program updated the TEMP (Revision 1.4), which was approved for MS B, to incorporate revisions to the DEF. DT to date has included environmental qualification of hardware, HWIL testing, and captive flight tests utilizing the missile seeker mounted in test bed King Air and Sabreliner aircraft. Platform integration on the AH-64E and the M299 launcher included weapons and avionics integration laboratories and was supported by numerous captive-carry flights utilizing inert JAGM all-up rounds. DASD(DT&E) FY 2016 Annual Report 115

122 Army JAGM The Army has completed seven risk reduction ground test launches and a risk reduction airborne test launch from an unmanned air vehicle (MQ-1C). The Army has also completed two risk reduction airborne test launches from an AH-64 Apache. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) provided subject matter experts to assist in developing a comprehensive DEF that satisfies the direction in the MS B Acquisition Decision Memorandum. The program added the revised framework as an addendum to the MS B TEMP following the March 2016 T&E Working Group. DASD(DT&E) provided guidance to the program office to identify the additional production assets to be made available in support of achieving the reliability threshold in accordance with the In-Flight Reliability Assessment Test Matrix. The program office was able to identify additional low-rate initial production and Navy test missiles that could support an expanded reliability growth demonstration if required. DASD(DT&E) participated in the CDR, supporting the decision to move the program forward into the system capability and manufacturing process demonstration portion of the EMD phase. The JAGM program successfully completed its detailed design and was deemed ready to enter system capability and manufacturing process demonstration. DASD(DT&E) initial assessment concluded that all KPPs are achievable. The JAGM program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The program continues testing in accordance with the approved TEMP. DASD(DT&E) assesses current risk to be moderate based upon incomplete integration on AH-64E and an aggressive test schedule. 116 DASD(DT&E) FY 2016 Annual Report

123 Joint Tactical Radio System (JTRS) Handheld, Manpack, and Small Form Fit (HMS) Manpack (MP) Radio Executive Summary: The JTRS HMS program is an ACAT ID MDAP that includes two separate products: the Rifleman Radio (RR) and the MP radio. The MP is a twochannel, software-defined, multiplewaveform, general-purpose user radio capable of supporting mounted and dismounted operations. The MP radio provides line-ofsight and beyond line-of-sight voice and data connectivity primarily for company and below echelons. The MP radio has three configurations: a tactical operations center kit for command posts, mounted configurations integrated into the Army s tactical and combat platforms, and a rucksack configuration to support Army dismounted operations. The MP radio provides the Warfighter with a software-programmable, networkable, multimode radio to implement simultaneous voice, data, and video/imagery communications. The FY 2012 NDAA required that the Secretary of the Army submit to congressional committees written certification that the acquisition strategy for the full-rate production of the HMS program and other covered programs includes a full and open competition (FOC). On February 25, 2016, the Army awarded production contracts to three manufacturers: General Dynamics, Harris Corporation, and Rockwell Collins. Lead DT&E Organization: ATEC C4ISRED Summary of FY 2016 DT&E Activities June 6 September 16, 2016, the ATEC U.S. Army Electronic Proving Ground Mission Command and Network Test Directorate conducted qualification testing to validate the manufacturers ability to meet the minimum functional requirements identified in the Performance Requirements Document and to validate that the radios are capable and ready to move forward into the customer test scheduled between February and April Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) provided four Defense Acquisition Executive Summary assessments to the USD(AT&L) in FY 2016; there were no potential or actual DT-related program problems to report. June 2016, DASD(DT&E) approved the detailed test plan for DT to start testing on schedule. July September 2016, DASD(DT&E) observed 1 week of qualification testing for each vendor to test execution and observe data collection in preparation for making an assessment to support delivery order 1, scheduled for July Qualification test results are competition sensitive. DASD(DT&E) assessed and recommended USD(AT&L) approval of the program s TEMP. The DEF, test schedule, and supporting resources are adequate for T&E. The HMS program did not request a waiver or deviation from requirements in the MP TEMP. DASD(DT&E) FY 2016 Annual Report 117

124 Army JTRS HMS MP Radio Conclusion: The HMS MP Radio program DT&E events are on track. The U.S. Army is executing the FOC MP radio T&E as planned within the TEMP. The Defense Acquisition Executive is scheduled to make a delivery order decision in July 2017 to resource radios for IOT&E in June DASD(DT&E) FY 2016 Annual Report

125 Joint Tactical Radio System (JTRS) Handheld, Manpack, and Small Form Fit (HMS) Rifleman Radio (RR) Executive Summary: The JTRS HMS program is an ACAT ID MDAP that includes two separate products: the RR and the Manpack (MP) radio. The RR is a handheld, single-channel, software-defined radio intended to be used in a dismounted configuration with Type 1 (Suite B) Secret and below encryption. The RR will provide voice and position location information data. The dismount radio package consists of a receiver/transmitter and ancillary items such as a receiver/transmitter case; antennas (L-band, ultrahigh frequency, and/or multiband); batteries with battery charger/cable; headset/microphone/ speaker; and cable assemblies. The RR utilizes only the Soldier Radio Waveform (SRW) as a mobile, ad hoc, networking waveform. The FY 2012 NDAA required that the Secretary of the Army submit to congressional committees written certification that the acquisition strategy for the full-rate production of the HMS program and other covered programs includes a full and open competition (FOC). On April 29, 2015, the U.S. Army awarded indefinite-delivery/indefinite-quantity contracts to Thales Corporation and Harris Corporation for the radios. On September 13, 2016, the Army Configuration Steering Board concurred in deferring procurement activities for the 1-channel RR and moving forward with the 2-channel leader radio. Lead DT&E Organization: ATEC C4ISRED Summary of FY 2016 DT&E Activities October 2015 March 2016, the Mission Command and Network Test Directorate (MC&NTD) of the Electronic Proving Ground (EPG), an ATEC test center, conducted the field portion of the customer test (CT) at Fort Huachuca, Arizona. The MC&NTD conducted performance-based testing to provide range and technical performance data using SRW and to verify technical, performance, and reliability requirements. The CT was developmental-like in scope and facilitated the selection of radios to participate in IOT&E. Each vendor supplied 100 FOC RRs for the CT. ATEC assessed the performance and functionality of each vendor s FOC RRs when used in a controlled field environment. The Army Research Laboratory (ARL) Survivability/ Lethality Analysis Directorate conducted adversarial cybersecurity DT&E during the CT time frame at EPG. October 2015 June 2016, the Communications-Electronics Research, Development, and Engineering Center (CERDEC) conducted the lab-based portion of the CT in addition to risk reduction testing for each vendor s radios at the Unified Lab for Tactical Radios Army (ULTR-A) at Aberdeen Proving Ground, Maryland. Testing evaluated baseline system-level performance of each vendor s FOC RR. Testing utilized approved and fielded U.S. Army Capability Set 15/16 mission plan configurations to ensure that operationally relevant scenarios were tested. May 2016, the Human Research and Engineering Directorate of the ARL conducted an RR Soldier excursion at EPG, Fort Huachuca, Arizona, to assess training adequacy and the RR s DASD(DT&E) FY 2016 Annual Report 119

126 Army JTRS HMS RR human-machine interface and obtain Soldier feedback. The 101st Airborne Division provided infantry and scout Soldiers to execute the test. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) provided four Defense Acquisition Executive Summary assessments to the USD(AT&L) in FY 2016; there were no potential or actual DT-related program problems to report. October 2015 March 2016, ATEC conducted the CT field test. DASD(DT&E) participated in the developmental test readiness review (DTRR) and provided verbal approval of readiness to start record testing. The candidate radios demonstrated the applicable Performance Requirements Document requirements during the qualification test; the Army Evaluation Center provided the necessary safety releases for the vendor radios; operators completed new equipment training for each radio; and ATEC completed data collection, instrumentation, and data reduction validation, verification, and accreditation. October 2015 May 2016, DASD(DT&E) staff evaluated the data of the CT for performance, interoperability, cybersecurity, and reliability against the KPPs and KSAs. Initial results of CT are competition sensitive. October 2015 June 2016, CERDEC conducted the ULTR-A portion of the CT and risk reduction testing for the radios. DASD(DT&E) participated in the DTRR and provided verbal approval of readiness to start record testing after reviewing and commenting on the test plan. The HMS program did not request a waiver or deviation from requirements in the RR TEMP. Conclusion: The program executed CT in accordance with the TEMP. On September 13, 2016, the Army Configuration Steering Board deferred procurement test activities for the 1-channel RR and moved forward with the 2-channel leader radio. This action canceled the scheduled December 2016 Defense Acquisition Executive decision to make a delivery order to resource radios for IOT&E in July The Army will have to execute an approved T&E plan for the 2-channel leader radio to support a competitive delivery in FY DASD(DT&E) FY 2016 Annual Report

127 M109A7 Family of Vehicles (FoV) Self-Propelled Howitzer (SPH) and Carrier, Ammunition, Tracked (CAT) Vehicle Executive Summary: The M109A7 FoV program consists of two individual platforms: an SPH and a CAT vehicle. The SPH is an aluminum armored, full-tracked, 155-millimeter vehicle, capable of carrying a minimum of 39 projectiles and a minimum of 26 modular artillery charge system canisters. The CAT vehicle supplies the SPH with ammunition as it provides tactical and operational fires during offensive and defensive operations. The CAT vehicle will be capable of carrying a 12,000-pound ammunition payload and can be configured for various ammunition needs and specifications. The SPH and CAT vehicle incorporate a newly designed hull, a modified Bradley fighting vehicle (BFV) power train and suspension system, the future BFV track, a modernized 600-volt electrical system, and a microclimatic conditioning system intended to improve sustainability over the current Paladin/field artillery ammunition support vehicle fleet. The SPH also includes an automated fire control system. The primary mission area for the M109A7 FoV is force application-engagement. The M109A7 FoV supports combined arms maneuver, wide-area security, and other full-spectrum operations as part of the land component of a joint task force. The M109A7 FoV is planned to be employed as part of a fires battalion in the armored brigade combat team and the fires brigades, but it will be fully capable of supporting any brigade combat team. Targets include the full range of materiel, personnel, and structures. As an ACAT II program, the Army Acquisition Executive approved entry into EMD in the 4th quarter FY As a result of program restructure and cost increases, the USD(AT&L) designated M109A7 FoV as an ACAT ID program in June The program conducted EMD DT&E from FY 2011 through FY In October 2013, the USD(AT&L) conducted a MS C review and authorized the program to enter the Production and Deployment (P&D) phase and begin low-rate initial production (LRIP). During FY 2014 through FY 2016, production qualification testing (PQT) included post-ms C testing to verify corrective action, producibility improvement, and obsolescence changes; first article testing (FAT); software developmental qualification testing; component-level live-fire and full-up system live-fire testing; a logistics technical manual (TM) review; software formal qualification test (FQT); and PQT. In July 2015, the program slipped the FRP decision as well as conditional materiel release, first unit equipped, and IOC milestones by 2 months to minimize overall program risk. In September 2015, the USD(AT&L) designated the program as an ACAT IC program and delegated the Army Acquisition Executive as the Milestone Decision Authority. The Government phase of PQT originally planned to begin in June 2015, and scheduled to end in July 2016, began in September 2015 and is planned to end in March/April Joint Requirements Oversight Council DASD(DT&E) FY 2016 Annual Report 121

128 Army M109A7 FoV SPH and CAT Vehicle Memorandum , dated December 6, 2016, changed the KPP 5 Rate of Fire requirement from a KPP to a KSA and adds clarifications to various rate-of-fire scenarios. Lead DT&E Organization: ATEC AFED Summary of FY 2016 DT&E Activities In September 2014, the PMO began line-replaceable unit FAT. The FAT, originally scheduled to end in December 2014, was completed in April In the 1st quarter FY 2016, the program completed FQT on software version (SV) This version primarily addressed cybersecurity, vehicle diagnostics, and other issues identified during early PQT. During FY 2016, the Army-conducted PQT included firing and automotive performance, SPH and CAT vehicle automotive durability miles, 19 SPH RAM missions (58.8 miles/104 rounds per mission), full-up system-level survivability testing, cybersecurity testing, continental U.S. (CONUS)-based environmental chamber testing, arctic region testing (Alaska), interoperability testing, and electromagnetic interference and electromagnetic compatibility survivability testing. The program plans to complete CONUS-based environmental chamber testing and corrosion testing in FY Summary of FY 2016 DT&E Engagement and Assessments Government PQT experienced delays because of production and supplier quality assurance/ quality control issues impacting availability of test articles as well as test article configuration management. Retrofits were conducted on test vehicles after delivery to test sites, delaying some Government PQT activities as well as complicating test article configuration management. Examples of challenges experienced during PQT included chrome tube fallbacks with Excalibur and M549 projectiles, cannon out-of-battery incidents, engine overheating, and Dynamic Reference Unit Hybrid Replacement position errors. Of the nine program KPPs, seven apply to the SPH and five apply to the CAT vehicle. DOT&E will assess three SPH KPPs (KPP 7 Howitzer Availability, KPP 2 Force Protection, and KPP 3 Survivability) and three CAT vehicle KPPs (KPP 10 CAT Availability, KPP 2 Force Protection, and KPP 3 Survivability). DASD(DT&E) will assess four SPH KPPs and two CAT vehicle KPPs. Based upon emerging analysis of PQT data, the SPH is currently assessed as meeting three KPPs and not meeting one KPP (KPP 6 Howitzer Reliability). The CAT vehicle is currently assessed as meeting one KPP and not meeting one KPP (KPP 9 CAT Reliability). The SPH reliability requirement is to achieve 75 percent probability of completing an 18-hour combat mission. Emerging PQT data show that the SPH achieved 72.9 percent probability of the SPH completing an 18-hour combat mission (80 percent confidence interval from 57.4 to 84.7 percent) based on a point estimate of 57 hours mean time between system abort (MTBSA). The 57 hours MTBSA is below the planned value of 87 MTBSA on the reliability growth curve. The SPH is not on track to achieve the reliability requirement with confidence by FRP. The CAT vehicle reliability requirement is to achieve 84 percent probability of completing an 18-hour combat mission. Emerging PQT data show that the CAT vehicle achieved 77.2 percent probability of completing an 18-hour combat mission (80 percent confidence interval from 64.8 to 86.6 percent) based on a point estimate of 69.8 hours MTBSA. The 69.8 hours MTBSA is below the threshold value of 103 MTBSA. The Army conducted PQT RAM testing on SPH and CAT LRIP vehicles at the force protection/ survivability Threshold 1 (T1) weight. PQT of the SPH at the force protection/survivability 122 DASD(DT&E) FY 2016 Annual Report

129 Army M109A7 FoV SPH and CAT Vehicle Threshold 2 (T2) and Objective weights shows that the additional weight has no significant impact on SPH and CAT vehicle automotive, reliability, or firing performance. Paladin Integrated Management (PIM) is required to fire all authorized U.S. and NATO standard 155-millimeter projectiles, fuzes, primers, and propelling charges, except for the M712 Copperhead projectile. EMD and P&D DT&E identified issues with the current propellant and primer stocks used by the Army. These issues, not caused by the PIM design changes, negatively impact SPH performance. The M82 primer may deform or rupture when used with the M232A1 5H propellant, requiring the crew to perform additional tasks to extract the primer. This issue adversely impacts the SPH rate-of-fire performance. The PM for Self-Propelled Howitzer Systems (SPHS) implemented new removal tools for stuck and ruptured primer for the program to reduce the impact of stuck or ruptured primer on SPH performance. The PM for Combat Ammunition Systems, in coordination with the PM SPHS, completed an M82 material change program solution to mitigate the occurrence of stuck or ruptured primers. The PM SPHS conducted PQT of upgraded M82 primers in August/September Emerging results indicate that the number of ruptured primers has been reduced, but the number of stuck primers has not been reduced. Recent PQT of the SPH configured with the new chrome tube has revealed fallback (projectile unseats and falls back from its firing position) issues with the M982 Excalibur projectile. The PM SPHS has established a tiger team to investigate the root cause and corrective actions needed to resolve the issue. An interim solution has been identified by modifying operator maintenance and loading procedures. A long-term solution investigation is ongoing. The DT cybersecurity assessment conducted in April May 2015 identified two critical and five high-risk vulnerabilities. The program has identified fixes for one critical and two high-risk vulnerabilities and incorporated the mitigation into SV in time to support IOT&E cybersecurity testing. These fixes to SV were validated during the recent OT cooperative vulnerability and penetration assessment. The other critical vulnerability and one of the remaining high-risk vulnerabilities are projected for remediation at SV 4.x. One high-risk vulnerability finding is still being investigated, and the risk incurred by the final high-risk vulnerability finding is being accepted. These initial findings were assessed as an insider/ nearsider threat as a result of having full access to the vehicle. Additionally, physical security procedures are in place to mitigate risk. Interoperability DT was successfully conducted in May 2016 with no significant issues identified in emerging reports. At the June 2016 Operational Test Readiness Review #2, DASD(DT&E) assessed entry into IOT&E as medium risk given system reliability issues: o SPH and CAT vehicle inconsistent failure modes and SPH legacy cannon failures have the potential to impact test execution. o There is a low probability of SPH demonstrating the reliability requirement with confidence (legacy cannon-related failures). o There is a medium probability of the CAT vehicle demonstrating the MTBSA requirement with confidence (assuming tactical operations center-related failures have been corrected). In November 2016, the program suspended IOT&E because of toxic fumes in the cab of two SPHs during firing. The program is conducting DT to investigate the cause of the toxic fumes; varying propellant charge, angle of fire, hatch configuration (open/closed), and fan settings. Emerging test results indicate that a combination of improper crew procedures, improper maintenance, and legacy breech malfunctions caused the levels of toxic fumes in the impacted SPHs to rise to dangerous levels. Testing also indicated instances of conflicting guidance in the TMs. The program is working to resolve the toxic fumes issue, improve DASD(DT&E) FY 2016 Annual Report 123

130 Army M109A7 FoV SPH and CAT Vehicle breech reliability, update the TMs, and make other SPH/CAT vehicle improvements before the start of IOT&E #2, tentatively scheduled for the 2nd quarter FY The program did not request a waiver or deviation from requirements in the TEMP. Conclusion: Multiple manufacturing process issues caused PQT delays; however, once the vehicles were modified to the correct LRIP configuration, the program conducted FY 2016 DT&E activities in accordance with the approved TEMP. Emerging analysis of PQT data shows that the program is on a path toward not meeting KPP 6 Howitzer Reliability and KPP 9 CAT Reliability. The program is developing a path ahead to address issues identified during PQT and the suspended IOT&E before the start of IOT&E #2 in the 2nd quarter FY DASD(DT&E) FY 2016 Annual Report

131 Mid-Tier Networking Vehicular Radio (MNVR) (AN/VRC-118(v)1) Executive Summary: The MNVR is an ACAT 1D, special interest program managed by the U.S. Army s Project Manager for Tactical Radios under the PEO for Command, Control, and Communications Tactical. The MNVR provides the Soldier with a softwareprogrammable, two-channel, 50-watt maximum, non-developmental item radio, manufactured by the Harris Corporation. The MNVR is designed to provide on-the-move internet protocol capability through simultaneous secure voice and data communications using two thresholdrequired, Government-provided waveforms: Soldier Radio Waveform (SRW) and Wideband Networking Waveform (WNW). The photo above shows a two-channel MNVR installation using the single-channel ground and airborne radio system mount. The program had a MS C review in September On October 3, 2016, the Defense Acquisition Executive (DAE) approved the MS C low-rate initial production (LRIP) for the program to procure 478 additional radios and directed the Army to submit a revised mid-tier radio and waveform plan by January 6, Based on radio performance in the Network Integration Evaluation 16.2 in May 2016, the Army is reevaluating the mid-tier radio requirements against the performance the Army required on this radio. Lead DT&E Organization: ATEC C4ISRED Summary of FY 2016 DT&E Activities June December 2015, the ATEC Electronic Proving Ground (EPG) Mission Command and Network Test Directorate conducted Government Regression Test (GRT) 1.2 at EPG, Fort Huachuca, Arizona. GRT 1.2 was conducted to validate fixes for performance, reliability, interoperability, and cybersecurity issues from the limited user test (LUT). Summary of FY 2016 DT&E Engagement and Assessments June December 2015, DASD(DT&E) and support staff observed GRT 1.2 and used the observations and test report to complete the MS C DASD(DT&E) program assessment. August 2016, DASD(DT&E) provided a DT&E program assessment to the DAE to support the MNVR program s MS C LRIP decision in September The DASD(DT&E) assessment was based on the 2015 GRT 1.1, a 2015 LUT, and the 2015 GRT 1.2. The MNVR program did not request a waiver or deviation from requirements in the TEMP. DASD(DT&E) Program Assessment DASD(DT&E) recommended approval of LRIP for MNVR based upon analysis of completed DT as of May DASD(DT&E) FY 2016 Annual Report 125

132 Army MNVR (AN/VRC-118(v)1) The MNVR program has met 11 of 14 KPPs and KSAs. This modified non-developmental item radio demonstrated near-threshold values for the KPPs and KSAs not achieved. Performance. The MNVR demonstrated simultaneous operations and multichannel route and retransmission with the WNW and SRW. The MNVR met data throughput requirements at 16 kilometers when transmitting point to point. The MNVR did not meet data throughput requirements at 6 and 10 kilometers point to point. The MNVR did not meet the SRW message completion rate (MCR) and throughput requirements for a 40-node network. The radio did demonstrate MCR and throughput requirements for a 33-node network. The accuracy of MNVRgenerated position location information has not been assessed. Reliability. The MNVR demonstrated 1,117 hours mean time between essential function failure and 98 percent operational availability during testing with Soldiers, achieving its sustainment KPP and reliability KSA threshold requirements of 477 hours and 96 percent. Interoperability. MNVR demonstrated interoperability by passing data into, across, and out of the mid-tier tactical internet with Army mission command systems. The program demonstrated host system platform integration on high-mobility multipurpose wheeled vehicles associated with the first unit equipped. Cybersecurity. The MNVR cybersecurity posture has improved throughout pre-ms C testing and the implementation of corrective actions, although vulnerabilities remain. Soldier operators from DT events rated the MNVR excellent for ease of use, adequate to support the mission, and sufficient to meet tactical conditions and environments. Conclusion: DASD(DT&E) recommended approval and the DAE approved the MS C LRIP for the MNVR program to procure 478 additional radios. Additionally, the DAE directed the Army to submit an analysis of alternatives addressing potential waveforms and alternative architectures to meet emerging mid-tier requirements. 126 DASD(DT&E) FY 2016 Annual Report

133 Stryker Family of Vehicles (FoV) Engineering Change Proposal (ECP) Executive Summary: The current Stryker Double V-Hull (DVH) FoV consists of seven variants: antitank guided missile DVH (ATVV), commander s vehicle DVH (CVV), engineer squad vehicle DVH (ESVV), fire support vehicle DVH (FSVV), infantry carrier vehicle DVH (ICVV), mortar carrier vehicle DVH (MCVV), and medical evacuation vehicle DVH (MEVV) on a common vehicle platform. The ICVV with a scout mission equipment package modification, ICVV-S, provides Stryker infantry and cavalry scouts with a unique DVH-based platform in lieu of the M1126 Infantry Carrier Vehicle (ICVV) Commonality reconnaissance vehicle they would have used in flat-bottom hull (FBH)-equipped Stryker brigade combat teams (SBCTs). The Stryker brigade operates across the full spectrum of military operations. The SBCT deploys rapidly, executes early entry, and conducts effective combat operations on arrival to prevent, contain, stabilize, or resolve a conflict through shaping and decisive operations. The Stryker s robust command, control, communications, and computers and network-enhanced situational awareness enable the formation to rapidly respond to situations, ensuring success on the battlefield. The Stryker ECP effort is to restore lost platform capability to support current embedded systems and facilitate integration of technologies currently in development under other current programs of record. The Stryker ECP effort consists of four integrated technologies: a mechanical power upgrade, an electrical power upgrade, a chassis/suspension upgrade, and an in-vehicle network architecture. The Army Acquisition Executive approved and directed the PEO Ground Combat Systems to execute the Stryker ECP efforts. The PEO will conduct three separate production decision point (PDP) reviews. The first two PDP reviews will result in the conversion of one brigade of FBH Stryker vehicles into ECP DVH Stryker vehicles. The conversion from FBH to ECP DVH involves extracting all relevant non-ecp components from an FBH, combining them with the new ECP components, and assembling them on a new DVH hull. PDP 3 will authorize the conversion of three brigades of DVH Stryker vehicles into three brigades of ECP DVH Stryker vehicles. Each PDP will be supported by test data and is basically a milestone that triggers the depicted aggregated materiel buy quantities. In July 2016, PDP 1 resulted in the production contract award that initiated the conversion of at least 130 Stryker FBH vehicles into ECP DVH vehicles. DVH ECP DT, side-by-side test, and live-fire (LF) test results informed this decision. PDP 2 is planned for March PDP 2 will initiate the conversion of at least 123 Stryker FBH vehicles into ECP DVH vehicles. DT and LF test results to date will inform this decision. PDP 3 is planned for the 3rd quarter FY 2018 and is the final check before materiel release of the Stryker ECP conversion. PDP 3 will initiate the conversion of the remaining 70 Stryker FBH ICVV Scout M mm Mounted Mortar Carrier (MCVV) M1134 Anti Tank Guided Missile (ATGMV) M1133 Medical Evacuation Vehicle (MEVV) Common Operating Picture Common Chassis & Drive Train Common KPP s Common Survivability Common TMDE, Spare Parts, Tools & M1132 Engineer Squad Vehicle (ESVV) Skills M1130 Commander s Vehicle (CVV) M1131Fire Support Vehicle (FSVV) DASD(DT&E) FY 2016 Annual Report 127

134 Army Stryker FoV ECP vehicles into ECP DVH vehicles. All follow-on operational test and evaluation and DT results will support PDP 3. Lead DT&E Organization: ATEC MSED Summary of FY 2016 DT&E Activities In September 2015, the program began Government DT&E using 15 vehicles representing the seven Stryker DVH variants. From November 2015 to March 2016, the program conducted extreme cold weather DT&E of the ICVV at the Cold Regions Test Center (CRTC), Alaska. Testing included 3,000 miles of reliability testing. In the 2nd quarter FY 2016, the program conducted cybersecurity DT&E on the ICVV. From June to December 2016, the program conducted extreme hot weather/high humidity DT&E at the Tropic Regions Test Center (TRTC), Panama. July 22, 2016, the Army held PDP 1. As of September 2016, the program completed reliability testing (9,000 miles) of the ICVV at the Aberdeen Test Center (ATC), Maryland. Summary of FY 2016 DT&E Engagement and Assessments In June 2016, DASD(DT&E) published a DT&E assessment to support the PDP 1 decision. Based on the limited DT&E conducted at the time of the assessment and the emerging test data, DASD(DT&E) recommended that prior to PDP 1, the PMO SBCT assess the impact that the performance and cybersecurity deficiencies have on the production configuration and the associated risk to the program. The Stryker FoV ECP program did not request a waiver or deviation from requirements in the TEMP. DASD(DT&E) Program Assessment In June 2016, DASD(DT&E) assessed system performance and published a DT&E assessment to support the PDP 1 decision. The summary of that evaluation follows: o Performance. Emerging results of automotive performance testing (acceleration, maximum and minimum speed, braking, grades, and slopes) showed that the increased weight associated with the ECP has little or no impact over baseline DVH vehicle performance. However, emerging results also show that the system has not yet demonstrated full compliance with several system performance specifications. The impact of these discrepancies is still subject to engineering analysis. o Reliability. At the ATC, the ICVV demonstrated a mean miles between system abort (MMBSA) point estimate of 4,609, or an MMBSA of greater than or equal to 2,154 with 80 percent confidence. At the CRTC, there were no system aborts; thus, a reliability point estimate cannot be calculated, but the determined reliability is an MMBSA of greater than or equal to 1,917 with 80 percent confidence. These numbers compare to the requirement of 1,000 MMBSA at 80 percent confidence and a reliability specification of 2,000 miles. As of August 31, 2016, the program completed and scored about 4,200 miles of the planned 9,000 miles of CVV testing at ATC. For this testing, the demonstrated reliability point estimate is 1,042 MMBSA, or an MMBSA of greater than or equal to 620 with 80 percent confidence. The ICVV has also conducted approximately 700 miles of the planned 3,000 miles of testing at the TRTC, and the ESVV has begun its planned 9,000 miles of testing at the Yuma Test Center, Arizona. 128 DASD(DT&E) FY 2016 Annual Report

135 Army Stryker FoV ECP o Cybersecurity. Cybersecurity DT on the ICVV identified six very high risk vulnerabilities and one high-risk vulnerability. The system has not yet demonstrated compliance with the system performance specification. Cybersecurity testing is planned for the CVV, MCVV, ATVV, and FSVV in FY The program is developing a plan to resolve the vulnerabilities identified during DT in coordination with DASD(DT&E). Based on the limited DT&E conducted at the time of the assessment and the emerging test data, DASD(DT&E) recommended that prior to PDP 1, the PMO SBCT assess the impact that the performance and cybersecurity deficiencies have on the production configuration and the associated risk to the program. Conclusion: The PM is effectively managing a DT&E strategy that is medium risk because of concurrency in test execution. The DT schedule has experienced minor delays because of correction of deficiencies identified during original equipment manufacturer testing, but DT scope is in accordance with the approved TEMP. DASD(DT&E) FY 2016 Annual Report 129

136 This page intentionally blank. 130 DASD(DT&E) FY 2016 Annual Report

137 6.3 Navy Programs This section includes summaries of the following 11 programs: CH-53K Heavy-Lift Replacement Helicopter COLUMBIA-Class Submarine Distributed Common Ground System Navy (DCGS-N) Increment 2 GERALD R. FORD Class Nuclear Aircraft Carrier (CVN 78) Littoral Combat Ship (LCS) and Mission Packages (MPs) MQ-4C Triton Unmanned Aircraft System (UAS) Multi-Mission Maritime Aircraft (P-8A Poseidon) Trident II Life Extension (D5LE) U.S. Navy Integrated Air and Missile Defense (IAMD) VIRGINIA-Class Submarine ZUMWALT-Class Destroyer (DDG 1000) DASD(DT&E) FY 2016 Annual Report 131

138 CH-53K Heavy-Lift Replacement Helicopter Executive Summary: The CH-53K is intended to replace the CH-53E to meet the Marine Corps heavy-lift requirements. Expeditionary Force 21 (EF 21) and the Marine Corps Service Campaign Plan outline how the Marine Corps will prepare for and execute the missions required to support the United States and its allies as the Nation s expeditionary force in readiness. Both of these capstone documents contain tenets put forth in Operational Maneuver from the Sea (OMFTS) and Ship-to-Objective Maneuver (STOM). The CH-53K will provide the heavy-lift payload, speed, and endurance required to implement the principles espoused in OMFTS and STOM in the execution of EF 21-aligned missions. The CH-53K incorporates composite material construction, a new engine design, digital instrument and gauge displays (i.e., a glass cockpit), and fly-by-wire flight controls among other advanced technologies designed to improve performance within the same overall aircraft footprint. It is intended to provide improvements in operational capability, interoperability, reliability, and maintainability while reducing total ownership costs. The CH-53K assault transport helicopter will be a dual-piloted, multiengine helicopter designed to meet the emerging Marine airground task force vertical heavy-lift, warfighting requirements. In December 2005, OSD approved MS B and authorized the CH-53K program to begin system development and demonstration. The program conducted a CDR in July In January 2012, OSD approved Revision 1 to the CH-53K Acquisition Strategy. This revision shifted the four RDT&E-funded aircraft in the first LRIP lot to the EMD phase as system demonstration test articles (SDTAs) to demonstrate manufacturing processes and support integrated T&E. This revision also increased the total procurement from 156 to 200 aircraft and aligned the program with changes to the schedule and budget that had occurred since the start of the program in Revision 2 to the Acquisition Strategy, approved by the USD(AT&L) in April 2013, moves MS C from the 4th quarter FY 2015 to the 4th quarter FY 2016 and restructures the LRIP plan, reducing the number of LRIP articles from 27 to 20 but adding two more RDT&E-funded SDTA assets during EMD. Revision 3 to the Acquisition Strategy (now in staffing) aligns the program with the Capability Production Document (CPD). The MS C TEMP establishes plans for the Production and Deployment phase of the program and captures changes to the defense acquisition program since program initiation. The current PM estimate for MS C is between March and April The test strategy relies on four engineering development model (EDM) aircraft to support DT&E and the SDTA aircraft that will support later DT efforts and IOT&E. The first of four EDM aircraft had its first flight on October 27, 2015; the remaining EDMs started flying throughout 2016 with EDM-4 entering test and flying in late August In September 2016, the PM enacted three updates to the PM test strategy that will improve testing throughput and help ensure completion of the test program in time to support the late 2019 IOC. All four EDMs are being fully instrumented and equipped for test and are expected to be available for the full range of flight testing in FY The SDTAs will be brought into the test program starting in the summer of 2017, and up to 60 additional personnel (both Navy and Sikorsky) will be hired to enable full use of the additional test assets. 132 DASD(DT&E) FY 2016 Annual Report

139 Navy CH-53K Lead DT&E Organization: NAWCAD HX-21 Summary of FY 2016 DT&E Activities In FY 2016, DT&E focused on envelope expansion on the EDM aircraft and completing risk reduction activities on the ground test vehicle (GTV) to include a full teardown and inspection. The GTV is a full-up aircraft, fixed to the ground, which allows the test team to fly the aircraft at high power levels while not leaving the ground. Using the GTV is a prudent risk reduction measure that will enable the program to make early discovery of major issues in advance of the flight test program. All four EDM aircraft entered test and started accumulating hours and completing test points in FY The program completed envelope expansion and required test points to allow the start of the OT-B1 Operational Assessment (OA) in early October Ground and air DT tasks (required envelope expansion, demonstration of OA flight performance tasks, etc.) were completed ahead of the OT-B1 OA event. The OT-B1 OA was successfully completed on October 19, As of October 29, 2016, the program has flown more than 200 hours on the EDMs and accumulated more than 412 hours on the GTV. SDTA aircraft 1 4 continued production, with SDTA-1 due to be delivered in the summer of OSD conducted a long-lead DAB in March 2016 to review the Navy s request to begin long-lead procurement for SDTAs 5 and 6. Summary of FY 2016 DT&E Engagement and Assessments In February 2016 and again in November 2016, DASD(DT&E) and the Deputy Assistant Secretary of Defense for Tactical Warfare Systems conducted program reviews to assess program development progress and risk and to inform the USD(AT&L) on the risk and recommended timing of the MS C DAB. Based on progress to date and recent demonstrated flight testing throughput, DASD(DT&E) assesses that the PM is on track to meet pre-ms C requirements by March/April Despite schedule pressures, the ITT is determined to fly all critical planned pre-ms C test points, which could impact the MS C date. DASD(DT&E) recommends that the Navy execute an eventbased schedule that conducts all planned testing and avoid deferring critical testing or capability beyond MS C to maintain schedule. The PM has documented changes to the approved plan, including the plan to assess cybersecurity vulnerabilities before MS C, and identified when deferred testing will be completed. Cybersecurity testing that was inserted into the program is now under way in support of MS C. The program updated the TEMP for MS C, and it is in final staffing. This TEMP complies with current guidelines and includes cybersecurity testing as well as testing in line with the new CPD. The CH-53K program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The program conducted FY 2016 DT&E activities in accordance with the approved TEMP, and additional testing (cybersecurity) has been inserted into the program schedule. As expected, DT has identified developmental issues with the system, and the program is taking aggressive action to correct the problems before MS C. Because of these issues, the pace of flight testing has lagged behind plan, but has recently accelerated. From a DT perspective, the program is on track to satisfy its MS C entrance criteria in the March/April 2017 time frame. DASD(DT&E) FY 2016 Annual Report 133

140 COLUMBIA-Class Submarine Executive Summary: The COLUMBIA-Class program is the follow-on fleet ballistic missile submarine (SSBN) class of submarines designed to replace the OHIO-Class SSBN and deploy the existing Trident II D5 life-extended submarinelaunched ballistic missile. The MS B DAB was completed on November 4, 2016, and when the Acquisition Decision Memorandum (ADM) is signed by the USD(AT&L), the COLUMBIA- Class program will be an ACAT 1D MDAP. The OHIO-Class SSBN fleet of 14 submarines will retire at the rate of one per year beginning in FY 2027, and the COLUMBIA-Class submarine will reach IOC and conduct its first patrol in the 1st quarter FY The mission of the COLUMBIA-Class SSBN force is to maintain an appropriate state of readiness to assist in deterring nuclear attack on the United States and its allies. The COLUMBIA-Class SSBN force of 12 submarines will provide a survivable sea-based strategic deterrent in the 2030 to 2080 time frame. The COLUMBIA-Class SSBN must be capable of launching missiles against preplanned or adaptively planned targets. It does not have a requirement for other missions or capabilities unrelated to survivable strategic nuclear deterrence. The United Kingdom (UK) is replacing its SSBN force in conjunction with the COLUMBIA-Class program. The COLUMBIA-Class program is leading a collaborative program to design a common missile compartment (CMC) for U.S. and UK SSBNs under the Polaris Sales Agreement. The risk mitigations for this effort include the Surface Launch Test Facility under construction at China Lake, California, and the Strategic Weapons System (SWS) Ashore Test Facility under construction at Cape Canaveral, Florida. These test facilities are scheduled to be operational in FY 2017 and FY 2020, respectively. The USD(AT&L) approved the COLUMBIA-Class Acquisition Strategy in January 2016 to build 12 submarines in three block procurements of two, three, and seven. The first procurement buys longlead materials with advanced procurement in FY 2019, the lead ship in FY 2021, and the second ship in FY This strategy is coordinated with the VIRGINIA-Class SSN Acquisition Strategy to align shipbuilding across the nuclear submarine shipbuilding enterprise. The Navy utilizes a Flag Oversight Committee/Steering Group that meets monthly to provide executive oversight and coordination for the program. Lead DT&E Organization: PMS DASD(DT&E) FY 2016 Annual Report

141 Navy COLUMBIA-Class Submarine Summary of FY 2016 DT&E Activities The program continued shaft and propulsor bearing testing at the Naval Research Laboratory Center for Corrosion Science and Engineering located at the Marine Corrosion Facility, Key West, Florida. SSP continued construction of the SWS Ashore Test Facility in Cape Canaveral, Florida, and is on schedule to commence testing of an OHIO-Class SSBN missile tube in test bay #1 in January 2017 in support of the OHIO-Class and COLUMBIA-Class SSBN. The program continued free-running model testing, captive model testing, and rotating arm testing at the Naval Surface Warfare Center, Carderock Division (NSWCCD) to support detailed design of the COLUMBIA-Class X-stern and fairwater control surface configurations. The program continued resistance and powering tow tank testing at NSWCCD to measure COLUMBIA-Class total ship resistance, provide shaft revolutions per minute and torque predictions, and provide propulsor data to enable design and production of the COLUMBIA- Class propulsor configurations being tested at the NSWCCD Acoustic Research Detachment (ARD), Bayview, Idaho. The program continued propulsor testing using the Very Large Test Apparatus at the Pennsylvania State University Applied Research Laboratory Garfield Thomas Water Tunnel, State College, Pennsylvania. Test data inform COLUMBIA-Class propulsor configurations tested at the ARD. In January 2016, the Milestone Decision Authority authorized release of the design RFP. In January 2016, the program office and NAVSEA technical community conducted a DEF workshop with DASD(DT&E) staff to establish a COLUMBIA-Class SSBN DEF that was included in the MS B TEMP. In February 2016, the program office provided a draft TEMP for review and comment. In May 2016, the program office released the TEMP for stakeholder signatures. In May 2016, the program completed Phase 3 of the Ship Control Station (SCS) Concept of Operations Exercise at the SCS prototype facility in Groton, Connecticut. Results show that COLUMBIA-Class SCS arrangements, screen design, functionality, human factor elements, and pilot/copilot paradigm support COLUMBIA-Class SSBN submarine operations. In June 2016, the program achieved technical authority capability of the Siemens Team Center Integrated Product Development Environment electronic design disclosure capability. In July 2016, the Navy approved for continued development the first draft of the Integrated Enterprise Plan (IEP) that aligns nuclear shipbuilder contractor and Government plans to enable delivery of COLUMBIA-Class on schedule and within budget without adversely affecting the VIRGINIA-Class SSN or CVN program. The IEP baselines plans for acquisition, build, resources, facilities, material, and funding at both shipyards. The IEP is being updated as the COLUMBIA-Class design matures and ensures that risks are identified and mitigated across the Government and contractor enterprise. In July 2016, the USD(AT&L) chaired the MS B OIPT. In August 2016, the program approved the stern area arrangements. In August 2016, the program commenced construction of the prototype CMC first article quad pack (FAQP) prototype of missile tubes. In September 2016, the contractor completed system-level qualification testing of the robotic cut/bevel capabilities needed to automate steel processing in support of the FAQP prototype. DASD(DT&E) reviewed the COLUMBIA-Class TEMP in February 2016 and provided comments. These concerns were resolved in October 2016 and OSD approved the TEMP and DASD(DT&E) FY 2016 Annual Report 135

142 Navy COLUMBIA-Class Submarine COLUMBIA-Class live-fire test and evaluation (LFT&E) alternative testing strategy and the fullup system-level testing waiver to support MS B in November In October 2016, the USD(AT&L) conducted the annual interim progress review (IPR) at the General Dynamics Electric Boat (EB) facilities in Connecticut and Rhode Island. This review demonstrated that EB has made significant effort to control current and future costs of the program and to mitigate schedule risk. In CY 2016, EB and the program office completed the planned sequential installation and certification testing of three of the seven large industrial fixtures to build the cylindrical hull portions of the COLUMBIA-Class forward compartment, engine room, and CMC with missile tubes. Construction of the FAQP prototype commenced in August When complete in August 2018, the FAQP will allow the shipbuilder to verify that all design requirements are met and that manufacturing processes are proofed. This is a significant risk reduction effort for the construction schedule. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) has been fully engaged with the program office as the program completed the COLUMBIA-Class preliminary design and planned for transition to detailed design following MS B. DASD(DT&E) attends quarterly design reviews to keep track of progress and challenges. Test results and supporting documents are provided upon request. DASD(DT&E) reviewed the draft detailed design RFP and recommended that the USD(AT&L) authorize its release at the December 2015 RFP Release DAB. DASD(DT&E) participated in several T&E WIPTs and numerous COLUMBIA-Class LFT&E working group meetings. In May 2016, the PDR was completed with concurrence of DASD(SE) and DASD(DT&E). DASD(DT&E) participated in the MS B OIPT on July 28, 2016, and recommended proceeding to MS B. In July and August 2016, DASD(DT&E) assessed COLUMBIA-Class program performance, interoperability, reliability, maintainability, and cybersecurity in support of the MS B decision. In September 2016, DASD(DT&E) provided a DT&E assessment recommending that the USD(AT&L) authorize the COLUMBIA-Class program to enter the EMD phase of acquisition based on DT planning and testing completed through the end of August The DASD(DT&E) assessment informed the October 4, 2016, annual IPR and concluded the following: o Post-MS B DT planning and resources support COLUMBIA-Class SSBN program efforts to mature an integrated submarine final design. o Demonstrated DT results and planned DT support entry into detailed design. o Program schedule risk is moderate and is being proactively mitigated by implementation of a Submarine Unified Build Strategy (SUBS). The SUBS effort, supported by the IEP, is developing a joint strategy for the procurement of the VIRGINIA-Class SSN and the upcoming procurement of the COLUMBIA-Class SSBN. The end result will be alignment of COLUMBIA-Class SSBN and VIRGINIA-Class SSN procurements to achieve best cost while continually assessing the nuclear shipbuilding industrial base. SUBS considers shipbuilder construction capabilities and capacities, schedules, contracting approaches, crossprogram budget strategies, funding, legislative authority, material availability, and supplier capacity and capability. o DT planning and testing are on schedule and informing the higher risk stern area design. 136 DASD(DT&E) FY 2016 Annual Report

143 Navy COLUMBIA-Class Submarine o DT&E interoperability planning is adequate. In May 2016, the PEO approved the Common Submarine Interoperability Support Plan. o COLUMBIA-Class reliability modeling and analysis plans are adequate to support development of a mature submarine design. o Cybersecurity test planning is adequate, continues to mature, and is informing the design effort. Program office efforts are focused on planning and executing cybersecurity T&E across the COLUMBIA-Class architecture s six enclaves. The program office has established a Cybersecurity Execution Management Plan and commenced Tabletop Mission Cyber Risk Assessments. The program office has engaged with the DASD(DT&E) Cybersecurity Technical Director, the NCR, and representatives from SCC LANT to increase the fidelity of DT planning for cybersecurity. On November 4, 2016, the Milestone Decision Authority conducted MS B at the Washington Navy Yard. The Milestone Decision Authority is expected to sign the ADM in early January The COLUMBIA-Class integrated product and process development design contract is planned for award in early CY The Surface Launch Test Facility at China Lake, California, will complete construction in early 2017 and the first COLUMBIA-Class missile tube evaluation launch is scheduled for June COLUMBIA-Class Integrated Power System (IPS) prototype component manufacturing commenced in IPS testing at the Compatibility Test Facility, NSWC Philadelphia Division, is scheduled in 2018 and The COLUMBIA-Class program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The COLUMBIA-Class SSBN program completed the Technology Maturation and Risk Reduction acquisition phase and will enter the EMD phase upon signature of the MS B ADM in January The COLUMBIA-Class program has moderate technical and schedule risks that are being closely managed by senior Navy leadership and being monitored by senior DoD leadership. DASD(DT&E) FY 2016 Annual Report 137

144 Distributed Common Ground System Navy (DCGS-N) Increment 2 Executive Summary: DCGS-N Increment 2 enables net-centric compatibility and interoperability among afloat, ashore, and national intelligence nodes; the joint service DCGS family of systems; the larger intelligence community; and users of the DoDIN; and provides intelligence data to coalition members. Users employ advanced analytic tools and automated processes that enable them to efficiently respond to increased data availability, reduce the time it takes to interpret and assess data, and produce actionable intelligence. DCGS-N Increment 2 is an ACAT IAM program that attained MS A approval in March 2016 and MS B approval in November The PMO plans for agile software development and follows the DoD acquisition IT Box model to manage requirements and deliver annual fleet capability releases (FCRs). Lead DT&E Organization: SSC LANT Summary of FY 2016 DT&E Activities During FY 2016, DASD(DT&E) teamed with the PMO to craft the program s TEMP and DEF. The USD(AT&L) waived the regulatory MS A and MS B approved TEMP requirement. June 12 16, 2016, DCGS-N Increment 2 completed FCR-0, the Naval Integrated Tactical-Cloud Reference for Operational Superiority (NITROS) risk reduction effort to reduce cloud computing infrastructure and services technical risks; FCR-0 was executed at a San Diego, California, laboratory-based application integration system integration test (AI SIT). July 2016, the Navy s TRIDENT WARRIOR 2016 (TW16) exercise provided for an early Navy assessment of DCGS-N Increment 2 work procedures, usability, reliability, and availability. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) performed continuous engagement, which included conducting DEF working groups, assisting in development of the program s cybersecurity T&E strategy, providing acquisition document reviews, and reviewing the program s draft TEMP. The DASD(DT&E) 2nd quarter FY 2016 DT&E assessment supported the program s March 2016 MS A: (1) DASD(DT&E) assessed the program s seven draft KPPs to be measurable and testable; (2) the draft reliability growth and software maturity strategies were presented in the draft MS A TEMP and Systems Engineering Plan; (3) most of the systems interfaces are understood and employ stable commercial or military standards for data exchange, with many previously integrated with Increment 1; (4) the PMO performed planning for the four DT&E cybersecurity phases and partially accomplished the first phase, Understand Cybersecurity Requirements. The program had a plan to identify and plan for testing cybersecurity requirements by MS B. The DASD(DT&E) 4th quarter FY 2016 DT&E assessment supported the program s November 2016 MS B. The DT&E assessment focused on risk reduction activities during AI SIT and TW16. DASD(DT&E) recommended DCGS-N Increment 2 entry into the EMD phase based 138 DASD(DT&E) FY 2016 Annual Report

145 Navy DCGS-N upon analyses of developmental planning and risk reduction activities: (1) EMD risk reduction planning was adequate to evaluate system functionality and technical parameters before the FCR-1 limited fielding decision; (2) DCGS-N Increment 2 testing of a prototype release in the Technology Maturity and Risk Reduction phase reduced most performance risk and provided a risk mitigation basis, focused on a cybersecurity-resilient Cross Domain Solution (CDS) and on level-2 data fusion; (3) the program s reliability growth and software maturity strategies were adequate for a software MAIS program; (4) DCGS-N Increment 2 interfaces are understood and employ most of the same DCGS-N Increment 1 stable commercial or military standards for data exchange; (5) program DT&E cybersecurity planning continues to be on schedule, and the program has completed cybersecurity DT&E Phase 1 requirements. DASD(DT&E) DEF Core Team engagement began in the 1st quarter FY 2016 and continues with quarterly DEF reviews. The DCGS-N program did not request a waiver or deviation from requirements in the draft TEMP. DASD(DT&E) Program Assessment The DT&E strategy provides an adequate DT&E plan that supports agile software development and obtaining FCR-1 verification data. DT&E planning is supported by the PMO s DEF and cybersecurity RMF efforts. The PMO plans for a high level of rigor by executing STAT during DT&E. Program integration of DCGS-N Increment 2 FCR-0 with the NITROS project for cloud computing afloat constructs with AI SIT and TW16 testing reduced process and performance risks. The PMO was left with CDS and data fusion risks at MS B. The PMO s reliability growth and software maturity strategies are adequate, but reliability and maturity prediction methods are not leveraged. DCGS-N Increment 2 reliability growth tracks software application faults and measures software failure rates during FCR development and test events. Most DCGS-N Increment 2 interfaces are understood and employ the same DCGS-N Increment 1 stable commercial or military standards for data exchange. The PMO continues to pursue potential CDS options for FCR-1. DT&E cybersecurity planning is on schedule and is in accordance with current OSD policies, guidance, and directives. DT&E strategy planning traces to Joint Capabilities Integration and Development System requirement documents, and the strategy is traceable to the program s DEF, correlating allocated requirements with test objectives, test environments, and test resources. Conclusion: The PMO s draft TEMP provides a DT&E plan that provides for agile software development and obtains FCR-1 verification data. DT&E planning is supported by a DEF and cybersecurity RMF efforts and reflects risk reduction activities during AI SIT and TW16. All KPP minimums are planned to be met by FCR-2, and this release is planned for IOT&E aboard a forcelevel platform. The PMO is planning for a high level of T&E rigor by executing STAT. DASD(DT&E) FY 2016 Annual Report 139

146 GERALD R. FORD Class Nuclear Aircraft Carrier (CVN 78) Executive Summary: The future aircraft carrier GERALD R. FORD class (CVN 78) is the planned successor to the NIMITZ-class (CVN 68) aircraft carrier. It is a large-deck, nuclear-powered aircraft carrier designed to increase the sortie generation capability of embarked aircraft, improve weapon handling efficiency, and increase self-defense capabilities. This report contains an overview of mission-critical systems across the air operations and combat systems mission areas. DASD(DT&E) has expressed concerns with the successful execution of the shipboard test program, which is the principal construction risk on CVN 78 when new design systems are energized for the first time. The current program of record delivery date of the ship is April This date is a 13-month delay from the FY 2015 report because of technical issues and delays in the CVN 78 shipboard test program. As a result of these technical issues and delays, the USD(AT&L) commissioned an Independent Review Team (IRT) to provide a comprehensive review of the CVN 78 program. The review included an assessment of the following systems: Power Generation, Electromagnetic Aircraft Launch System (EMALS), Advanced Arresting Gear (AAG), Dual-Band Radar (DBR), and Advanced Weapons Elevator (AWE). The IRT completed its review of the five systems in December Lead DT&E Organization: PMS 378T Summary of FY 2016 DT&E Activities CVN 78 construction and testing continued at Huntington Ingalls Industries in Newport News, Virginia. The delivery date is expected to be April The focus of air operations in FY 2016 was the EMALS program, which has completed onboard testing of the four catapults and all dead-load testing until the ship goes to sea for further testing. The AAG program is less mature than EMALS because of unexpected component failures and software control problems that occurred earlier in the system s development, especially for offcenter and skewed arrestments. In 2016, the AAG program began testing with actual aircraft, which is crucial to DT, at the Runway Arrested Landing Site (RALS) at the Naval Air Warfare Center, Lakehurst, New Jersey. As of December 2016, for the F/A-18E/F, 261 roll-in arrestments and 39 fly-in arrestments have been completed. All available test events are completed and the Air Recovery Bulletin (ARB) generation commenced in January 2017 with an expected release date of March Testing for all other aircraft types (E-2, C-2, T-45, etc.) will occur later to ensure that all types can be operated during IOT&E. Combat system testing continued at the Land-Based Test Site (LBTS) at the Surface Combat Systems Center, Wallops Island, Virginia. In FY 2016, four land-based engineering tests and DTs were performed from December 2015 through August These test events revealed issues with short-range clutter, false tracks, and dual tracks (in which only one target was present). Each event revealed steady improvement over the previous events, with corrections focused on the interaction of the Ship Self-Defense System (SSDS), Cooperative Engagement Capability (CEC), and DBR; however, discovery of software and integration issues in the system is still occurring. The final land-based tracking test is scheduled later in Onboard CVN 140 DASD(DT&E) FY 2016 Annual Report

147 Navy CVN 78 78, testing is ongoing but is behind schedule because of delayed installation of combat system cabling. The Combat System Trial Rehearsal (CSTR) completed in the 2nd quarter FY CSTR is the integrated demonstration of the combat systems to include tracking aircraft, which is a best-practice risk reduction event before the ship goes to Builder s Sea Trials (BST). Summary of FY 2016 DT&E Engagement and Assessments Sortie Generation Rate (SGR). The SGR KPP drove the redesign of many legacy aircraft carrier fixtures such as the island, weapons elevator, fuel station locations, and ordnance handling paths. Initially, the Virtual Carrier Model (VCVN) was developed by the shipbuilder to predict SGR performance, but it was canceled because it did not provide an accurate prediction of SGR. To replace the VCVN model, NAVAIR developed the Sea Strike, Sea Basing Aviation Model that will be used for analysis of SGR in support of CVN 78 DT. DT started in the 1st quarter FY 2016 and is expected to complete in the 4th quarter FY M&S results indicate that the ship, its redesigned flight deck, and improved ordnance storage and handling systems have the physical capability to meet the SGR KPP if supporting systems (AWE, EMALS, AAG) meet their performance and operational availability (Ao) requirements and if the crew is proficient. To date, there have been reliability issues with EMALS, AAG, and DBR, and the Navy is addressing the issues. The ship is unlikely to meet the full SGR KPP at IOC but will grow toward the capability as the crew gains experience and as system reliability and Ao improve. Power Generation Main Turbine Generator (MTG). Four MTGs provide the ship s electrical power. In June 2016, a failure of the number 2 MTG occurred, requiring replacement of the rotor. The problem was identified as a defect in the vendor s voltage regulator system sensing component. The vendor has corrected the voltage regulator defect, and the number 2 MTG rotor was replaced and testing resumed. Also, in July 2016, a failure of the number 1 MTG occurred because of an overvoltage event of the unit. The problem was identified as a design deficiency in the voltage regulating and protection system. The design deficiency was identified and corrected in the number 4 MTG and is being corrected in MTGs 2 and 3. Number 1 MTG unit repairs will be completed during the Post-Shakedown Availability (PSA) in late Until then, the ship will operate at sea using three of the four MTG units (numbers 2, 3, and 4). EMALS. Performance land-based testing previously demonstrated that the major EMALS components (armatures, bogie wheels) can meet performance specifications. The LBTS completed more than 3,600 dead-load launches and 452 aircraft launches. EMALS certification testing is complete and the final system software requirement verification is under way. EMALS has progressed and confidence in system performance has risen in However, the statistical reliability demonstrated at the LBTS was well below the system s contractual technical specification. Ao depends not only on component reliabilities but also on repair and logistics delay times. Certain EMALS electronic components appear amenable to rapid repair or rapid software fault reset, which could offset a significant portion of the less-than-expected component reliability according to analysis by NAVAIR. Onboard CVN 78, the 2016 effort focus was on the completion of installation and testing of the four catapults. The test team conducted a combination of 1,885 armature motions including 242 dead-loads using the four catapults. In 2017, further testing will continue when the ship goes to sea during Independent Steaming Events after delivery. Data from shipboard underway testing in 2017, follow-on land-based testing, and ongoing maintenance demonstration activity as well as crew training will enable a more definitive estimate of Ao in AAG. The AAG program is less mature than EMALS because of unexpected component failures earlier in the program and software control problems that were found during off-center and skewed arrestments. In March 2016, AAG conducted its first roll-in arrestment of an F/A-18E at DASD(DT&E) FY 2016 Annual Report 141

148 Navy CVN 78 RALS. Testing continued in earnest in August 2016 and completed the F/A-18E/F aircraft performance test program for the limited envelope ARB in December Resolving system design and performance issues delayed the start and completion of the aircraft performance testing at RALS; however, 261 roll-in arrestments and 39 fly-in arrestments have been conducted in support of the ARB generation. The other types of aircraft (C-2A, E-2D, F/A-18C/D, remainder of F/A-18E/F, and T-45) will be tested in later 2017 and Through analysis and testing, the program office has identified areas of improvement to include redesign of some components including the water twister, dynamic control system, and cable shock absorber. These redesigned components are funded and planned to be incorporated onboard CVN 78 during the PSA and the Planned Incremental Availability. Onboard the ship, system installation, testing, and certification continue. AWE System. There are 11 AWEs on CVN 78. The shipbuilder underestimated the complexity of the system, so the shipbuilder decided not to build an AWE LBTS facility. With numerous installation and testing issues, the shipbuilder and the prime vendor formed a red team and have defined the software problem, supply deficiencies, and redesign. However, at ship delivery, only two AWEs will be fully operational. The remaining nine AWEs will be completed and operational by the end of the PSA. DBR. The DBR, unique to CVN 78, is composed of the S-band Volume Search Radar (VSR) and the X-band Multifunction Radar (MFR). Shipboard DBR testing progress to date is slower than planned because of first-time integration issues. All six DBR arrays (three MFR arrays and three VSR arrays) have been successfully energized and radiated at full power. Targets of opportunity have been tracked across adjacent MFR and VSR array faces radiating at high power demonstrating functionality. However, VSR has radiated at low power because of port radiating restriction. DT is continuing at the Wallops Island LBTS. However, because of the single-faced arrays (only one production MFR array and one VSR array, which is an engineering development model but not representative of the ship s VSR), certain functions, such as tracking through the corners, can be tested aboard CVN 78 only at sea. Air Traffic Control (ATC) is performed with the AN/TPX-42A Direct Altitude and Identity Readout system integrated with the DBR; OE-120/UPX Identification, Friend or Foe system; dual AN/UPX-41 digital interrogators; and the navigation data network. ATC capabilities were demonstrated at the LBTS in 2016, and tracking quality improved but still exhibited short-range (less than 12 miles) false tracks and dual tracks in which only one target was present. In addition, combat systems testing involving the DBR, SSDS, and CEC was conducted in conjunction with ATC at the Wallops Island LBTS. Again, although tracking quality improved through 2016, the latest events still exhibited spurious short-range false tracks, dual tracks in which only one target was present, and track jumps from one position to another. The Navy is correcting the issues, and final testing will complete in Onboard the ship, CSTR and ATC Trial Rehearsal using Lear Jets will be conducted before BST. Though the events demonstrate functional performance, the results should identify any additional significant issues with the MFR and VSR production units integrated with the CEC, SSDS, and AN/TPX 42A. Conclusion: CVN 78 is a very complex weapon system of systems; it is the first of the GERALD R. FORD class and includes numerous new design critical systems (EMALS, AAG, and AWE) that replaced legacy steam and hydraulic systems as well as new power generation design and a dualband phased-array radar replacing legacy rotating antennas. Construction delays and the shipboard test program, including the above-mentioned systems, have progressed slower than planned, which has led to more than a year delay in the delivery of the ship. DASD(DT&E) is closely monitoring 142 DASD(DT&E) FY 2016 Annual Report

149 Navy CVN 78 the shipboard test program for any additional impacts on ship schedule including Full-Ship Shock Trials, which are required before initial operational deployment. DASD(DT&E) FY 2016 Annual Report 143

150 Littoral Combat Ship (LCS) and Mission Packages (MPs) Executive Summary: LCS consists of three major programs: Seaframes (SFs) (PMS 501), Mission Modules (MMs) (PMS 420), and Logistics/Sustainment (PMS 505). The SFs (ships) program consists of two high-speed hull types: USS FREEDOM (LCS-1) variants are steel monohulls with aluminum superstructures, and USS INDEPENDENCE (LCS-2) variants are an all-aluminum tri-hull design. Odd-numbered SFs are FREEDOM variants, and even-numbered SFs are INDEPENDENCE variants. Combined diesel and gas turbine waterjets propel each ship. The MMs program is responsible for procuring, integrating, testing, and delivering three types of MMs: mine countermeasures (MCM), surface warfare (SUW), and antisubmarine warfare (ASW). MMs are interchangeable with either SF variant as dictated by fleet requirements. When an MM is married with its corresponding crew and aviation detachment, it is known as an MP. The Logistics/Sustainment program is a dedicated LCS organization to provide worldwide maintenance, repair parts, technical specialists, storage facilities, transportation, configuration management, contracting, software and hardware changes, and LCS-specific support for both hull variants (14 each (planned)) and 64 MMs (planned). The fielding plan described in Revision A to the LCS TEMP indicated that the MCM and SUW MMs would be fielded in four increments and the ASW MP would be fielded in a single increment. However, recent programmatic changes to the LCS MM program have resulted in a TEMP strategy that tests and fields the full MP capability in a final increment for each MP. Lead DT&E Organization: NSWC PHD Summary of FY 2016 DT&E Activities Navy policy reassigned the first four LCS SFs (LCS 1 4) to serve as test and training ships; the remainder of the SFs will support fleet operations and deployments. Lack of ship availability historically has been a delaying factor in timely completion of at-sea testing for emerging components and training of crews. The Navy s policy change should facilitate DT, training, and earlier fielding of new capabilities. USS FREEDOM (LCS-1) completed Combat System Ship Qualification Trials in April 2016 following a shipyard technical refreshment, including an update of its combat system; USS FREEDOM subsequently participated in the 2016 Rim of the Pacific (RIMPAC 16) multinational fleet exercises. USS JACKSON (LCS-6) completed Full-Ship Shock Trials (FSST) in July FSST is designed to characterize a ship s ability to survive the broad degrading effect of underwater shock and to identify significant shock deficiencies in order to develop design changes to improve ship survivability. LCS-6 FSST data analysis is ongoing, with the final report due in April USS MILWAUKEE (LCS-5) completed two of three planned FSST shots in September Several significant shock-related deficiencies were revealed that must be investigated for potential redesign to improve overall ship survivability, and the Navy believes that these two 144 DASD(DT&E) FY 2016 Annual Report

151 Navy LCS and MPs shots provided sufficient data for accurate modeling to support design-level assessments and that the third shot is not required. DASD(DT&E) believes that the Navy should provide its analysis showing the adequacy of the two-shot data set. DOT&E does not concur in the Navy s conclusions and believes that the third shot will yield significant additional information about ship and system survivability that was not uncovered during the first two shots. LCS-5 FSST data analysis is ongoing, with the final report due in mid USS CORONADO (LCS-4), embarked with the SUW Increment 2 MP, conducted tracking and live-fire DT from January through June 2016 to verify system performance after updates to its integrated combat management system (ICMS). These activities included a SeaRAM upgrade completed in April 2016, plus a SeaRAM live-fire test in early June These events also marked completion of IOT&E phase II. The IOT&E report is expected to be released by the Commander, Operational Test and Evaluation Force in early USS CORONADO then participated in RIMPAC 16 during which the ship test-fired a Harpoon missile as an engineering demonstration. USS CORONADO subsequently deployed to Singapore in August USS MONTGOMERY (LCS-8) was commissioned on September 13, 2016, and transited to the ship s home port of San Diego, California. LCS-8 is currently conducting Post-Delivery Tests and Trials (PDT&T). USS DETROIT (LCS-7) was delivered in August 2016 and is undergoing PDT&T. LCS-7 is scheduled to support Longbow Hellfire testing in early 2017 in support of SUW MP testing. Summary of FY 2016 DT&E Engagement and Assessments Engineering Plant: During 2016, six SFs (three of each variant) experienced major propulsion system failures and required shipyard availabilities to effect repair. The emergence of these failures caused the Navy to review its training programs and to examine the possible need for design changes. These changes will be implemented in the future and backfitted to LCS SFs and in the new frigate design. Cybersecurity: Since the beginning of LCS deployments in 2013, three SFs (LCS-1, LCS-3, and LCS-4) have undergone major cybersecurity testing with results showing progressive improvements, but the SFs are still not performing at an acceptable level. The most recent cyber testing was in June July 2016 aboard USS CORONADO (LCS-4); because of an approaching deployment deadline, testing was necessary before all information assurance fixes could be installed; unofficial initial results found that performance was as expected, but a later check of the final fixes found them to be effective on the focused areas targeted, but overall cyber risk remains. DASD(DT&E) has increased cyber technical assistance with cybersecurity tabletop training and lessons learned from prior cybersecurity testing to assist PEO LCS. LCS TEMP: PEO LCS is updating the LCS/LCS MP TEMP, which DASD(DT&E) last approved in 2013, to address programmatic changes due to schedule and budget challenges. Revision B is expected to be submitted for OSD approval in the 3rd quarter FY DASD(DT&E) is working closely with PEO LCS and various Navy and OSD stakeholders to ensure that updated DT requirements are adequate and balanced with schedule and resource constraints and that the TEMP incorporates a DEF based on statistical test design and analysis methodologies. DASD(DT&E) focus areas include system performance and reliability, system interoperability, and cybersecurity testing. MP Readiness: DASD(DT&E) continues to engage with PEO LCS offices to ensure that MPs are adequately tested and exhibit necessary performance and reliability to support introduction of LCS SFs and MPs. o MCM MP: The Navy elected not to proceed to IOT&E because of reliability issues with the remote multi-mission vehicle (RMMV) within the Remote Minehunting System (RMS). DASD(DT&E) FY 2016 Annual Report 145

152 Navy LCS and MPs This decision led to cancellation of the RMS program in February 2016 and reformulation of the MCM MP development and test schedule. Two alternatives to RMMV are currently under consideration: the common unmanned surface vehicle and the unmanned surface vehicle/minehunting unit. An unmanned undersea vehicle, the Knifefish, is undergoing DT for deep and buried threats as a new capability. Two improved sonar sensors (AQS-20C and AQS-24C) are upgrades being developed to detect, identify, and locate threat mines in the majority of the water column. The airborne laser mine detection system, which detects mines on the near-surface, also continues to demonstrate improved performance, as evidenced by changing out obsolete components, new baseline, and improving tactics and techniques; however, there has been no formal testing to characterize the improved effects. A final MCM MP configuration decision is expected in 2018 to support MP DT and IOT&E in the 2020 time frame (a 5-year delay from the previous plan). The MCM MP has yet to be integrated and tested on a USS FREEDOM variant to ensure space and weight compatibility, operability, and effectiveness from this platform. o SUW MP: The LCS ICMS continues to be refined to improve SUW accuracy and eliminate latency issues. Follow-on testing with the full SUW MP (including surface-to-surface missiles) is planned for 2017 and 2018 for both SFs. In addition, the role of the armed helicopter needs to be integrated and tested to be vetted as part of the LCS SUW kill chain. o ASW MP: ASW MP test planning and execution are in development; DASD(DT&E) is providing guidance and technical assistance on statistical methodologies to meet testing requirements. ASW MP DT is expected to commence in late The LCS program requested and was granted approval to conduct additional DT events in early 2017 that are not in the current TEMP, in advance of TEMP (Revision B) approval. DASD(DT&E) reviewed these tests and determined that they were necessary and low risk; the USD(AT&L) approved this testing to enable the LCS and LCS MM programs to maintain schedule. DASD(DT&E) Program Assessment DASD(DT&E) provided a formal assessment of the LCS family of programs in support of the USD(AT&L) annual in-progress review in April DASD(DT&E) assessed test results and overall system performance at that time for each SF and MP. DASD(DT&E) noted several areas of concern, including compressed testing timelines, SF availability for testing, schedule-driven MP test schedules, cybersecurity testing, and the need for an updated TEMP. Conclusion LCS SFs continue to make improvements to the ICMS and MP integration and are working to make reliability improvements, but the most critical problem encountered in 2016 related to operation of the complex propulsion systems in both variants. It is too early to determine whether the findings of an LCS-wide stand-down for training has resolved the problems or simply mitigated one aspect. Design changes are being considered for the areas in which training will not suffice. The SUW Increment 2 MP, embarked on USS CORONADO, completed DT&E in accordance with the approved TEMP in FY 2016; the SUW MP now must incorporate and test the surfaceto-surface missile capabilities as well as integrate the armed helicopter into its kill chain in order to complete SUW MP testing. 146 DASD(DT&E) FY 2016 Annual Report

153 Navy LCS and MPs System reliability issues in the MCM MP resulted in postponement of the MCM MP DT until FY 2020; the Navy must fund and support testing requirements for an RMS alternative to validate MCM system capabilities. ASW DT has not yet begun; testing may hold discovery items not yet considered, especially for tactics, techniques, and procedures development; the hardware and software resident in the MP make it unique and potentially a game changer. Cybersecurity is still an expanding opportunity for test and training to improve earlier ship preparations; DASD(DT&E) is providing technical assistance with implementation of cyber tabletop exercises and lessons learned elsewhere and will provide support for interpretation of results from virtual ship testing. Both LCS SF variants completed FSST in 2016; final FSST reports are needed to assess SF shock effects and identify potential modifications, if necessary, that would be especially applicable to frigate design. LCS test assets and compressed test schedules remain challenges in crafting a robust DT program for all MP elements, but new availability of ship test assets may permit more timely and thorough testing and earlier implementation of MP capabilities. DASD(DT&E) FY 2016 Annual Report 147

154 MQ-4C Triton Unmanned Aircraft System (UAS) Executive Summary: The MQ-4C Triton UAS provides persistent maritime intelligence, surveillance, and reconnaissance (ISR) within a large volume of the maritime battlespace as part of the Navy maritime patrol and reconnaissance family of systems. The Triton UAS consists of the high-altitude, long-endurance MQ-4C Triton aircraft; sensor payloads; line-of-sight and beyond line-of-sight communications; a mission control station (MCS); and support elements. The MQ-4C aircraft design is based on the Air Force RQ-4B Global Hawk with modifications that strengthen the structure and provide a capability for limited flight in icing conditions. The MQ-4C is equipped with the multifunction active sensor maritime surveillance radar to detect, identify, and track surface targets and produce high-resolution imagery. Electro-optical and infrared (EO/IR) sensors provide fullmotion video and still imagery of surface targets. Other sensors provide a capability to detect, identify, and locate threats and cooperative ships and provide signals intelligence. The MQ-4C continued DT&E in FY 2016 with a focus on sensor integration. Lead DT&E Organization: NAWCAD VX-20 Summary of FY 2016 DT&E Activities Sensor performance and heavy weight envelope expansion flight testing continued, with 54 flights conducted for a total of 362 flight hours. The third DT aircraft has become a ground test asset to reduce risk for development and integration of the full baseline Integrated Functional Configuration (IFC) 3.1. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) assessed DT&E and program progress as adequate to support the baseline production decision. DASD(DT&E) assisted in updating the T&E strategy for development and integration of improved multiple intelligence (multi-int) capabilities. Initial sensor integration testing has been completed, and the results indicate a nominal level of system maturity at this mid stage of DT&E. Three of the five DT aircraft are now at the primary flight test location at Naval Air Station Patuxent River, Maryland. System performance characterization is progressing slower than initially planned, and schedule risk to completion remains high because of delays in delivering system capabilities for T&E. Full baseline capability was originally scheduled to begin flight testing in FY 2014, and the latest schedule indicates the start will be in FY Flight test progress to date is slower than planned in the TEMP because of routine issues discovered in flight test and sensor performance and integration issues. The MQ-4C Triton program did not request a waiver or deviation from requirements in the TEMP. 148 DASD(DT&E) FY 2016 Annual Report

155 Navy MQ-4C Triton UAS DASD(DT&E) Program Assessment DASD(DT&E) conducted a formal assessment of the MQ-4C Triton UAS program in FY 2016 to support the LRIP decision for the baseline ISR capability. The DASD(DT&E) concluded that overall, the Triton UAS met, with the exception of software, the MS C entrance criteria directed in the MS B Acquisition Decision Memorandum dated April 18, DASD(DT&E) assesses post-ms C DT&E planning as adequate for IFC3.1, but the planning needs to be updated to support the MS C multi-int acquisition strategy. There is a significant concurrency risk driven by the new multi-int acquisition strategy in that the knowledge available from T&E is currently limited to IFC2.2 ISR integration, while the new acquisition strategy is for the production, retrofit, and fielding of the multi-int configuration. The DASD(DT&E) assessment highlighted the program concurrency risk and system maturity: o Concurrency Risk. The program acquisition strategy has changed since MS B and now has production and fielding of the multi-int configuration delivered in IFC4. However, the first test aircraft will not be modified to the multi-int hardware configuration until the 3rd quarter FY 2018 and will not begin flight testing until the 4th quarter FY 2019, representing a commitment to procure 12 unmanned aircraft before flight testing. o Performance. The Triton UAS has demonstrated limited performance in flight test. Endurance is limited by takeoff gross weight restrictions to approximately 16 hours against a planned 24-hour capability; however, the Triton system has demonstrated a capability to achieve the minimum mission radius KPP. Additional flight testing and software updates are required to attain full endurance. The radar sensor has demonstrated better than threshold detection performance, with several issues noted limiting classification performance. The EO/IR sensor has limited performance data available, with auto-focus and tracking issues limiting the ability to provide target classification. Significant issues have been discovered related to the operator interface with the Triton system at the MCS, with multiple critical deficiencies currently open and fixes scheduled for IFC3.1. The multi-int upgrade is forecast to exceed the AC power reserve available to simultaneously operate all sensors. o Interoperability. The system has demonstrated narrowband and wideband links for C2 and transmission of ISR sensor data to a simulated CVN ship and P-8 aircraft. Ultrahigh frequency satellite communications have not been used in flight because of the potential for electromagnetic interference. The program has demonstrated progress in developing the capabilities to integrate with the Navy wide-area network; however, IFC3.1 is required to achieve this capability. o RAM. The system has demonstrated a mean flight hours between abort of approximately 47 hours over 421 hours with sensors installed. Operational availability is calculated at 0.24, driven by low overall reliability and high corrective maintenance times. o Cybersecurity. The program completed its assessment of cybersecurity-related requirements and identified potential vulnerabilities. Additional cybersecurity T&E is awaiting implementation of full cybersecurity controls and capabilities in IFC3.1. Triton cybersecurity T&E risk for the remaining effort is low to moderate, given the proactive approach to cybersecurity that the program has taken. Conclusion: The MQ-4C Triton is in the mid stage of development, with DT&E results indicating a moderate risk for baseline ISR technical performance at this time. The modified Triton UAS acquisition strategy to accelerate multi-int capability has increased the concurrency risk between development and production. DASD(DT&E) FY 2016 Annual Report 149

156 Multi-Mission Maritime Aircraft (P-8A Poseidon) Executive Summary: The P-8A is a derivative of an existing Boeing aircraft with design changes to support the Navy s maritime patrol mission. The baseline P-8A is replacing the aging P-3C, while planned increments address expanding its role to broader-area antisubmarine warfare (ASW) and highaltitude ASW weapon capability (HAAWC). The P-8A was approved for FRP in early 2014 and redesignated as an ACAT 1C program in FY The USD(AT&L) approved a multi-increment upgrade program with Increment 2 integrating the multistatic active coherent (MAC) ASW capability, the automatic identification system, acoustic system updates, and high-altitude ASW capability. Increment 3 adds specific additional capabilities as available and evolves the mission system to an open architecture to enable more rapid future capability insertion. Increment 2 has successfully commenced flight test, but completion of the second of three ECPs is delayed. Lead DT&E Organization: NAWCAD VX-20 Summary of FY 2016 DT&E Activities The program completed lab and ground DT&E and commenced flight testing of Increment 2, ECP 2 for the MAC/Rapid Capability Insertion algorithms ASW capability, the automatic identification system, acoustic system updates, and high-altitude ASW sensors. The program conducted DT&E on software upgrades for the AN/APY-10 radar, radar radio frequency safety interlock ground test, common data link ground and flight testing, Joint Range Extension Applications Protocol C lab testing, SIPRNET token lab test, flight testing of software upgrades to an international maritime satellite, dry-bay fire protection system new functional procedure checkout, joint service aircrew mask testing, quick reaction capability, and flight testing of the low-cost acoustic processor system. The program completed HAAWC safe separation testing and all captive-carry capability testing in preparation for simultaneous launch of multiple Harpoon antiship missiles. Completion of Increment 2, ECP 2 DT&E, including simultaneous launch of multiple Harpoon missiles, is delayed until the 1st quarter FY 2017 because of noncooperation of a foreign submarine target and test range maintenance issues. The program office restructured Increment 3 requirements and test planning based on the projected availability of capabilities and reductions to the overall budget. The program office revised the baseline program TEMP to include the MAC program and Increment DASD(DT&E) FY 2016 Annual Report

157 Navy P-8A Poseidon Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) engaged in numerous Increment 2 data assessments to support testing with no restrictions to data access. DASD(DT&E) supported Navy efforts to revise the P-8A master TEMP to include Increment 3 and MAC testing. DASD(DT&E) actively participated in Increment 3 test planning sessions to include cybersecurity and updates to the DEF. DASD(DT&E) supported the program s delay of OT on Increment 2, ECP 2 pending the completion of DT&E live MAC and Harpoon testing. DASD(DT&E) assesses the program s technology and test plans for Increment 3 as adequate. The technology and test plans include two blocks of testing; the first adds specific additional capabilities as available, and the second installs an open-architecture, service-oriented extension to the baseline tactical open mission software architecture. This new architecture will enable netready processing and more rapid future capability insertion, as well as improve sustainment. The P-8A program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The P-8A program continues to build, test, and field incremental capabilities to the fleet. The Navy closes prioritized open discrepancies discovered in prior testing in conjunction with ongoing testing. The program is assessed as low risk as it executes Increments 2 and 3 DT&E. DASD(DT&E) FY 2016 Annual Report 151

158 Trident II Life Extension (D5LE) Executive Summary: The Trident II missile is a D5 ballistic missile launched from OHIO-class strategic submarines (SSBNs). The D5 missile is capable of delivering nuclear warheads in the event the Nation fails to deter nuclear war. Because D5 is a highly reliable and accurate missile, the U.S. Navy decided to use it on the OHIO Replacement submarines rather than to design a new missile. To accomplish this, the guidance systems, command sequencer, and electronics of the missile inventory must be modified to extend their life. The SSP Office procured 108 new U.S. D5 missiles and D5LE Strategic Programs Alteration (SPALT) kits in FY 2011 FY 2015, and deliveries will be completed by FY In addition, SPALT kits for converting the existing D5 inventory to D5LE (269 U.S. missiles and 47 United Kingdom missiles), as well as spare SPALT kits, are being procured in FY 2016 FY Pending successful completion of the D5LE SPALT Development Program, the first D5LE missiles will be deployed in FY Conversion of the entire U.S. fleet to D5LE missiles will complete in FY Lead DT&E Organization: SSP Summary of FY 2016 DT&E Activities SSP uses a comprehensive, layered DT approach. Piece parts and components undergo performance and environmental testing, and manufacturing processes are validated. Subsystems undergo design and performance verification testing. Subsystems are assembled into packages and go through package qualification and package acceptance testing, while assembly processes are validated and margins and limits are measured and verified. System verification testing and integration testing are conducted with HWIL simulation. Lastly, operationally realistic flight tests are conducted to validate that the upgraded missiles perform with the same level of reliability and accuracy as the D5 missiles. In 2016, the fifth and sixth D5LE test missiles were successfully launched to verify that the alterations function as designed. Summary of FY 2016 DT&E Engagement and Assessments Reliability testing is an integral part of piece part and package acceptance testing, lab testing, and flight testing. DASD(DT&E) reviewed FY 2016 test results and determined that reliability and performance met requirements. DASD(DT&E) will continue to work with SSP to review future test results, as they become available, to verify that missile performance and reliability are being maintained. The Trident II D5LE program did not request a waiver or deviation from requirements in the TEMP. Conclusion: To date, test results of D5LE reliability indicate that the D5LE missiles will meet the reliability requirement. 152 DASD(DT&E) FY 2016 Annual Report

159 U.S. Navy Integrated Air and Missile Defense (IAMD) An assessment of several Navy programs that, once integrated, provide Navy IAMD. These systemof-systems (SoS) programs include Air and Missile Defense Radar (AMDR), DDG-51 Flight III Destroyer, Aegis Modernization, and Standard Missile (SM)-6. Executive Summary: The Navy s primary mission of delivering credible capability for deterrence, sea control, and power projection is enabled by the surface Navy s IAMD capability. This capability, made possible by a group of systems that are being developed or modernized, will be addressed as individual programs. IAMD is the centerpiece of the current Aegis Modernization combat system (CS) program upgrade, which is a significant improvement to Aegis. The Navy is also upgrading the DDG-51 to the Flight III configuration with delivery in 2022 to utilize the new AMDR, which the Navy is developing specifically to further enhance ballistic missile defense (BMD) and advanced threats. When integrated with other new and modernized systems, AMDR will provide the Flight III with an increased IAMD capability. With this capability, Navy DDGs that are upgraded to the new Aegis Baseline (BL) 10 will be capable of conducting BMD and antiair warfare (AAW) engagements simultaneously using four surface Navy programs (DDG-51 with its Aegis Advanced Capability Build (ACB), AMDR, and SM-6). Each of these individual programs and their test plans/results that are integrated into the IAMD capability are discussed below. DASD(DT&E) FY 2016 Annual Report 153

160 Navy U. S. Navy IAMD DDG-51 Flight III Destroyer The ARLEIGH BURKE (DDG-51)-class ship is a multi-mission surface combatant capable of simultaneously engaging antiair, antisurface, and antisubmarine warfare threats while performing strike operations. DDG-51 class ships operate offensively and defensively as part of a carrier strike group, surface action group, amphibious task force, and underway replenishment group. The Navy is currently building the Flight IIA configuration (DDG-79 through DDG-123) with the SPY-1D(V) radar, and starting with the congressionally added third ship of the FY 2016 procurement, a new configuration, Flight III, will include cooling and power upgrades to support the new AMDR. DASD(DT&E) is primarily focused on development and testing of the new Flight III variant. Lead DT&E Organization: NSWC PHD DDG-51 Flight III AMDR-S Replaces SPY-1D(V) Air and Missile Defense Radar AMDR is the Navy s next-generation radar system that will address current and future BMD and air defense challenges. The AMDR suite consists of an S-band radar (AMDR-S), an X-band radar, and a radar suite controller. AMDR-S is a new development IAMD radar providing added sensitivity for long-range detection and engagement of advanced threats. The initial X-band radar for the AMDR suite for ship sets 1 12 is a horizon-search radar based on existing technology, the AN/SPQ-9B X-band radar. AMDR will require no new development efforts for the AN/SPQ-9B and will accept the AN/SPQ-9B existing performance and logistics infrastructure. Starting with the 13th ship set, the Navy intends to develop and integrate a future X-band sensor into the AMDR suite. Lead DT&E Organization: AMDR Cross-Product Team Aegis Modernization The Aegis Modernization program consists of successive ACB upgrades to the Aegis Weapon System Mk 7, which is the automated segment of the Aegis CS. These upgrades are developed on an approximate 4-year cycle with ACB 12 conducting testing in 2012, ACB 16 in 2016, and ACB 20 in The ACB 12 upgrade, called Aegis BL 9 when integrated aboard an Aegis ship, provides Aegis DDGs with a comprehensive AAW and BMD mission modernization of their CS between 2013 and The Navy is also installing BL 9 on some USS TICONDEROGA (CG-47)-class cruisers and Flight I USS ARLEIGH BURKE 154 DASD(DT&E) FY 2016 Annual Report

161 Navy U. S. Navy IAMD (DDG-51) destroyers. New-construction DDGs, beginning with USS JOHN FINN (DDG-113), will be delivered with BL 9. Lead DT&E Organization: NSWC PHD Standard Missile-6 SM-6 combines the tested legacy of the SM-2 propulsion and ordnance with a repackaged advanced medium-range air-to-air missile active seeker, allowing for enhanced performance at extended ranges. The SM-6 Block I missile increases the battlespace using its autonomous active seeker mode either with Aegis in a stand-alone configuration or beyond the horizon with a Cooperative Engagement Capability (CEC) configuration. When the firing ship is employed with Naval Integrated Fire Control Counter Air, SM-6 Block I will provide extended-range AAW defense to the full extent of the missile s kinematic limit both above and beyond the radar horizon. The program conducted a series of Block 1 follow-on test and evaluation (FOT&E) tests into 2016 and testing of the SM-6 Block 1A configuration. Lead DT&E Organization: NSWC PHD Summary of FY 2016 DT&E Activities DDG-51 Flight III Destroyer DDG-51 Flight III first ship at-sea DT is scheduled to begin in Planning for DDG-51 integrated system testing using Aegis ACB 20 and AMDR has begun. The program is conducting early CS integration and testing (I&T) as part of the AMDR DT at the Pacific Missile Range Facility (PMRF), Hawaii, throughout FY A DAB In-Process Review was scheduled for FY 2016 to address ECPs for the Flight III configuration but has been delayed into FY The program conducted a DEF Core Team to develop the program s DEF for inclusion in the Flight III TEMP. This DEF is expected to be integrated with the AMDR and Aegis Modernization DEFs for the same testing time frames. Air and Missile Defense Radar AMDR entered into the EMD phase in early 2014 and has been conducting DT since FY Testing of AMDR began in late FY 2015 with string testing of components at the developer s site (DT-2) and continued into FY Concurrent with the DT-2 radar testing is a CS integration test (CIT) planned at the Combat System Engineering Development Site. CIT-1 will use a radar back-end and antenna emulator to replicate the radar. DT-3 testing will consist of land-based testing with the engineering development model (EDM) at the PMRF into late FY The AMDR EDM was delivered and installed at PMRF in mid- FY 2016, and installation and checkout procedures closed out FY Following DT-3 at PMRF, CIT-2 will use the CS interface support equipment (ISE) to replicate the CS integrated with the AMDR EDM. As currently scheduled, limited time exists at the test facility at PMRF to fully conduct the AMDR DT-3 and the CIT-2 integrated testing with the AMDR and the CS ISE before the scheduled MS C. The Navy originally planned to conduct an operational assessment (OA) in FY 2017 during the planned DT at the PMRF. However, because the AMDR is not connected to the Aegis CS and cannot be operated by Sailors and the threat presentations are not operationally realistic, little was DASD(DT&E) FY 2016 Annual Report 155

162 Navy U. S. Navy IAMD expected to be learned from the OA, so DOT&E recommended that it be canceled to reduce the program s cost. The USD(AT&L) agreed and canceled the requirement for the OA. Aegis Modernization Aegis DT in FY 2016 was limited to surface warfare (SUW) testing on DDG-65 and DDG-51. During the DDG-53 dedicated OT in January 2016, the Navy conducted DT live-fire (LF) event 04 in which an SM-2 was successfully used to defeat a subsonic, low-altitude threat. The Aegis program was an early implementer of a DEF and design of experiments, which the ACB 16 TEMP has incorporated. The ACB 20 TEMP will ultimately integrate the entire SoS testing that includes the CS (ACB 20), AMDR, and Evolved Sea Sparrow Missile (ESSM) Block 2. Although AMDR and ESSM will continue to maintain separate TEMPs, AMDR and ESSM testing will be integrated into the Flight III TEMP for efficiencies of testing and use of resources. Standard Missile-6 During DDG-53 OT in January 2016, the Navy conducted DT LF events D-1A, D-1B, D-1D, and D-1Ga as part of the SM-6 FOT&E program. Additionally during this phase, the Navy conducted an Agile Prism test that was unsuccessful. SM-6 Block IA continued testing with one guided test vehicle (GTV) (GTV-3A) flight test at White Sands Missile Range (WSMR), New Mexico, in August Summary of FY 2016 DT&E Engagement and Assessments DDG-51 Flight III Destroyer The Navy was required to provide an addendum to the TEMP with test planning and updated resource requirements before its DAB review. The Navy intends to update the Aegis Modernization TEMP as a starting point for the Flight III TEMP. This approach will require the DDG-51 Flight III TEMP to include hull, mechanical, and electrical systems in addition to the ACB 20 test plan. Although Flight III (ACB 20) testing builds off of ACB 16 testing, Flight III testing will require the first full use of the Aegis M&S test bed, which is still in development. Critical to verification, validation, and accreditation of the M&S is the upgrade to the Self-Defense Test Ship (SDTS) with the AMDR, which will allow verification and validation of the Aegis probability of raid annihilation (PRA) self-defense scenarios. The Navy was directed by Resource Management Decision 700A2 to budget $175 million for long-lead radar and Aegis equipment for the SDTS in FY 2019 FY An important aspect of this effort is the development of the Aegis Test Bed (ATB) M&S, which the SDTS test data are intended to anchor. However, the SM-2, SM-6, AMDR, and ESSM models for the ATB are not fully funded. These models need to complete development to support Flight III and AMDR testing. The lack of funding creates a high risk to the planned conduct of cost-effective M&S testing of the various PRA scenarios, which is intended to reduce the need for complicated and expensive live self-defense testing. The overall impact of the design compromises made to fit the AMDR into the DDG-51 seaframe cannot be fully assessed until the full SoS is tested in an end-to-end integrated test at sea on the first ship of the class. That testing, by the nature of the development timelines, is serial testing and will not take place until 41 percent of the AMDRs are already purchased, the ACB 20 CS development is complete, and at least 10 Flight III ships will be on contract. This serial development will be mitigated somewhat by the integrated testing of ACB 20 and the radar at Wallops Island, Virginia, before at-sea testing, but full end-to-end testing of the radar powered 156 DASD(DT&E) FY 2016 Annual Report

163 Navy U. S. Navy IAMD and cooled by ship systems integrated with the final CS will not take place until the first Flight III ship is delivered in Early CS I&T scheduled to be conducted during AMDR DT in FY 2017 will not include direct ACB 20 functionality (because it has not been designed yet) and will be primarily based on the current Aegis BL 9 CS. This plan induces risk because the final CS build will not be available until the first platform is delivered in Additionally, the first three ships will be configured with ACB 20 Phase 0 incorporating Technical Insertion (TI) 16 equipment. The follow-on ships will be part of the new multiyear procurement and will have ACB 20 Phase 1 TI 20 equipment. Air and Missile Defense Radar (AN/SPY-6) An updated AMDR TEMP is expected in The AMDR testing, including scenarios using the SDTS for PRA validation, will also be integrated into the draft Flight III TEMP for efficiencies of testing and use of resources. When the Navy developed the pre-ms B AMDR TEMP in 2012, DASD(DT&E) staff engaged with the program office to focus attention on the need for earlier integration testing, as full integrated testing of the AMDR with the Aegis ACB 20 (CDR is in FY 2019) would not be possible until post-ms C. The early CIT that evolved was an attempt to integrate the AMDR EDM with a substitute CS that would be based on the current ACB 12 and would replicate the ACB 20 system as much as possible. The Aegis program office developed the AMDR-Aegis CS ISE, which incorporates a sensor management architecture that mimics the expected data exchange patterns consistent with the anticipated Aegis baseline. The ISE will support early integration events at the Aegis integration facility and the advanced radar detection laboratory facility before MS C. This testing is critical to highlight radar/cs integration risk as early as possible. However, the AMDR program s focus on radar-only testing in DT-3 at the expense of CS integration testing contradicts the inherent risk observed on previous programs, such as the CEC/Aegis integration issues that laid up two Aegis cruisers in 1998, the SPY-1D(V)/Aegis integration issues observed at sea after initial DT/OT at the land-based test site (LBTS), and the recent CVN 78 Dual-Band Radar issues. The plan for BMD testing in the approved AMDR MS B TEMP has changed because of BMD target safety issues and the limited time for required launcher modifications before the last AMDR DT event before MS C. The T&E stakeholder-approved alternative is to launch outbound BMD targets from the PMRF with minor target trajectory and deployment modifications to optimize AMDR testing. The primary purpose of BMD testing at this phase is to assess AMDR target discrimination algorithms, and the planned outbound target presentations appear adequate to support software and M&S development. The test plan is schedule driven, based on range availability, with associated inherent risks of using new developmental BMD threat targets for the first time with these tests. Some of the BMD targets are experiencing development issues and are not expected to be available before the currently scheduled MS C. The Navy is planning to move the single EDM array from PMRF in 2018 to support CS integration testing post-ms C at the Aegis LBTS at Wallops Island, Virginia (the Navy is now considering initially moving this array to the Combat System Engineering Development Site at Moorestown, New Jersey, for integration testing). Although removal of the array from PMRF is an important part of the CS integration effort, the DASD(DT&E) has previously recommended that this test array or a production array be kept at PMRF for as long as possible because the PMRF site provides valuable BMD tracking opportunities that are not available at Wallops Island and are critical for early M&S development. The opportunity for additional presentations of BMD target complexes is underscored by the risk of the planned BMD testing discussed above. DASD(DT&E) FY 2016 Annual Report 157

164 Navy U. S. Navy IAMD The Navy has previously used production arrays for land-based testing and then returned them for ship installation at the completion of testing. Testing of the AMDR s IAMD KPP and the DDG-51 Flight III PRA KPP was cause for the development of an Aegis SDTS program to collect data in the self-defense region. Critical to this effort is the development of the ATB M&S, which the SDTS test data are intended to anchor. However, the AMDR model for the ATB is not fully funded. This model needs to complete development to support Flight III and AMDR testing. The lack of funding creates a high risk to the planned conduct of cost-effective M&S testing of the various PRA scenarios, which is intended to reduce the need for complicated and expensive live self-defense testing. Aegis Modernization The majority of Aegis Modernization LF testing in 2016 consisted of OT. During the DDG-53 combat systems ship qualification testing (CSSQT) in January 2016, the Navy conducted LF event 04 in which an SM-2 was successfully used to defeat a subsonic, lowaltitude threat. A previous attempt during this same test period resulted in a reliability failure of the SM-2. Other Aegis testing was accomplished as part of the SM-6 testing described below. During DDG-51 IAMD CSSQT in June 2016, the Navy conducted SUW tracking exercises, which were mainly successful, but the same issues that were seen in previous SUW testing continue to be present. Also during the DDG-51 IAMD CSSQT, the Navy conducted LF events for LF-01, LF-02, and LF-04. In LF-01, two SM-2s were fired to defeat two subsonic, low-altitude threats; one was successful and one was a failure because of unexpected issues with a new-capability tracking algorithm. In LF-02, an SM-2 was successfully used to defeat a crossing subsonic, low-altitude threat. In LF-04, an SM-2 successfully defeated a low-altitude threat while simultaneously conducting a long-range surveillance and track of two simulated ballistic missile targets. Standard Missile-6 During DDG-53 CSSQT in January 2016, the Navy conducted LF events D-1A, D-1B, D-1D, and D-1Ga as part of the SM-6 FOT&E program. During these tests, the SM-6 engaged threats and scenarios that resolved the remaining performance KPPs including longest flight and intercept to date, longest cross-range intercept to date, and the first multiple simultaneous intercepts using SM-6 with Aegis BL 9. During D-1Ga, the SM-6 successfully completed an attempted scenario from 2015 against a low-altitude target with electronic attack capability. The SM-6 Block IA program conducted a GTV-3 test at WSMR in August 2016 that resulted in a misfire/dud. A repeat of this scenario was attempted in November 2016 but was also unsuccessful, and a failure review board is currently in progress. The launch availability KPP requires the carrying of SM-6 missiles onboard for 8 months before firing. The 8-month storage onboard has been completed, and all missiles except one have been fired successfully. The remaining missile is being planned for a future firing. Conclusion: The AMDR, DDG-51 Flight III Destroyer, Aegis Modernization, and SM-6 programs did not request a waiver or deviation from requirements in the TEMP. 158 DASD(DT&E) FY 2016 Annual Report

165 VIRGINIA-Class Submarine Executive Summary: The VIRGINIA-class fast attack submarine is an ACAT IC program that previously delivered 12 submarines. In 2016, the program delivered USS ILLINOIS (SSN 786), the third of eight Block III submarines, on schedule and within budget. Block III submarines have the same capability as previously delivered submarines but at reduced cost and improved reliability. The major changes include replacing the large spherical-array sonar with a smaller, large-aperture bow (LAB) array sonar that uses a water-backed array of passive hydrophones and active transmitters; replacing the 12 vertical launch system tubes with two VIRGINIA payload tubes (VPTs) (six Tomahawk missiles per tube) and providing a new payload support electronics system and common weapon launcher; and incorporating many other design changes to reduce the cost per unit and improve reliability. In May 2014, NAVSEA awarded the contract to build 10 Block IV submarines (two per year). Design changes are intended to further reduce the total ownership cost of VIRGINIA platforms with the same capability. The Block V procurement is in the planning stages and is scheduled for award in FY The VIRGINIA payload module is targeted for insertion in Block V hulls and will leverage the success of the VPT as it utilizes similar tubes developed for Block III. The CDD adds KPPs for strike capacity, cost, and schedule, while increasing Tomahawk land-attack missile strike capacity from 12 to 40. To provide the additional 28 missiles, a new 80-foot compartment is added behind the sail with four SSGN-like missile tubes (each carrying seven missiles). This increased strike capacity is needed to replace the strike capacity that will be lost when the four SSGNs begin decommissioning in the mid-2020s. The flexible design allows for future missile systems. Lead DT&E Organization: NUWC NPT Summary of FY 2016 DT&E Activities USS MINNESOTA (SSN 783) o In July 2016, the program conducted an acoustic trial (ACTRL) with 198 test events and 200 stern assessment events in support of the OHIO Replacement program. The ACTRL identified two major issues that are being addressed. o In August 2016, the program conducted a VIRGINIA-Class Warfare Material Certification Program test. The test accomplished all Technology Insertion (TI) 12/Advanced Processor Build (APB) 11 Integrated Combat System (ICS) test objectives. Six exercise torpedoes (EXWEPs) were successfully launched from torpedo tubes 1 through 4, and all six were evaluated in situ by the Atlantic Undersea Test and Evaluation Center (AUTEC) range as bullseyes. The test also successfully simulated an at-sea strike mission salvo launch. o In September 2016, the program conducted an underwater electromagnetic (UEM) measurement trial. DASD(DT&E) FY 2016 Annual Report 159

166 Navy VIRGINIA-Class Submarine USS JOHN WARNER (SSN 785) o In January 2016, the program conducted a UEM measurement trial. Near-field electric signature was not calibrated because of a failed range sensor and is planned to be calibrated before deployment. o In January 2016, the program conducted an ACTRL with 240 test events. The trial identified two major issues that are being addressed and 20 other issues. The program also successfully completed sonar calibration for Block III submarines. o In February 2016, the program conducted a weapon system accuracy test (WSAT). Dockside testing identified six major issues that were all corrected before at-sea testing. o The classified WSAT Report indicates that most major deficiencies previously identified on USS NORTH DAKOTA (SSN 784) were corrected on USS JOHN WARNER (SSN 785). o In September and October 2016, the Commander, Operational Test and Evaluation Force (COTF) conducted cybersecurity testing on USS JOHN WARNER (SSN 785). USS ILLINOIS (SSN 786) o In September 2016, the program conducted a WSAT. The test indicated that all test objectives were met for TI-10/APB-09 ICS. Six EXWEPs were successfully launched from four torpedo tubes; all six were evaluated in situ by the AUTEC range as bullseyes. An atsea strike mission salvo launch was also successfully demonstrated. Summary of FY 2016 DT&E Engagement and Assessments The USS JOHN WARNER (SSN 785) WSAT Report of February 2016 and other postconstruction trials (acoustic and UEM) indicate that the developmental status of the ship is projected to meet system performance requirements. At the September 2016 VIRGINIA-Class Operational Test Readiness Review, DASD(DT&E) concurred in commencing cybersecurity follow-on operational test and evaluation. USS JOHN WARNER (SSN 785) conducted planned testing in October Not all test events were completed. Test completion is scheduled for March COTF will publish results following test completion. Based on the test results from WSAT, ACTRL, and UEM measurement tests on the VIRGINIAclass Block III ships, the VIRGINIA program is on track to support the Block III DT/OT currently scheduled from January to July However, the Navy extended its post-shipyard availability for USS NORTH DAKOTA (SSN 784) to accomplish previously unplanned repair to the LAB array in dry dock. Completion is now expected in December If completion slips into January 2017, DT and OT events may need to be rescheduled. DASD(DT&E) is working with the T&E WIPT to develop the DT portion of TEMP Revision H for the Block V test program. The VIRGINIA-Class Submarine program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The VIRGINIA-Class Submarine program is on track to continue DT of the Block III configuration in 2017 and continues to identify and mitigate cost, technical, and schedule risks that can impact upcoming tests and trials. The VIRGINIA-Class Submarine program aggressively identifies and corrects material problems to improve the quality of submarines delivered to the fleet, while minimizing the potential negative effects on availability. 160 DASD(DT&E) FY 2016 Annual Report

167 ZUMWALT-Class Destroyer (DDG 1000) Executive Summary: The ZUMWALT-class destroyer (DDG 1000) is a large multi-mission surface combatant ship with an emphasis on land attack. It incorporates several new design endeavors such as electric drive, a reverse tumblehome (inward slanting) hull design for signature reduction, the advanced gun system (AGS), and a Total Ship Computing Environment (TSCE) integrating virtually all data systems aboard the ship. TSCE is the primary enabler for a reduction in crew size to almost half the complement of the DDG 51- class destroyers. The DDG 1000 ZUMWALT-Class Destroyer program completed a Nunn-McCurdy certification in FY 2010, and a cost-saving effort in the Acquisition Decision Memorandum directed the Navy to down-scope the Dual-Band Radar by removing the Volume Search Radar hardware from the ship baseline. The PEO for Integrated Warfare Systems modified the DDG 1000 Multifunction Radar (MFR) to achieve a volume search (VS) capability. The first ship was recently delivered to the Navy by the shipbuilder, Bath Iron Works (BIW), with hull, mechanical, and electrical (HM&E) systems activated and transited to its home port in San Diego, California, where the TSCE and combat systems will be activated and tested. The Long-Range Land-Attack Projectile (LRLAP) was the guided ordnance fired by the AGS. Because of LRLAP unit cost increases, the Navy decided to seek a more affordable projectile. Several alternatives are under consideration, including negotiations with the LRLAP original equipment manufacturer to reduce unit cost. Any alternative other than LRLAP will require extensive modification to the AGS magazine, ammunition handling system, ordnance initialization interface, and TSCE gun fire control software. Lead DT&E Organization: NSWC PHD Summary of FY 2016 DT&E Activities November 24, 2015, BIW completed dockside integrated propulsion test. December 4, 2015, PMS 500 completed TSCE Release 8 mission system tests. December 14, 2015; June 16 and October 20, 2016, NSWC PHD completed live tracking tests at the Wallops Island, Virginia, land-based test site using the MFR with VS capability. April 21, 2016, BIW completed ballast/deballast/dewatering system at-sea demonstration in conjunction with Alpha trials. April 25, 2016, BIW completed at-sea propulsion plant system testing. May 4, 2016, BIW completed magazine sprinkling system control valve and operational test in the active electronic countermeasures and pyrotechnics magazine, sonobuoy magazine, and small arms magazine. May 5, 2016, BIW halted the dark ship recovery demonstration emergency diesel generator test. The test will be rescheduled at a later date. DASD(DT&E) FY 2016 Annual Report 161

168 Navy DDG 1000 May 20, 2016, the ship was accepted by the Navy. October 15, 2016, the ship was commissioned in Baltimore, Maryland. Summary of FY 2016 DT&E Engagement and Assessments PMS 500 evaluated TSCE Release 8 mission system tests, conducted in a laboratory environment, as satisfactory. TSCE 8.x releases are planned to compete design functionality and correct deficiencies discovered during activation and test phases. Future updates may be required pending results of DT&E. DASD(DT&E) assisted PMS 500 in revising and improving the cybersecurity DT&E plan in the TEMP. As planned, only HM&E systems were activated upon acceptance by the Navy, and commercial radars were temporarily installed for navigation and collision avoidance during transit to the ship s home port of San Diego, California. Dockside integrated propulsion test and at-sea propulsion plant system testing were evaluated as satisfactory. Magazine sprinkling system control valve and operational test was satisfactory in the active electronic countermeasures and pyrotechnics magazine, sonobuoy magazine, and small arms magazine. Testing in the peripheral vertical launchers, AGS magazines, sonobuoy storeroom, aviation small arms magazine, and aviation weapons assembly room will be scheduled during the combat systems activation and integration availability in FY Dark ship recovery demonstration emergency diesel generator test was halted because of deficiencies and will be repeated at a later date. Most combat systems were installed but not activated or integrated upon acceptance. The systems will be activated and integrated during a post-delivery availability in FY 2017 FY Although MFR aerial threat tracking improved with each subsequent Wallops Island-based live flight event, interactions between TSCE, Cooperative Engagement Capability, and the radar continued to exhibit instances of false and dropped tracks. Focus of MFR activity at Wallops Island is optimizing engagements for the unmanned Self- Defense Test Ship (SDTS) anti-ship cruise missile defense event in FY False and dropped tracks continue to be one goal of testing, and most tracking corrections will be in an FY 2017 software load. IWS 2 is also leveraging MFR development for CVN 78 to increase efficiency and prevent redundant testing. DASD(DT&E) recommends that test-identified shortfalls in the MFR be addressed and that solutions be verified by land-based tests of the radar to achieve acceptable levels of false and dropped tracks. Improved MFR performance will reduce risk of ordnance launch at false tracks during live missile tests aboard the unmanned SDTS. DASD(DT&E) also recommends that PEO IWS 9 and PMS 500 develop controls to maintain close oversight of contractor efforts during the combat systems activation because of the unprecedented scope and complexity of the TSCE enterprise. During FY 2016, the DDG 1000 program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The DT&E program for HM&E systems supported the ship s transit to its home port of San Diego, California. Further maturation of the MFR in land-based testing will reduce risk in live missile tests against stressing anti-ship cruise missile targets aboard the unmanned SDTS. Close management of the combat systems activation effort is required to reduce schedule, cost, and performance risk. 162 DASD(DT&E) FY 2016 Annual Report

169 6.4 Air Force Programs This section includes summaries of the following 11 programs: Air and Space Operations Center Weapon System (AOC-WS) Increment 10.2 B61-12 Life Extension Program (LEP) Tail Kit Assembly (TKA) Combat Rescue Helicopter (CRH) Defense Enterprise Accounting and Management System (DEAMS) F-22 Increment 3.2B Modernization Family of Advanced Beyond Line-of-Sight Terminals (FAB-T) Global Positioning System (GPS) Enterprise Joint Space Operations Center (JSpOC) Mission System (JMS) Increment 2 KC-46A Tanker Modernization Small Diameter Bomb Increment II (SDB II) Space-Based Infrared System (SBIRS) DASD(DT&E) FY 2016 Annual Report 163

170 Air and Space Operations Center Weapon System (AOC-WS) Increment 10.2 Executive Summary: The AOC-WS is the operations command center of the joint or combined force air component commander and provides the capability to plan, task, execute, monitor, and assess the activities of assigned or attached forces. The AOC-WS Increment 10.1 configuration established the standard AOC baseline capabilities but did not bring the AOC-WS into a net-centric environment or fully realize the intent of the Air Force C2 Enabling Concept. AOC-WS Increment 10.2 is the first increment for modernization and will lead Air Force operational C2 transition to a net-centric capability. DT1 in August 2015 demonstrated that the system would require extensive rework to close more than 400 high-priority defects before proceeding to DT2 and an operational assessment (OA) that were planned for November The delay of DT2 resulted in a schedule breach for the MS C decision, then scheduled for January In February 2016, the PEO decided to extend risk reduction testing (RRT) instead of executing DT2, based on the continued discovery of discrepancies in January and February 2016 and the contractor s inability to meet entrance criteria before the start of DT2. The Air Force Service Acquisition Executive (SAE) notified Congress of the critical change on March 30, 2016, and initiated the critical change report (CCR) process. DT2 was delayed by this declaration and is now projected to occur in March The intervening time will be used to close discrepancies and verify fixes. Lead DT&E Organization: 96th Test Wing Summary of FY 2016 DT&E Activities January 25 30, 2016, the 46th Test Squadron (46 TS) conducted a limited system and performance RRT at the Combined Air Operations Center Experimental (CAOC-X) in the Ryan Center, Joint Base Langley-Eustis, Virginia, to fill gaps from the data collected in DT1. March 21 April 8, 2016, AOC-WS Increment 10.2 conducted RRT at CAOC-X to verify fixes to discrepancies identified in DT1. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) participated in daily status updates during the RRT to monitor progress toward meeting the DT2 entrance criteria. The PMO conducted a review and clarification of the DT2 entrance and exit criteria to ensure high confidence of success going into DT2/OA. DASD(DT&E) coordinated with DOT&E, the 46 TS, and other OSD stakeholders to correct and verify the more detailed criteria. DASD(DT&E) facilitated a DEF Core Team event with the PMO test team to support the MS C TEMP update. DASD(DT&E) participated in the test readiness review for DT2, providing an assessment and a recommendation not to proceed with the test to the PEO, who accepted that recommendation. 164 DASD(DT&E) FY 2016 Annual Report

171 Air Force AOC-WS DASD(DT&E) served as a member of the R4 team during the CCR process. R4 evaluates the management structure of the program office to determine whether it is sufficient to continue the program development effort. The Air Force SAE notified Congress of a critical change to the program driven by the numerous defects identified in DT that precluded the program s ability to meet the MS C decision within 1 year of the original estimate. The CCR has been completed, and the final CCR was submitted to Congress on November 21, The AOC-WS program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The test team noted a high probability of undiscovered significant problems because of the limited scope of DT1. The program office and PEO leadership concurred in the DASD(DT&E) recommendation to delay DT2 and declared a critical change when the program missed its threshold for MS C in July DASD(DT&E) supports the draft master schedule that was developed during the CCR process and recommends that the program office submit an update to the MS B TEMP to reflect the significant changes to the test strategy reflected in the new schedule. DASD(DT&E) FY 2016 Annual Report 165

172 B61-12 Life Extension Program (LEP) Tail Kit Assembly (TKA) Executive Summary: The air-delivered B61 thermonuclear bomb plays a critical role in supporting the airborne leg of the nuclear triad for the United States and allies abroad. The B61-12 weapon is capable of two delivery profiles: ballistic free fall (F-16C/D) and inertial guidance (B-2A, F-15E, F-35A, and B-21). The overall B61-12 LEP program is composed of a closely coordinated, DoD-managed TKA and the Department of Energy-managed bomb assembly. The TKA program entered a two-phased development phase in November Phase 1, now projected to complete in December 2017, focuses on TKA subsystem competitive prototyping, preliminary design, and critical design activities. Phase 2, awarded in December 2015, focuses on developmental and operational testing and manufacturing processes leading up to MS C. The program has conducted various captive carry and flight testing during FY 2016, collecting data to augment system performance models and to characterize the flight environment. The B61-12 TKA program is progressing through DT&E in accordance with its approved TEMP, but the program needs to update the TEMP, which was due before the start of EMD Phase 2. Lead DT&E Organization: 96th Test Wing Summary of FY 2016 DT&E Activities The B61-12 TKA program office conducted the following: o Guided Test Vehicle (GTV) flight test 3 on November 16, o CDR from January 19 to 22, o GTV 5 on May 24, 2016; GTV 4 on June 17, 2016; and GTV 6 on December 8, o EMD 2 integrated baseline review in July Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) assesses the program T&E schedule to be medium risk. Multiple test events have been delayed since entering EMD and there is about a 9-month delay to the baselined T&E program. These events are critical for informing a MS C decision, which has slipped 6 months to accommodate the delays. Even with this slip, there is compression of the test event schedule in FY 2017 and FY 2018 just prior to MS C. The B61-12 TKA program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The B61-12 TKA program is progressing through DT&E in accordance with the approved TEMP to collect appropriate data to inform a MS C decision. The planned future testing, however, is at medium risk because of schedule compression. 166 DASD(DT&E) FY 2016 Annual Report

173 Combat Rescue Helicopter (CRH) Executive Summary: The CRH is a rotarywing, dual-piloted, multiengine aircraft based on the in-production/fielded UH-60M helicopter. The primary mission of the CRH is to recover isolated personnel from hostile or denied territory. The CRH will operate from main operating bases and forward operating locations as well as from, near, and aboard single- and multi-spot U.S. Navy surface ships. The ACAT IC program conducted its PDR in April The Air Force was designated as the Milestone Decision Authority in May No DT has occurred to date to assess system performance. Lead DT&E Organization: 96th Test Wing Summary of FY 2016 DT&E Activities The CRH program completed the air vehicle PDR in April The Assistant Secretary of the Air Force for Acquisition held an in-progress review (IPR) Air Force Review Board and system demonstration test article purchase decision review in August Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) completed the MS B DEF assessment in July DASD(DT&E) conducted an assessment of DT&E planning supporting the IPR in August No DT has occurred to date to assess system performance. DT resources are adequate and integrated test planning is on schedule to support EMD exit criteria. A review of contractor modeling and engineering analysis indicates that the current CRH design is capable of meeting KPPs and KSAs. The program schedule remains aggressive between test aircraft delivery and MS C. Detailed interoperability test planning continues. The program has defined the CRH system architecture and identified all external and internal interfaces and controls. The current RGC is based on the legacy UH-60M helicopter and presents a reasonable growth slope from production through fielding. The program has completed the first phase of cybersecurity DT&E and is currently in Phase 2. The CRH program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The CRH program is currently executing under an accelerated schedule to deliver CRH assets 6 months ahead of the original fielding date in FY The current program schedule remains aggressive between test aircraft delivery and MS C. DASD(DT&E) FY 2016 Annual Report 167

174 Defense Enterprise Accounting and Management System (DEAMS) Executive Summary: DEAMS is a financial management system based upon Oracle s Enterprise Business Suite. The Air Force is implementing DEAMS to enable auditability of operations in the Air Force, U.S. Transportation Command, and Defense Finance and Accounting Service. DEAMS provides an enterprise-level view of critical financial data supporting decision making at all levels. It will replace legacy financial and accounting systems with a funds management solution qualified by the Joint Financial Management Improvement Program. DEAMS goals have been to deploy by 2017 to support financial improvement and audit readiness; reengineer, improve, and standardize financial business processes, rules, and language; and provide an authoritative source for Standard Financial Information Structure-compliant accounting data. Formal notification of the critical schedule breach to the senior official occurred in the May 2016 Defense Acquisition Executive Summary Report. DEAMS did not achieve full deployment decision (FDD) approval by the Acquisition Program Baseline threshold date of August 2015 because the program did not demonstrate effectiveness, suitability, or survivability during IOT&E and subsequent verification of fixes (VoF) testing. FDD was also not achieved by February 2016, slipping the milestone by more than 1 year, resulting in a critical change. DEAMS has updated the TEMP based upon recommendations resulting from the critical change process. Recommendations have been made in the areas of change management, transaction backlog, reports and query tools, and cybersecurity. Also, recommendations have included slowing upcoming DEAMS deployments to allow higher confidence in the delivery of key infrastructure and functionality improvements. Lead DT&E Organization: AFLCMC/HNIZ Summary of FY 2016 DT&E Activities DASD(DT&E) participated in the critical change process and provided inputs to the Management Team preparing the critical change report. DASD(DT&E) provided inputs to the DEAMS program to update the TEMP. The PMO resolved deficiencies found during FY 2016 DT events, leaving no unresolved Severity 1 defects. DASD(DT&E) provided input on deficiency burndown prior to fielding software. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) collaborated on assessment of the VoF test for evaluating the deficient areas as directed by the USD(AT&L) Acquisition Decision Memorandum dated September 30, DASD(DT&E) evaluated the technical maturity of DEAMS based on resolved Severity 1 defects and established plans for resolution of Severity 2 defects and clusters of Severity 3 defects. DASD(DT&E) assessed data and provided guidance relative to criteria for an FDD. 168 DASD(DT&E) FY 2016 Annual Report

175 Air Force DEAMS DASD(DT&E) confirmed that the VoF identified significant quantities of unprocessed transactions and user frustration leading to an evaluation that the system was not effective, suitable, or survivable. DASD(DT&E) supported the decision by the PEO to halt the VoF after the DEAMS PM and Air Force Operational Test and Evaluation Center determined that the remaining week of data collection was unlikely to significantly affect VoF results. DASD(DT&E) corroborated corrective actions to improve reporting capabilities when neither the Oracle Business Intelligence Enterprise Edition enhanced reporting tool nor the Defense Departmental Reporting System interface performed satisfactorily. DASD(DT&E) collaborated on Oracle R12 upgrade assessment planning centered on risk identification and mitigation in the areas of software testing complexity, required stability of the DEAMS baseline, and time needed to execute change management and training. DASD(DT&E) advised and coordinated the DEAMS TEMP update, which was approved on January 19, The DEAMS program did not request a waiver or deviation from requirements in the TEMP. Conclusion: DASD(DT&E) recommends that the program address issues in the areas of user testing, software testing complexity, change management, and cybersecurity. DASD(DT&E) FY 2016 Annual Report 169

176 F-22 Increment 3.2B Modernization Executive Summary: Increment 3.2B is an upgrade providing improved air-to-air capability to guarantee air superiority, improved air-to-ground capability using cooperative techniques with other F-22A aircraft to achieve passive target location accuracy, and enhanced electronic protection. Planned Increment 3.2B software DT&E included a series of four developmental Operational Flight Program (OFP) versions tested in Planned DT&E completion is more than 9 months later than the May 2013 MS B baseline program schedule but is still within acquisition threshold baseline. Lead DT&E Organization: 412th Test Wing Summary of FY 2016 DT&E Activities October December 2015, completed developmental flight test of the final incremental engineering OFP software. December 2015 March 2016, completed developmental flight test of the baseline full capability OFP software. March September 2016, flew developmental flight test of OFP software with correction of deficiencies discovered in DT&E. In FY 2016, the program flew 199 developmental flight test sorties on a total of 118 test missions, completing more than 603 flight test hours. Completed four of six cybersecurity T&E phases. Summary of FY 2016 DT&E Engagement and Assessments January 2016, DASD(DT&E) assessed the corrections of deficiencies discovered during DT&E as requiring further testing to evaluate full system performance and that OT aircraft are required to augment the developmental fleet for multiship test events. April 2016, DASD(DT&E) assessed that unplanned DT&E required before MS C is critical. Technical issues affecting displays and the enhanced stores management system (ESMS) require further evaluation. Live-fire weapons MS C entrance criteria were not completed. Resource issues increased schedule risk. June 2016, DASD(DT&E) provided a DT&E program assessment to support the MS C Air Force Review Board. October 2016, DASD(DT&E) assessed that expected corrections to known ESMS issues remain unresolved. Most of the other capability enhancement areas, including geolocation, embedded training, and legacy radar capabilities, continue to have issues. Unplanned dedicated test points/ missions required to collect additional data continue to pressure the schedule. The program did not request a waiver or deviation from requirements in the TEMP. The program office provided an updated TEMP in support of MS C. DASD(DT&E) recommended that the 170 DASD(DT&E) FY 2016 Annual Report

177 Air Force F-22 Increment 3.2B Modernization USD(AT&L) approve the MS C TEMP update. The USD(AT&L) signed the TEMP on September 7, DASD(DT&E) Program Assessment DASD(DT&E) provided a DT&E assessment to the Air Force in June 2016 with an update in July 2016 to support the MS C Air Force Review Board. The assessment recommended delaying the low-rate initial production (LRIP) decision until cockpit display issues are fixed and verified, and corrections for other critical deficiencies are identified and resourced. DASD(DT&E) assessed MS C entrance criteria for Deficiency Reports (DRs) as not met. Seven out of eight performance evaluation areas were assessed as not satisfactory. Urgent Category II DRs and open Priority 1 Watch Items were serious enough to warrant reconsideration of readiness to proceed with the LRIP decision. MS C entrance criteria for geolocation time synchronization, weapon integration, operational assessment, mean time between critical failure, and durability life test were met. The live-fire AIM-9X and AIM-120D MS C entrance criteria missions were successfully accomplished. Conclusion: Planned DT&E completion is more than 9 months later than the May 2013 MS B baseline program schedule but is still within acquisition threshold baseline. Known development technical issues and resource challenges increase the risk of completion of the replanned program schedule to medium. DASD(DT&E) FY 2016 Annual Report 171

178 Family of Advanced Beyond Line-of-Sight Terminals (FAB-T) Executive Summary: FAB-T enables strategic nuclear and nonnuclear C2 with extremely high frequency, protected, and survivable communications terminals for beyond line-ofsight communications. FAB-T provides nuclear-survivable terminals capable of communicating with Milstar (low data rate (LDR)) and advanced extremely high frequency (LDR/extended data rate) satellite constellations for airborne, ground-fixed, and transportable applications for nuclear C2. On October 26, 2015, the Defense Acquisition Executive formally approved MS C and entry into the Production and Deployment phase of the FAB-T command post terminal (CPT) subprogram. The FAB-T DT&E program continued with Block 2 software and DT&E of the ground-transportable antenna (GTA) and replacement antennas for the ground-fixed and airborne terminal configurations. Lead DT&E Organization: 96th Test Wing Summary of FY 2016 DT&E Activities November 20, 2015, Raytheon completed Block 2 software qualification testing. November 30, 2015 March 11, 2016, the 46th Test Squadron (46 TS) conducted the FAB-T SoS interoperability test to exercise the first of two phases of nuclear command, control, and communications DTs. January 20, 2016, Raytheon completed the GTA electromagnetic interference/electromagnetic compatibility testing. January 29, 2016, Raytheon completed system-level functional qualification testing (FQT) of the terminal with Block 2 software. April 2, 2016, Raytheon completed ground-fixed antenna (GFA) FQT. April 28, 2016, Raytheon completed the maintainability demonstration on the GTA and GFA. May 4, 2016, Raytheon completed GTA FQT. June 1, 2016, Raytheon completed the data-set manager DIACAP testing. June 23, 2016, Raytheon completed the terminal DIACAP testing. June 30, 2016, the 46 TS completed cybersecurity certification T&E. July 29, 2016, Raytheon completed the GFA high-altitude electromagnetic pulse test. August 26, 2016, the FAB-T Program Office completed voice conferencing risk reduction testing. September 1, 2016, the FAB-T Program Office completed mission planning risk reduction testing. September 7 9, 2016, the 46 TS conducted the first of three phases of FAB-T telemetry, tracking, and commanding DTs. September 28, 2016, Raytheon commenced the GFA environmental qualification testing. 172 DASD(DT&E) FY 2016 Annual Report

179 Air Force FAB-T Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) provided a program assessment to support the October 26, 2015, low-rate initial production (LRIP) decision, with a recommendation for continued FAB-T RGT to verify requirements with confidence. DASD(DT&E) engaged in Raytheon program management reviews (PMRs) on the development effort of the GTA and replacement antennas for the ground-fixed and airborne terminal configurations. DASD(DT&E) also engaged in PMRs on the production status of airborne and ground-fixed CPTs. DASD(DT&E) engaged in test readiness reviews for Raytheon qualification tests and functional tests to assess progress against the high-risk contractor test schedule. DASD(DT&E) prepared quarterly assessments for the Defense Acquisition Executive Summary (DAES) to inform DT&E status on system performance, T&E, cybersecurity, and interoperability. DASD(DT&E) is engaged through the FAB-T Integrated Test Team to review contractor and 46 TS test execution to evaluate satisfaction of the DEF developmental objectives to support key program decisions. Additionally, DASD(DT&E) tracks the integrated development for FAB-T with the Presidential and National Voice Conferencing capability. DASD(DT&E) reviewed the Air Force request to procure additional modification kits to support the installation of FAB-T CPTs during LRIP. DASD(DT&E) recommended concurrence to procure an additional 12 kits. DASD(DT&E) engaged with the FAB-T program office to assist in development of a plan for achieving and verifying reliability requirements. DASD(DT&E) provided DAES assessments and engaged with the program noting issues with development and testing of the ground-transportable CPT and the new ground-fixed and airborne antennas to support LRIP decisions planned in FY New antenna development and testing were subsequently extended and LRIP decisions delayed until antennas and system configurations completed full developmental qualification testing. DASD(DT&E) did not assess the force element terminal subprogram of FAB-T during FY 2016 as the Air Force prepares to conduct its cost capability analysis in FY The FAB-T program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The FAB-T program accomplished a variety of DTs on all the terminal configurations, although hardware issues resulted in developmental delays. Some of these hardware issues have been resolved and development is expected to steadily progress to support production decisions in 2017 as terminal configurations successfully complete their DT&E objectives. DASD(DT&E) FY 2016 Annual Report 173

180 Global Positioning System (GPS) Enterprise Executive Summary: GPS is a dual-use, military/ civil system providing real-time, accurate, worldwide positioning and timing services, thus enabling navigation to an unlimited number of users. Each satellite transmits an RF signal containing time and ephemeris data from which user equipment (UE) determines position, velocity, and time. The system operates in all weather and specified electromagnetically challenging environments while supporting peace and wartime operations in air, space, land, and sea domains. [Type a quote from the document or the summary of an interesting point. You can position the text box anywhere in the document. Use the Drawing Tools tab to change the formatting of the pull quote text box.] GPS IIF satellites are fully deployed; there are none left to launch. The Operational Control Segment (OCS) and the UE are in sustainment. All three segments are being upgraded through the GPS modernization programs. Lead DT&E Organization: SMC/GPEV Summary of FY 2016 DT&E Activities September December 2015, SMC/GPEV conducted the Integrated System Test (IST) 3-3 Phase 1 lab test to demonstrate the ability of Military GPS User Equipment (MGUE) cards to meet the accuracy KPP. September December 2015, Lockheed Martin conducted GPS III thermal vacuum testing and environmental testing of the space vehicle in preparation for launch. November December 2015, the military code live-sky test demonstrated key changes to the military code message format. January May 2016, Raytheon conducted the Next-Generation Operational Control Segment (OCX) Block 0 formal qualification test (FQT) software integration deficiency report (SWIDR); this test used verification/evidence to reduce the backlog of FQT deficiency reports (DRs). March April 2016, the OCX Block 0 configuration item qualification test (CIQT) retest verified DR fixes/corrective actions and established an information assurance hardened 1.5/Launch and Checkout System baseline. April 2016, the adjacent-band compatibility test developed interface tolerance masks for legacy GPS receivers. April May 2016, the GPS III electromagnetic interference (EMI)/electromagnetic compatibility (EMC) test demonstrated EMC with the space vehicle mission system and identified interference with the Nuclear Detonation (NUDET) Detection System (NDS) sensor that is currently being worked. April June 2016, the OCX Block 0 CIQT SWIDR verified that configuration DRs were fixed before the next Golden Dry Run (GDR). June 2016, Precision Test Solutions initiated Kemet capacitor qualification testing for GPS III to determine the reliability/suitability of space vehicle components. Recently, the Air Force has chosen to fly with unqualified parts and with unquantified risk. 174 DASD(DT&E) FY 2016 Annual Report

181 Air Force GPS Enterprise June August 2016, the OCX Block 0 CIQT GDR 3 test demonstrated a 95 percent pass rate of test requirements and a 57 percent reduction in failures over previous dry runs, and the results were accepted for run-for-record (RFR) status. July 2016, SPAWAR began the IST 3-3 Phase 2 lab test to demonstrate the functional requirements of the selected vendor MGUE cards and allow certification of the accuracy KPP. Government testing of the MGUE cards was delayed because of late delivery, and the cards had less functionality than expected. July September 2016, the GPS III transient filter unit augmentation concept high attenuation EMI/EMC emissions reducer testing demonstrated new RF filter performance designed to limit the NDS EMI issue discovered in an earlier test. August 2016, the military code live-sky test demonstrated satellite vehicle number 49 as a military-code-only capable space vehicle. August 2016, the Architecture Evolution Plan (AEP) adversarial cyber testing demonstrated the feasibility of the NCR to replicate AEP; this testing also evaluated network vulnerabilities. August 2016, MITRE completed an AEP cooperative vulnerability identification event in which a blue team conducted a cyber evaluation of the AEP 7.0 system in the consolidated test environment. September 2016, Raytheon began the OCX Block 0 CIQT RFR to collect additional test data that will be used with other collected data as evidence/justification for system acceptance. Summary of FY 2016 DT&E Engagement and Assessments Overall GPS DASD(DT&E) reviewed the updated GPS Systems Engineering Plan. DASD(DT&E) identified that the plan confused verification with evaluation, and DASD(DT&E) provided recommendations to avoid this confusion. DASD(DT&E) provided a GPS Contingency Operations (COps) independent DT&E assessment as part of a modification effort to support MS B. The GPS program did not request a waiver or deviation from requirements in the Enterprise Test and Evaluation Master Plan (E-TEMP). DT&E Strategy DASD(DT&E) reviewed Revision B to the GPS E-TEMP. DASD(DT&E) recommended changes to the enterprise and segment-/system-level DEFs, additional testing for cybersecurity evaluation, and changes to the acquisition strategy to better support test planning. DASD(DT&E) sponsored DEF development/improvement meetings for the following segments of GPS: the overall enterprise, MGUE, OCX, COps, and the selective availability anti-spoofing module mission planning system. Each meeting segment focused on informing key acquisition decisions, developing objectives to support these decisions, and identifying test events that would provide data for the objectives. DASD(DT&E) sponsored a series of technical exchange meetings to better develop the reliability testing approach for the program office. DASD(DT&E) assisted in development of reliability evaluation, testing opportunities within each system test, and collection of reliability data. DASD(DT&E) provided recommendations to MGUE managers for improvements to the Reliability Program Plan to increase confidence, through a reliability demonstration test, about how to meet an untestable mean time to failure requirement. DASD(DT&E) engaged with the GPS Program Office and through MGUE Test Working Groups, E-TEMP Working Groups, Threat Working Groups, and the Integrated Test Team to develop a new design of experiments (DOE) and test strategy for MGUE as card maturity continues to hamper overall testing progress. DASD(DT&E) FY 2016 Annual Report 175

182 Air Force GPS Enterprise DASD(DT&E) provided recommendations to PMs for the draft Revision B GPS E-TEMP in FY 2016 to improve the E-TEMP in the areas of developmental evaluation, information assurance/cybersecurity, reliability, and interoperability. MGUE DASD(DT&E) provided the GPS MGUE independent DT&E assessment to support a MS B decision. DASD(DT&E) led development of MGUE test designs using rigorous DOE. DOE has been shown to require fewer MGUE test cases than a non-doe T&E strategy. DASD(DT&E) reviewed the IST 3-3 Phase 2 Test Plan that evaluates the three vendors MGUE cards for readiness to support technical requirements. DASD(DT&E) identified improvements in entry/exit testing criteria and recommended that a more stringent DOE approach be incorporated into test planning. DASD(DT&E) supported quarterly reviews by the USD(AT&L) with the Assistant Secretary and Principal Deputy Assistant Secretary of the Air Force for Acquisition and the OCX Nunn- McCurdy and MGUE Independent Review Teams. OCX (formal assessment with Nunn-McCurdy) The program office is planning and implementing automated software development and test processes designed to decrease the frequency of software defects and reduce follow-on test and schedule risk. The program office is planning to procure additional hardware and web-enabled software environments necessary to complete software development, test, and integration. The T&E management structure is adequate in the long term if the program office successfully implements increased use of skilled test personnel and automated software testing and procures sufficient test resources. Until the restructured program s T&E strategy is reflected in a revised TEMP and demonstrated to be effective, the program s T&E schedule remains a risk area. DASD(DT&E) identified the need for the OCX (and OCS) program to characterize the current cybersecurity posture and update that posture for each subsequent capability delivery. DASD(DT&E) proposed the need for a CDD system survivability KPP requirement specific to OCX, cybersecurity endorsement, and independent cyber assessment in support of a future DAB. DASD(DT&E) recommended that the PM develop additional segment test labs to allow for multiple test environments to conduct parallel test activities. Conclusion: The program office DT&E strategy and execution are identifying design, coding, manufacturing, and integration defects for review, analysis, and corrective actions for all segments. The MGUE vendors card delivery with executable software ready for testing continues to be a risk for the test program. The E-TEMP, with a DASD(DT&E)-approved DT&E approach, is being updated to reflect the current test strategy and program of record. Actions to increase expert test personnel, funding, and resources identified in the OCX Nunn-McCurdy certification are needed to ensure sufficient testing of the progress toward OCX completion. 176 DASD(DT&E) FY 2016 Annual Report

183 Joint Space Operations Center (JSpOC) Mission System (JMS) Increment 2 Executive Summary: JMS provides space situational awareness (SSA) and C2 capability for the Commander, Joint Functional Component Command for Space. As the information hub of the space surveillance network (SSN), JMS enables an operator to process and analyze space observations from SSN sensors and other sensors to produce a comprehensive inventory/catalog of space objects. The catalog provides position and velocity information used to identify potential collisions, reentry, orbital debris, or other events of interest. JMS will also provide capabilities and applications to improve C2 of assigned space forces. Lead DT&E Organization: 96th Test Wing Summary of FY 2016 DT&E Activities September 29, 2015 February 28, 2016, JMS contractors and the Government integration team conducted integration testing at the SPAWAR JMS Enterprise Development and Integration (JEDI) lab in San Diego, California, and at the Mission Integration Enclave (MIE) at Vandenberg AFB, California. The 46th Test Squadron (46 TS) conducted scenario injection and generation system and end-to-end (E2E) message testing with the SSN sensors. October 26 November 23, 2015, the 46 TS conducted DT&E of the Web Integrated Space Situational Awareness version 6.0 application, part of the JMS SSA toolset. February 22 March 4, 2016, the 46 TS conducted a cybersecurity assessment of Service Pack (SP) 9 at the JEDI lab. March 7 11, 2016, the NCR conducted cybersecurity testing against the JMS cross-domain solution. April 4 22, 2016, the 46 TS conducted the first SP9 DT&E event (DT&E 1) at the JMS MIE. May July, 2016, the JMS and Space Fence programs conducted interface risk reduction testing in the contractor development environment. June 29, July 11, September 21 30, October 19, November 15, and December 12 16, 2016, the 46 TS conducted E2E sensor testing. July 18 August 5, 2016, the 46 TS conducted SP9 DT&E 2 at the JMS MIE. August 22 September 2, 2016, the 46 TS and the 25th Air Force conducted a cybersecurity risk assessment (CRA) of JMS SP9 at the JMS MIE. October 10 21, 2016, the 46 TS conducted SP9 DT&E 3 at the JMS MIE. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) engaged in the developmental test readiness review (DTRR) in September 2015 to assess progress against the entrance criteria. This DTRR provided status and identified liens against the entrance criteria that were worked off leading to a delta-dtrr in February DASD(DT&E) engaged with the program office and Integrated Test Team (ITT) to draft a TEMP update to support the critical change report (CCR) and program rebaseline. DoD sent the JMS Increment 2 CCR to Congress on September 13, The updated TEMP is in DASD(DT&E) FY 2016 Annual Report 177

184 Air Force JMS coordination and expected to be at DASD(DT&E) for approval by the 2nd quarter An updated DEF is included in the TEMP update. DASD(DT&E) is engaged through the JMS ITT, reviewing test planning and results to evaluate the DEF objectives to support key program decisions. DASD(DT&E) engaged with the JMS program to categorize deficiencies, clarified through the DEF, as liens and work them off before DT&E. The JMS Program Office in partnership with Navy SPAWAR, serving as the system integrator, evaluated approximately 85 percent of the mission thread scenarios and had a high failure rate in the following areas: manual difference correction, high accuracy catalog, orbital parameters distribution, observation processing, look angles, and conjunction assessment. DASD(DT&E) participated in DT&E 1, 2, and 3 to evaluate system progress. System performance and functionality did improve between these tests, but the system still had major deficiencies that need to be addressed before integrated T&E or OT&E. DASD(DT&E) participated in the JMS CRA, which confirmed a moderately strong security posture. Findings should be implemented to further strengthen JMS defenses. The JMS program implemented the RMF, aligned with the DASD(DT&E) four-step cyber process as part of the updated DEF, and is working with the NCR to enact a solid strategy to address cyber deficiencies affecting capability. DASD(DT&E) provided four Defense Acquisition Executive Summary Reports to the USD(AT&L) in FY 2016 regarding T&E, interoperability, and cybersecurity. o T&E: DASD(DT&E) reported accumulative successes as JMS progressed from integration to DT&E 1, DT&E 2, and DT&E 3. With each test, the system became more stable and deficiency discovery was reduced. However, major deficiencies were still open after DT&E 3, which led to the addition of a DT&E 4 event and another event to compare JMS performance with legacy systems. These added events resulted in a 6-month delay to OT&E, although the overall JMS critical path schedule was not affected. By April 2016, 7 of 10 technical measures, identified as entrance criteria from the DEF, were met. The remaining three measures were not tested pending completion of M&S. Despite the open M&S items, progress against the essential catalog capabilities allowed JMS to proceed into DT&E. The 46 TS completed only 60 percent of its planned test cases because of system deficiencies that prevented some test cases from running. System improvements were noted as the system progressed through the DT&E events. o Interoperability: DASD(DT&E) reported that the primary JMS Space Fence interface was designed and is being implemented during JMS SP11. The JMS program conducted several successful E2E sensor tests. o Cybersecurity: DASD(DT&E) observed and reported on the JMS CRA, which confirmed a moderately strong security posture. The JMS program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The JMS program rebaselined because of poor performance driven by an aggressive schedule and cuts in manpower and funding. The program concluded FY 2016 by executing to the rebaselined schedule, with residual schedule risk because of short corrective action periods between planned events. 178 DASD(DT&E) FY 2016 Annual Report

185 KC-46A Tanker Modernization Executive Summary: As the initial phase of a comprehensive aerial refueling recapitalization strategy, the KC-46 program will replace approximately one-third of the capability provided by the aging KC-135 fleet with 179 aircraft. The KC-46 will provide additional fuel and cargo capacity and will support aeromedical evacuation with greater fuel efficiency for Air Force, Navy, Marine Corps, and allied nation coalition force aircraft. The program completed MS C in August 2016, which was 1 year later than originally scheduled. As of September 29, 2016, the program had flown 1,105 hours and completed approximately 12,153 of 27,506 planned test conditions (38%) of the DT requirements. Lead DT&E Organization: 412th Test Wing Summary of FY 2016 DT&E Activities January 2016, refueling boom testing with the Air Force C-17 aircraft twice revealed axial loads exceeding the boom s structural limit. Post-flight inspection showed that the boom was not damaged. The contractor pursued software and hardware control solutions for the excessive boom structural load, but laboratory, ground, and flight testing over 7 months showed that only the hardware solution resolved the load issues. Test Aircraft 3 and 4 (EMD-3, EMD-4) flew for the first time and joined the test force. March 2016, testing of the wing air refueling pod (WARP) drogue with the Navy F/A-18 showed a near-limit load on its air refueling probe. WARP testing was suspended briefly and then successfully completed in May 2016 following a software change. July 2016, flight testing resumed with the new boom hardware and control system. The boom axial system showed good stability and load control in free air and with receivers. The program completed all air refueling entrance criteria for MS C with 82 boom contacts, 80 centerline drogue system contacts, and 136 WARP contacts, with a combined offloaded 156,900 pounds of fuel. The program completed amended type certificate testing for initial airworthiness, fuel system, engine fire extinguishing system, and a portion of the aero stability and control conditions. The program also completed supplemental type certificate testing for initial air worthiness, flutter, WARP buffet, and a portion of the aero stability and control, auto-landing, and fuel system testing. As of September 29, 2016, the program had flown 1,105 hours and completed 12,153 of the 27,506 planned test conditions for that point in time under the current schedule. DASD(DT&E) FY 2016 Annual Report 179

186 Air Force KC-46A Summary of FY 2016 DT&E Engagement and Assessments DT&E completion has been delayed by about 2 years primarily because of required electrical wiring redesign, removal, and reinstallation; required redesign of parts of the fuel system; as well as inadvertent introduction of a caustic fluid to the air refueling system during ground testing. DASD(DT&E) assesses that the flight test program has not achieved the projected flight test execution rates because of optimistic assumptions. DASD(DT&E) continues to work with the program office to develop realistic thresholds and objectives for test-related Acquisition Program Baseline criteria to avoid breaches in the future. DASD(DT&E) participated in the two CDRs for the boom telescoping system, providing engineering and operational inputs to support evaluation of the contractor s approaches. DASD(DT&E) participated in the post-ms C test schedule revisions to increase flight test productivity. WARP delivery delays will slip completion of the DT&E program and force IOT&E to be executed in phases, further delaying its completion. DASD(DT&E) remains concerned about the insufficient calendar time planned for correction of significant discrepancies and/or deficiencies discovered during DT before the planned start of OT based on previous experience with like programs. The start of IOT&E remains event driven, requiring the closing of significant open deficiencies before commencement. DASD(DT&E) continues to support military-specific cybersecurity and interoperability data analysis for military certifications. The KC-46A program did not request a waiver or deviation from requirements in the TEMP. DASD(DT&E) Program Assessment DASD(DT&E) released a DT&E assessment in July 2016 to support the MS C decision. DASD(DT&E) assessed that the program met all DT&E-related entrance criteria for MS C. Flight testing showed that the hardware modifications to the boom telescoping system proved effective in resolving the axial load issues. In addition, the major hardware issues discovered during DT&E, including the aircraft wiring and fuel system, had been resolved and the system configuration achieved hardware stability. Significant discrepancies remain to be resolved in the remote vision system and the air refueling operator s station ergonomics. DASD(DT&E) assessed that about 66 percent of DT&E remained to be accomplished after MS C. Conclusion: The program will likely complete DT&E about 2 years behind the initial baseline recorded in the August 2012 Integrated Master Schedule. 180 DASD(DT&E) FY 2016 Annual Report

187 Small Diameter Bomb Increment II (SDB II) Executive Summary: The SDB II (Guided Bomb Unit-53/B) is an Air Force-led ACAT ID program, with Navy participation, providing the Warfighter with the adverse-weather capability to attack mobile targets from standoff ranges (greater than 40 nautical miles) through weather. It uses a multimode seeker (semi-active laser/imaging infrared/millimeter wave) and dual-frequency weapons data link (Link 16/ ultrahigh frequency). SDB II is the second increment of the miniature munitions weapons system capability program consisting of a 250-pound class precision-guided air-to-ground munition utilizing three engagement modes. DT is in two phases with the first phase supporting Air Force F-15E aircraft fielding in FY 2018 and the second phase supporting F-35B (Marine Corps) and F-35C (Navy) aircraft fielding in FY DT consists of a contractor system verification program coupled with a Government-led 28-shot validation program. Raytheon Missile Systems is executing 16 guided test vehicle (GTV) releases, four in each engagement mode, and 10 live warhead shots to meet statutory live-fire test requirements. Six of the 28 Government test shots have been conducted as of this report to increase confidence in the system s capabilities and to gather additional captive-carry and free-flight reliability data before entering dedicated OT. To augment flight testing, the contractor is conducting 12 captive flight test (CFT) events with a missile seeker mounted on a UH-1 helicopter to further characterize sensor performance in differing environments. The program continues testing in accordance with the approved TEMP. Government-led testing was scheduled to complete in May 2017, but completion is now projected for August The 36 successful free-flight missions completed to date have demonstrated sufficient system performance, and reliability is trending to meet requirements by the end of IOT&E. Lead DT&E Organization: 96th Test Wing Summary of FY 2016 DT&E Activities May 2016, the Systems Program Office (SPO) completed a program management review to assess program status and the remaining DT&E. June 2016, the 96th Test Wing completed CFT at Redstone Arsenal, Alabama, and Eglin AFB, Florida, obtaining data to characterize the tri-mode seeker performance against a series of targets. August 2016, the SPO verified modifications to the design to meet environmental performance requirements. Following the failure of GTV 11, the 96th Test Wing successfully conducted the following shots in September and October 2016 at White Sands Missile Range, New Mexico: o Four GTV shots (normal mode). o Four GTV shots (laser mode). o Three GTV shots (coordinate attack mode). o Six live-fire shots. DASD(DT&E) FY 2016 Annual Report 181

188 Air Force SDB II October 2016, the SPO conducted Functional Configuration Audit (FCA) II of the design maturity to establish configuration control. FCA II completion is projected for the 4th quarter 2017 after closure of open issues. Summary of FY 2016 DT&E Engagement and Assessments DASD(DT&E) reviewed captive-carry and flight test data and provided a quarterly Defense Acquisition Executive Summary assessment. DASD(DT&E) observed Group 4 Government confidence testing at Eglin AFB. DASD(DT&E) participated in the Integrated Flight Simulation Working Group to verify the SDB II performance model and validate its accuracy against flight test data. The SDB II program office executed the DT program in accordance with the TEMP. The SDB II program did not request a waiver or deviation from requirements in the TEMP. Conclusion: The program continues testing in accordance with the approved TEMP with Government-led testing now projected to complete by August The 36 successful free-flight missions completed to date have demonstrated sufficient system performance, and reliability is trending to meet requirements by the end of IOT&E. Completion of Government DT and OT in time to support a January 2019 F-15E required asset availability (threshold) remains a concern. 182 DASD(DT&E) FY 2016 Annual Report

189 Space-Based Infrared System (SBIRS) Executive Summary: SBIRS is an integrated system of systems consisting of nonsurvivable and survivable space and ground elements. It provides unambiguous, timely, and accurate missile warning and missile defense information to the President of the United States, the Secretary of Defense, Unified Commanders, and other users. Additionally, SBIRS provides reliable, accurate, and timely infrared (IR) target signature and threat performance data to Warfighters, the intelligence community, weapon system developers, and other users. SBIRS also provides an IR view of the battlefield to enhance battlespace awareness to assist Warfighters in their combatant roles and missions. Lead DT&E Organization: SMC/RSE Summary of FY 2016 DT&E Activities November 2015, the program office (PO) completed work on an extensive ground control user interface change to improve operability of the Block 10 (B10) ground system by 460th Operations Group (460 OG) operators. December 2015, the PO completed a 10-day exercise using the new B10 ground system to have full C2 of all SBIRS ground and space operational assets, including failover/fallback of the system between the Mission Control Station (MCS) Increment 2 (MCS-2) and the MCS Backup (MCSB) Increment 2 (MCSB-2), and demonstrating a continuity of operations between the MCS and MCSB. These activities supported Warfighter confidence before the integrated test and evaluation (IT&E) soak period. January 2016, the PO completed a 2-week B10 IT&E readiness soak with 460 OG operators, demonstrating readiness to enter IT&E. This DT activity demonstrated effectiveness and suitability before entering IT&E. The Air Force Program Executive Officer for Space (AFPEO/SP) concurred in IT&E entry with the concurrence of officials from the Air Force Operational Test and Evaluation Center (AFOTEC), 14th Air Force (14 AF) Headquarters (HQ), and Air Force Space Command (AFSPC) HQ. February 2016, the PO supported the full set of power upgrades at MCS-2 with regression testing incorporated into the IT&E campaign. The PO proceeded ahead of the IT&E schedule while also completing B10-related tests with MDA to demonstrate interoperability and suitability objectives. March 2016, the PO completed the final system and crew readiness soak with the 460 OG on the new power baseline, demonstrating confidence in the upgrades and crew readiness to resume IT&E. The PO executed C2 cutover (C2CO) to the B10 after receiving 14 AF HQ approval to proceed. The B10 ground system controls the worldwide SBIRS space and ground operational assets. C2CO marked the beginning of the IT&E reliability, maintainability, and dependability (RM&D) data collection period. April 2016, the PO completed all IT&E effectivity tests; RM&D data collection continued. DASD(DT&E) FY 2016 Annual Report 183