System Test and Evaluation Policy

Transcription

1 ATEC Regulation 73-1 Test and Evaluation System Test and Evaluation Policy Headquarters U.S. Army Test and Evaluation Command Alexandria, VA 16 March 2006

2 This page intentionally left blank.

3 SUMMARY of CHANGE ATEC Regulation 73-1 System Test and Evaluation Policy This revision dated 16 March 2006 Clarifies ATEC support to Source Selection Evaluation Boards. Adds responsibilities of the Office of the Chief Counsel. Adds policy in support of Rapid Acquisition Initiatives. Adds policy in support of clothing and individual equipment (CIE) and organizational CIE. Adds policy in support of fielding selected RAI programs to the whole Army. Adds policy on Chemical and Biological Defense Program systems. Emphasizes the ATEC System Team (AST) responsibility for maintaining information in the ATEC Decision Support System. Clarifies the roles and responsibilities of the AST Chair and members. Emphasizes and clarifies the use of abbreviated evaluations for Acquisition Category III systems not on the Office of the Secretary of Defense Test and Evaluation Oversight List. Clarifies expected Early Strategy Review and Concept In-Process Review briefing content. Adds the Test and Evaluation Strategy document as a system planning document prior to Milestone B. Removes ATEC Senior Level Review Boards from policy. Clarifies the requirements for Test Support Packages prior to operational testing. Adds requirement for Test and Evaluation Master Plan status checks at operational test readiness reviews. Emphasizes the use of the Versatile Information Systems Integrated On-Line (VISION) Digital Library for archiving T&E products. Clarifies the marking and destruction of sensitive documents. Includes guidance for Outline Test Plan staffing and inclusion in the Test Schedule and Review Committee.

4 The revision dated 6 February 2004 Incorporates all approved policy guidance published before this updated regulation. Incorporates Joint Capabilities Integration and Development System (CJCSI C). Institutionalizes the capability-based evaluation strategy using the lens concept. Modifies the responsibilities and authority of the ATEC System Team Chair. Introduces combined test organizations and defines the ATEC role in combined test organizations/combined test teams. Describes the use of earned value management for test and evaluation. Implements rock drills to synchronize test and evaluation efforts across ATEC. Institutionalizes review boards for system evaluation plans and system evaluation reports. Focuses the Milestone B system evaluation report on risk assessment. Emphasizes early involvement in the review of requirements and contractual documents. Institutionalizes the quick-look report as a vehicle to track key system performance parameters and operational test progress. Streamlines reliability, availability, and maintainability data collection for operational testing. Implements and defines dummy data runs before operational test readiness review II. Adds guidance concerning the evaluation decision review. Clarifies the bottom-line summary as the first formalized presentation of a system s evaluation conclusions.

5 Department of the Army *ATEC Regulation 73-1 United States Army Test and Evaluation Command 4501 Ford Avenue Alexandria, VA March 2006 Test and Evaluation SYSTEM TEST AND EVALUATION POLICY History. This regulation was last revised on 6 February It includes the policies and responsibilities from ATEC Policy Guidance 04-2 and Interim Policy Guidance 04-6, 05-2, and ATEC Pamphlet 73-1 will provide additional guidance and procedures. Summary. This regulation promulgates ATEC policies for test and evaluation (T&E) of Army materiel and information technology systems. ATEC exercises overall management of assigned T&E programs. This regulation addresses guidance for developmental testing, operational testing, and system evaluation. Department of the Army (DA) officials use ATEC products (plans and reports) described in this regulation as input to acquisition decisions. Applicability. This regulation applies to Headquarters (HQ) ATEC and its subordinate command activities (SCAs). Army management control process. This regulation contains management control provisions and provides a Management Control Evaluation Checklist in appendix C. Supplementation. Supplementation of this regulation is authorized. When supplements are approved and issued by the SCAs, one copy of each supplement will be furnished to HQ ATEC (CSTE-TC). Suggested improvements. The proponent for this regulation is the Test Technology Directorate (TTD), ATEC. Users are invited to send comments and suggested improvements on DA Form 2028 (Recommended Changes to Publications and Blank Forms) through the chain of command to Commander, ATEC, ATTN: CSTE-TC, 4501 Ford Avenue, Alexandria, VA Distribution: BE FOR THE COMMANDER: OFFICIAL: THOMAS R. FAUPEL COL, GS Chief of Staff THOMAS C. CLOHAN Acting Deputy Chief of Staff for Information Management *This regulation supersedes ATEC Regulation 73-1, dated 6 February ATEC Regulation March 2006

6 Contents Chapter 1 Introduction Purpose References Explanation of abbreviations and terms Scope Policy basis for T&E Fundamental ATEC T&E policy...4 Chapter 2 Responsibilities... 7 Section I Headquarters, ATEC Staff Elements Commanding General Technical Director Chief of Staff Chief Counsel Public Affairs Officer (PAO) Deputy Chief of Staff for Information Management (DCSIM) Deputy Chief of Staff for Operations and Plans (DCSOPS) Deputy Chief of Staff for Resource Management (DCSRM) Test Technology Directorate (TTD) Deputy Chief of Staff for Engineering, Logistics, and Environment (DCSELE) ATEC Safety Manager...11 Section II Commanders/Directors of SCAs SCA Commanders/Directors Commander, U.S. Army Developmental Test Command (DTC) Commander, U.S. Army Operational Test Command (OTC) Director, Army Evaluation Center (AEC)...13 Section III ATEC System Team (AST) AST Chair AST member...15 Chapter 3 T&E in Support of Army Acquisition Overview Early T&E involvement Joint Capabilities Integration and Development System (JCIDS) T&E Working Groups ATEC System Team (AST) Combined Test Organization/Combined Test Team (CTO/CTT) Rapid Acquisition Initiatives (RAI) Fielding selected RAI programs to the whole Army Clothing and individual equipment (CIE) and organizational CIE programs Advanced Technology Demonstration (ATD) and Advanced Concept Technology Demonstration (ACTD) Multi-Service acquisition programs Joint T&E (JT&E) programs Chemical and Biological Defense Program (CBDP) Systems ATEC Regulation March 2006

7 Chapter 4 T&E Planning Section I System-Level Planning System-level planning process Evaluation strategy planning Level of evaluation Test strategy planning System planning tools System planning documents...30 Section II Event-Level Planning Process Event-level documents Event-level reviews...34 Chapter 5 Test Execution and Data Management DT execution guidelines OT execution guidelines RAM data collection Data management Data retention and disposition Data release Document markings and destruction Contractor involvement in testing...47 Chapter 6 T&E Reporting Documentation Overview System-level reports Event-level reports Technical notes...52 Appendix A References Appendix B Test Document Processing, Meeting, and Distribution Guidelines Appendix C Management Control Evaluation Checklist Glossary Table List Table 5-1. Data Retention Guidelines Table B-1. Processing Guidelines Table B-2. Meeting Guidelines Table B-3. Distribution Guidelines Figure List Figure 3-1. ATEC T&E Process Figure 4-1. Collaborative T&E Strategy Development Figure 6-1. System-Level Reporting Process ATEC Regulation March

8 Chapter 1 Introduction 1-1. Purpose This regulation establishes the ATEC policies and responsibilities for test and evaluation (T&E) of Army, Joint-Service, and Multi-Service systems References References are listed in appendix A Explanation of abbreviations and terms Abbreviations and terms used in this regulation are spelled out in the glossary. Terms critical to the reader s understanding of current T&E terminology are defined at first use Scope This regulation provides ATEC T&E policies and responsibilities for all systems developed and managed under Army Regulation (AR) ATEC Pamphlet 73-1 outlines the common procedures and processes that are used to implement the policies contained in this regulation Policy basis for T&E a. The ATEC T&E mission areas stem from statutory requirements stated in Title 10 United States Code (10 USC). b. The policies and responsibilities described in this regulation are based on Chairman of the Joint Chiefs of Staff (CJCS) Instruction (CJCSI) E, CJCS Manual (CJCSM) B, Department of Defense (DoD) Directive (DoDD) and DoD Instruction (DoDI) ; AR 73-1; and on other Army policies listed in appendix A Fundamental ATEC T&E policy The following paragraphs contain the fundamental statements of ATEC T&E policies. The remainder of this regulation provides specific ATEC policy for implementation. a. ATEC will accomplish its T&E mission by integrating tests and evaluations at the system or system of systems (SoS) level. Evaluation planning is always focused at the system or SoS level with the objective of determining in a realistic operational environment whether a system is operationally effective in accomplishing the overall mission and is suitable, safe, and survivable for use by Soldiers. The evaluation plan will identify all of the essential issues to be addressed and the data required to answer the issues. b. ATEC will form an ATEC System Team (AST) for all Office of the Secretary of Defense (OSD) T&E oversight programs to ensure the integrated coordination of all actions related to a system s T&E. Smaller, non-oversight systems will have an AST, even if it is composed of a single individual to monitor the program as required. The AST is responsible for developing and coordinating the integrated T&E program and for providing a focal point for outside agencies dealing with ATEC regarding that system. AST Chairs are designated jointly by the SCAs; the 4 ATEC Regulation March 2006

9 ATEC Technical Director (TD) will resolve any disagreements. The AST members will coordinate their recommendation through their individual chain of command. The AST Chair reports to and represents the ATEC Commanding General (CG) on AST matters. In most cases, Army Evaluation Center (AEC) will provide the AST Chair. c. ATEC will plan and integrate T&E programs through the AST. The SCAs execute and manage T&E programs. d. ATEC will conduct formal reviews (early strategy reviews (ESRs) and T&E concept inprocess reviews (CIPRs)) to provide guidance during the development of the system-level evaluation plans. The ATEC CG or designee chairs the ESR/CIPR for all OSD T&E oversight programs. The ESR and CIPR are prerequisites to completion of a system evaluation plan (SEP). ATEC will not approve a SEP until the system has successfully completed an ESR and a CIPR, all issues raised at the ESR and CIPR have been resolved, and the SEP has been staffed with AEC, Developmental Test Command (DTC), and Operational Test Command (OTC). e. The chain of command at the various organizational levels within ATEC will be responsible for successfully accomplishing the ATEC T&E mission. DTC plans, conducts, and reports developmental tests (DTs) for ATEC and for other customers according to their requirements. OTC plans, conducts, and reports operational tests (OTs) for ATEC and other customers according to their requests. AEC uses data from these and other credible events to produce integrated evaluations of the system in support of milestone decisions, full-rate production (FRP) decisions, and other key events. f. ATEC will evaluate systems in a realistic operating environment. For many systems, this is a SoS or a Joint environment. The command will accomplish this by determining capabilities and limitations at multiple levels of complexity starting with the individual system or its major subsystems and culminating with a synergistic evaluation of the system s contribution to the unit missions. g. Based on the issues and required data identified in the integrated evaluation strategy, ATEC develops an integrated program of tests or other events to provide the required data to support the evaluation. The integrated test strategy can include separate DTs, live fire tests (LFTs), independent OTs, combined and/or integrated DT/OT, modeling and simulation (M&S), and other credible events that generate data throughout the T&E program. The objective is to develop a synergy among all planned events to provide all the information for the system evaluator to report to the decision-maker. The strategy will support the integrated evaluation concept with the minimum essential testing to generate the required data. h. ATEC SCAs will fully document all of their plans and reports as they accomplish their integrated T&E mission. The primary planning documents are the SEP and the event design plan (EDP). The principal ATEC product generated from integrated T&E activities is the system evaluation report (SER), which is provided to the decision-maker at each major milestone and FRP decision within the system s acquisition life cycle. At other key points, a system assessment (SA) may be prepared. Formats are provided in ATEC Pamphlet 73-1 and are adaptable to encourage creativity and to provide flexibility for the wide spectrum of systems that ATEC tests and evaluates. ATEC will not release draft documents to external agencies; signature authority for ATEC plans and reports is presented in appendix B. All ATEC efforts will be documented by the AST in ATEC enterprise management systems to include the ATEC ATEC Regulation March

10 Decision Support System (ADSS) and the Versatile Information Systems Integrated On-Line (VISION) Digital Library (VDL). i. In developing integrated T&E strategies, ATEC will consider using combined and integrated DT/OT where appropriate. Generally, the use of DT/OT events is intended to save resources and to possibly shorten the timeline to complete testing. The concept for combining or integrating DT and OT is to conduct a single test program that produces credible qualitative and quantitative information that can be used to address both developmental and operational issues. DT/OT should not be used unless objectives of both the required DT and the OT can be fully supported. As a general rule, DT/OT is more appropriate earlier rather than later in the T&E program, but each AST should work to combine DT and OT whenever it is practical. For OSD T&E oversight programs, an initial operational test (IOT) will not be conducted with DT, for example, DT/IOT. For non-osd T&E oversight programs, the incorporation of an IOT with a DT should be the exception rather than the rule. Combining of DT and OT for chemicalbiological defense programs (CBDP) is governed by CBDP T&E Executive guidance. j. ATEC will consider using M&S while planning all T&E. M&S can be used to assist test planning, enhance instrumentation, serve as a test driver, serve as an additional source of data required for evaluation, and save valuable resources during test conduct. However, as a condition for proceeding beyond low-rate initial production (LRIP), the evaluation will not be composed of data based solely on computer M&S. k. In developing T&E strategies, ASTs will consider all command assets that have application to the system under test to ensure the most cost effective and complete support to the Army acquisition community and to the users of the system. This includes a review of all ATEC test ranges and facilities, field engineering expertise, M&S capabilities, resources (manpower and funding), and instrumentation. ATEC T&E will fully consider all applicable assets within the command in designing T&E plans, such as ATEC test ranges and facilities, field engineering expertise, M&S capabilities, resources, natural operating environments (climate and terrain), and instrumentation. ATEC will also consider external assets such as the manpower and personnel integration (MANPRINT) support provided to ATEC by the Army Research Laboratory, Human Research and Engineering Directorate. l. After publication of the SEP, ATEC will conduct rock drills for ATEC CG-designated systems to synchronize efforts across ATEC. The rock drills will identify and document each task required for executing the T&E strategy. See paragraph 4-5 for more information. m. ATEC will use ADSS to track and manage all T&E efforts for all programs, regardless of acquisition category (ACAT) designation. n. Integrated T&E goes beyond answering just technical issues. ATEC will conduct T&E to learn and understand what the acquisition team needs to know about system capabilities and limitations. ATEC will also test to learn where the program manager (PM) needs to focus to deal with the most important system challenges. ATEC will evaluate a system to provide the acquisition team with a valuable understanding of the effectiveness, suitability, and survivability (ESS) and safety of a system in accomplishing the intended mission. o. All T&E plans will consider the safety and well being of ATEC personnel, player personnel, and support contract personnel during the conduct of any T&E event. Safety will be covered broadly in EDPs and more specifically in detailed test plans (DTPs). An approved 6 ATEC Regulation March 2006

11 safety release is required prior to any test event involving non-test certified Soldiers using the equipment. ATEC will comply with the policies in AR regarding human use committees (HUCs). p. Reliability, availability, and maintainability (RAM) data will be obtained primarily from Government developmental testing. Government DT will be designed to collect comprehensive RAM data under conditions that duplicate operational field conditions as closely as possible. RAM data collection during operational tests should focus on identifying what fails and the conditions under which the failure occurs and what it takes to get the system operating again. Secondary sources may include PM-conducted or -sponsored contractor testing; operational testing; and analytical studies, models, simulation, and other sources. q. ATEC personnel will use AR , DA Pamphlet 73-1, and ATEC Pamphlet 73-1 as guidance to include the appropriate threat portrayal in all test events. Test and evaluation personnel must coordinate with ATEC TTD personnel as well as the U.S. Army Training and Doctrine Command (TRADOC) Deputy Chief of Staff for Intelligence, Fort Leavenworth, Kansas, and DA G-2 to ensure that the threat represented in T&E conforms to the approved threat and that threat test documents are prepared in a timely manner. r. ATEC SCAs will collect, reduce, and manage data obtained from any credible event in a manner that permits an analyst unfamiliar with the events to quickly understand how the data are generated and how they are organized for analysis. ASTs are encouraged to consider using any credible prior test information (including information from commercial manufacturers; users; and other Services, agencies, or countries) available on the system. s. ATEC will release data as quickly as possible to the appropriate members of the acquisition community. The ATEC CG retains the authority to make the final determination on the release of ATEC OT data for OSD T&E oversight programs. t. As the executive agent for the Army G3/5/7, ATEC will chair and conduct the Test Schedule and Review Committee (TSARC) to provide Army-level centralized management of resources for OT, force development test and/or experimentation (FDT/E), and troop support for DT. u. ATEC personnel will not normally be voting source selection members. The approval authority for participation as a source selection voting member is the ATEC CG. ATEC personnel may provide technical or T&E expertise to source selection, to include Source Selection Evaluation Boards (SSEBs), when requested. Personnel who are source selection voting members should avoid participation in future evaluations of that system. Chapter 2 Responsibilities Section I Headquarters, ATEC Staff Elements 2-1. Commanding General The ATEC Commanding General will ATEC Regulation March

12 a. Determine when a Combined Test Organization/Combined Test Team (CTO/CTT) will be established, and will sign the charter defining the organization for ATEC. b. Chair the ESR and CIPR for OSD T&E oversight programs. c. Approve requirements for RAM data collection beyond the standard level for specific OSD T&E oversight program operational tests. d. Chair the rock drill for all designated systems. e. Approve the SEP, SA, and SER, and concur with the test and evaluation master plan (TEMP) for all OSD T&E oversight programs. f. Approve the Capabilities and Limitations (C&L) reports for rapid acquisition initiative (RAI) programs. g. Chair operational test readiness reviews (OTRRs) for OSD T&E oversight programs. h. Chair the General Officer TSARC as the executive agent for the Army G3/5/7. i. Recommend materiel release to the Materiel Release Review Board (MRRB) for OSD T&E oversight programs Technical Director The ATEC Technical Director (TD) will a. Provide oversight of T&E planning, execution, and reporting. b. Recommend approval and/or disapproval of T&E documents requiring ATEC CG approval. c. Chair OTRRs when requested by ATEC CG or OTC CG. d. Chair the ESR and CIPR when requested by ATEC CG Chief of Staff The ATEC Chief of Staff will a. Coordinate all staff actions that support accomplishing the ATEC T&E mission. b. Ensure policies, directions, and orders from ATEC CG are executed Chief Counsel The ATEC Chief Counsel will provide guidance on the application of 10 USC 2399 and Federal Acquisition Regulation (FAR), Subpart 9.5, to potential conflicts of interest in the use of support contractors involved in ATEC test and evaluation activities Public Affairs Officer (PAO) The ATEC PAO will provide guidance on the release of information that pertains to military matters, national security issues, or subjects of significant concern to DoD. This includes all test and evaluation information requested by the public or media. 8 ATEC Regulation March 2006

13 2-6. Deputy Chief of Staff for Information Management (DCSIM) The ATEC DCSIM will a. Develop, coordinate, and implement ATEC policies on automation, communications, record management, printing and publications, library management, and visual information support to the ATEC T&E mission. b. Design, operate, and maintain the ATEC Internet and intranet sites, to include publication of ATEC policies and procedures. c. Provide guidance to those instrumentation programs that use information management hardware and software. d. Maintain the ATEC Technical Library. e. Develop and manage the ATEC Enterprise system and its individual components (ADSS, VISION, etc.) Deputy Chief of Staff for Operations and Plans (DCSOPS) The ATEC DCSOPS will a. Provide staff oversight of ATEC T&E activities. (1) Coordinate, plan, and conduct the TSARC, including Outline Test Plan (OTP) submissions. Chair associated working-level TSARC. (2) Develop and staff the Five Year Test Program (FYTP). (3) Provide cost management oversight for T&E programming, budgeting, and executing. (4) Coordinate T&E estimates for program objective memorandum (POM) input. (5) Manage the ADSS. (6) Schedule T&E briefings for the ATEC Command Group (e.g., ESR, CIPR, rock drills, TSARC, etc.). (7) Manage T&E early involvement through a dedicated T&E Coordination Office/Liaison Office structure. (8) Coordinate with the TRADOC Army Capabilities Integration Center (ARCIC) for RAI programs selected to be fielded to the whole Army. b. Generate the initial ADSS record for each system. c. Manage Army support for Joint T&E. d. Serve as initial point of contact (POC) for Advanced Technology Demonstrations (ATDs), Advanced Concept Technology Demonstrations (ACTDs), Army Experimentation and Transformation, and other similar efforts. e. Coordinate the HUC review process. f. Provide staff oversight on aviation support issues concerning DT and OT. g. Manage security operations supporting T&E. ATEC Regulation March

14 h. Coordinate ATEC support for the Army s RAI. i. Coordinate ATEC support for clothing and individual equipment (CIE) and organizational CIE (OCIE) programs. j. Serve as the proponent for the development, upgrade, maintenance, and use of the ADSS program. k. Monitor existing ASTs for effectiveness and compliance with T&E policies Deputy Chief of Staff for Resource Management (DCSRM) The ATEC DCSRM will a. Manage and administer T&E funding programs. b. Program and budget funds to support T&E from the year of execution through the end of the POM cycle. c. Manage the manpower authorizations within ATEC to adequately support the T&E mission Test Technology Directorate (TTD) The ATEC TTD Director will a. Document and promulgate ATEC T&E policy. b. In conjunction with SCAs, develop T&E methods, procedures, and tools to improve ATEC T&E processes. c. Provide scientific, technical, and policy advice to the ATEC workforce on T&E matters. d. Develop and maintain the ATEC Test and Evaluation Basic Course computer-based training and resident seminar. e. Manage the ATEC threat program. (1) Support the validation and accreditation of threat targets (actual threat systems, surrogates, integrated and add-on stimulators, and virtual) and of threat simulators (actual threat systems, surrogates, simulators, and stimulators), and the verification, validation, and accreditation (VV&A) of threat portrayal in M&S. (2) Review TEMPS, SEPs, and EDPs for threat adequacy, consistency, and realism. (3) Assist the ASTs in obtaining test threat support documents, including Threat Test Support Packages (TTSP). f. Manage the ATEC Modeling and Simulation and Instrumentation (M&S&I) program. Coordinate and promulgate ATEC policies for the development and use of M&S&I in support of T&E. (1) Support VV&A of models used to collect T&E data. (2) Support certification of instrumentation used to collect T&E data. g. Serve as the proponent for the development, upgrade, maintenance, and use of the VDL program. 10 ATEC Regulation March 2006

15 h. Serve as the ATEC representative on Director, Operational Test and Evaluation (DOT&E) Central Test and Evaluation Investment Programs (CTEIP) to include the Resource Enhancement Program and the Joint Improvement Program. i. Provide input to U.S. Army T&E Management Agency (TEMA) for the Army Test Resources Master Plan (ATRMP). The ATRMP goals are prioritized to shape the future Army T&E infrastructure and focus investments Deputy Chief of Staff for Engineering, Logistics, and Environment (DCSELE) The ATEC DCSELE will a. Provide oversight to ATEC activities regarding compliance with applicable environmental laws and regulations. b. Provide major command (MACOM) level management and oversight for ATEC s ammunition program and provide SCA-level management and oversight for DTC s ammunition program ATEC Safety Manager The ATEC Safety Manager will provide oversight to ATEC activities regarding compliance with applicable safety regulations and policies. Section II Commanders/Directors of SCAs SCA Commanders/Directors SCA Commanders/Directors will a. Provide AST members and Chairs as required. b. Empower AST members with the broadest scope of authority possible, and ensure that the scope is mutually understood. c. Review quality and timelines of input and/or products. d. Support the ATEC enterprise management systems. Ensure that ADSS is kept current. e. Support ATEC s RAIs. f. Attend all ATEC CG-designated program rock drills in person. g. Ensure that AST participation is a priority task for the AST members; resolve workload conflicts and priorities. h. Ensure AST members function within the coordinated single position of the AST. i. Support accreditation of M&S used in testing. Perform verification and validation (V&V) of M&S for testing as appropriate. j. In accordance with ATEC Regulation 70-15, manage the development of ATEC M&S and instrumentation projects. k. Support DoD and DA science and technology programs. ATEC Regulation March

16 2-13. Commander, U.S. Army Developmental Test Command (DTC) In addition to the SCA Commander/Director responsibilities in paragraph 2-12, the DTC Commander will a. Plan, execute, and report developmental tests. b. Research, develop, and acquire needed test capabilities and methodologies. c. Provide test facilities and test expertise in support of the acquisition of Army and other defense systems and for other customers on a cost reimbursable basis. d. Provide data collection and instrumentation support to other SCAs when requested. e. Manage the development of test operations procedures (TOPs) and international test operations procedures (ITOPs) to standardize testing. f. Approve DT planning and reporting documents in accordance with guidance in this regulation. g. Develop and submit OTPs to the DCSOPS for test events requiring resources outside ATEC. h. Operate the Army portion of the Major Range and Test Facility Base (MRTFB), except Ronald Reagan Ballistic Missile Defense Site and the High Energy Laser Support Test Facility. i. Provide safety releases in a timely manner before a system is used for testing, training, maintenance, or demonstrations by non-test certified Soldiers. j. Provide safety confirmations for milestone decision reviews (MDRs), materiel release decisions, and fielding decisions. k. Execute development, upgrade, maintenance, and daily operation of the Army Test Incident Reporting System (ATIRS). l. Execute development, upgrade, maintenance, and daily operation of the VISION program. m. Review contractor DT plans and witness contractor testing as required. n. Ensure DT documentation is reviewed by the HUC as appropriate. o. Provide the Accreditation Action Officer (AAO) when the primary use of M&S is DT support Commander, U.S. Army Operational Test Command (OTC) In addition to the SCA Commander/Director responsibilities in paragraph 2-12, the OTC Commander will a. Plan, execute, and report operational tests for systems assigned to ATEC. b. Research, develop, and acquire needed test capabilities and methodologies. c. Provide test facilities and test expertise in support of the acquisition of Army and other defense systems and for other customers on a cost reimbursable basis. 12 ATEC Regulation March 2006

17 d. Provide data collection and instrumentation support to other SCAs when requested. e. Approve OT planning and reporting documents in accordance with guidance in this regulation. f. Develop and submit OTPs to the DCSOPS for test events requiring resources outside ATEC. g. Perform the operational tester duties for joint tests; FDT/E; customer tests; joint, combined, or integrated DT/OT events; Urgent Materiel Releases; RAIs; and Concept Experimentation Program (CEP). h. Conduct and report results from ACTDs, ATDs, and Advanced Warfighting Experiments (AWEs) when requested by the demonstration manager or TRADOC. i. Conduct OTRRs, when authority is delegated by ATEC CG. j. Develop training programs for test team members. k. Develop and maintain the ATEC RAM standard software for collecting and processing RAM operational test data and providing test incident reports to the ATIRS. l. Develop the OT database structure, dictionary, and linkage information in coordination with the AST and overall lifecycle system-level database requirements. m. Develop the dummy data, populate the database, and produce a dummy authenticated level 3 database for validation of evaluation readiness by AEC. n. Ensure OT documentation is reviewed by the HUC as appropriate. o. Provide the AAO when the primary use of the M&S is OT support Director, Army Evaluation Center (AEC) In addition to the SCA Commander/Director responsibilities in paragraph 2-12, the AEC Director will a. Perform the duties of system evaluator for all systems assigned to ATEC. b. Research, develop, and acquire needed analysis and evaluation capabilities and methodologies. c. Perform independent logistics supportability assessments and report them to the Assistant Secretary of the Army for Acquisition, Logistics, and Technology (ASA(ALT)). d. Approve evaluation planning and reporting documents in accordance with this regulation. The AEC Director will inform ATEC CG and the ATEC Technical Director when approving a SER or SA with adverse findings of operational ESS or when nonconcurring on the TEMP. The AEC Director approves all evaluation plans and reports for non-oversight systems. e. Execute the ATEC continuous evaluation mission. f. Oversee the development of the evaluation strategy for each system using ESRs and ensure the documentation of this strategy in the TEMP and SEP. g. Ensure resource requirements are included in all OTPs submitted to DCSOPS for TSARC approval. Ensure the correctness of system and continuous evaluation data in ADSS. ATEC Regulation March

18 h. Develop the overarching system-level data model. i. Assist in the development and submission of OTPs to the TSARC for test events requiring resources outside ATEC. j. Validate the data reduction and analysis process using the authenticated dummy level 3 database, and verify that the AST is ready to compute and analyze evaluation measures. k. Chair the Evaluation Decision Review (EDR) for OSD T&E oversight programs prior to OTRR 2, and report results to ATEC CG and the OTRR Chair (if not ATEC CG). l. Prepare ATEC evaluation briefings, assessments and reports with the assistance of AST members. m. Provide a memorandum stating the ATEC position on the type of materiel release being requested. n. Recommend materiel release to the MRRB for non-oversight programs. o. Chair the ESR/CIPR for non-oversight programs. p. Provide the AAO when the primary use of the M&S is evaluation. Section III ATEC System Team (AST) AST Chair The AST Chair will a. Synchronize and integrate all T&E activities for the assigned systems. Assigned systems include regular acquisition systems, RAI, fielding selected RAI programs to the whole Army, CIE, OCIE, ATD, and ACTD. b. Represent the ATEC CG on AST matters. The Chair has direct access to the ATEC CG to resolve issues, when required. c. Serve as the lead voice for the AST and the command on the system, and be responsible for ensuring ATEC attendance at outside meetings and other events. d. Lead AST meetings and briefings, steer the development of AST consensus, and be ultimately responsible for generating, staffing, and approval of ATEC T&E documents and products. Coordinate staffing actions to meet the staffing timeline and suspense of each SCA. e. Ensure that each AST member is aware of AST-related presentations, meetings, or discussions with ATEC leadership and T&E integrated product team (IPT) agencies, especially PMs, where an AST position is being developed or considered. f. Concur with the OTP prior to submission to DCSOPS for inclusion in the TSARC. g. Consolidate and submit AEC funding requirements to the PM for non-otp evaluation costs. h. Coordinate and schedule all AST meetings and briefings to SCA and ATEC leadership. i. Be a team leader who leverages the expertise of the team members to determine the proper T&E approach for ATEC. 14 ATEC Regulation March 2006

19 j. Consolidate and staff comments on system-related non-atec documents. k. Keep ATEC and SCA leadership informed by continually documenting efforts and recording program changes in ADSS AST member Each AST member will respond to multiple levels of leadership. They report to the AST Chair for the assigned systems and report to/within the SCA for SCA-related activities. AST members will support and assist the AST in developing ATEC requirements for the system T&E and will represent ATEC with the coordinated ATEC position on program issues. While AST participation is a priority, it is recognized that the workload of some members may present conflicts. These conflicts should be resolved within each member s chain of command. The AST member will a. Obtain an SCA position before any AST meeting. b. Coordinate SCA responsibilities for system test and evaluation, and report back to AST members. c. Respond to all taskings from the AST Chair, and inform the Chair as soon as possible when unable to complete a tasking. d. Immediately bring to the attention of the SCA chain of command those issues that cannot be resolved within the AST. e. Review and validate all OTPs prior to submission to DCSOPS for inclusion in the TSARC. Each AST member is responsible for ensuring their resource requirements are correctly documented in the OTP and for concurring with other members requirements. f. Brief their area of responsibility at rock drills. Each member will provide full support to the AST Chair at the rock drill. g. Support the AST Chair by putting SCA-specific information into ADSS. Chapter 3 T&E in Support of Army Acquisition 3-1. Overview Today s Army has a growing need to ensure that fielded systems and the new systems in development are designed to work as part of a larger SoS. Each component of the Army s SoS must retain compatibility and interoperability in all required operational environments. These components within the SoS must be integrated and tested routinely during development and especially prior to fielding or each time a new system is added to the larger SoS. ATEC, in the process of completing its mission, must test and evaluate the systems in the SoS and Joint environment at each stage of the acquisition cycle. As part of the evaluation, ATEC will assess the synergism of the global system s capability and capacity to support the overall unit s mission. ATEC Regulation March

20 The framework for the Army and ATEC T&E process is pictured below. It fits within the DoD acquisition framework, as ATEC s processes and products are integrated into the OSD model for a complete and comprehensive system evaluation. System Level AST Standup ESR CIPR SEP or ASEP Rock 1 Drill Bottom Line Summary SAR SER 5 or ASER or SA TEMP 5 Event Level Safety 5 Confirmation EDP DTP TRRs Test Execution Data Release Test Report Safety 2 Release HUC 2 Dummy Data EDR Pilot 3 Test End-to-End 3 Draft 4 SER DAG 3 Quick 3 Look Reports Scoring Conf Notes: 1 ACAT I or CG-designated systems 2 For events using non-test certified soldiers 3 For OT 4 For last event prior to MS or FRP 5 Required for MS B, MS C, FRP DR Figure 3-1. ATEC T&E Process 3-2. Early T&E involvement Early involvement by the T&E community in the acquisition process is a cornerstone for enabling program success. This regulation implements this concept by establishing procedures for developing and obtaining early command approval of the test and evaluation concept tailored for the individual program. ATEC early involvement in the acquisition process will ensure that T&E programs, strategies, and objectives are consistent throughout the acquisition program. Since T&E results figure prominently in the decisions reached at design and milestone reviews, early T&E involvement in the acquisition process serves to add value to the final product of any acquisition program Joint Capabilities Integration and Development System (JCIDS) a. The JCIDS process provides a concepts-centric methodology to identify joint capabilities that will allow joint forces to meet the full range of military challenges out into the future. In order to accomplish this transformation, this process must produce capability proposals that consider the full range of doctrine, organization, training, materiel, leadership and education, 16 ATEC Regulation March 2006

21 personnel, and facilities (DOTMLPF) solutions in order to advance joint war fighting. Information is specified in CJCSI E and CJCSM B. b. ATEC evaluations are based on the joint military capability needs identified, assessed, and prioritized in the JCIDS process. The documentation developed during the JCIDS process provides the formal communication of capability needs between the operator and the acquisition, T&E, and resource management communities. The JCIDS documents are the initial capabilities document (ICD), the capability development document (CDD), and the capability production document (CPD). Based upon the capability needs identified in these documents, ATEC will develop a SEP and a supporting integrated program of events to provide the required data T&E Working Groups a. Multidisciplinary integrated concept teams (ICTs) are formed using representatives from the Army, academia, and industry to identify potential solutions for military requirements before Milestone (MS) A. These ICTs are terminated or transitioned to IPTs led by the materiel developer (MATDEV), such as the T&E Working-level Integrated Product Team (WIPT) around MS A. b. An overarching IPT (OIPT) for the program assists the PM by focusing on strategic guidance, program assessment, and issue resolution. c. The T&E WIPT serves as a working advisory body to the PM and integrates all DTs, OTs, and LFTs; Army system evaluations; and M&S activities. Other subordinate IPTs may be identified to assist the T&E WIPT. d. ATEC will establish an AST upon receipt of a new capability document or upon notification of a new program. The AST coordinates ATEC input into the T&E WIPT. ATEC will provide membership to the T&E WIPT from each SCA. ATEC representation to the T&E WIPT will be led by the AST Chair or designated representative. e. ATEC personnel will not normally be voting source selection members. The approval authority for participation as a source selection voting member is the ATEC CG. ATEC personnel may provide technical or T&E expertise to source selection when requested ATEC System Team (AST) ATEC will plan, manage, and coordinate T&E for the assigned system through the AST. a. An AST will be established upon notification of a new requirement from any source or receipt of justification for new requirements (i.e., ICD, CDD, or significant upgrades/ improvements to an existing system). When a new or potential requirement becomes known, ATEC DCSOPS will coordinate the formation of the AST with the SCAs. b. Prior to the first AST meeting, the AST Chair is designated jointly by the SCAs. Replacement selections for the AST Chair will be made in sufficient time to allow a seamless transition. If agreement cannot be reached, the ATEC TD will make the selection. c. The AST composition for Major Defense Acquisition Programs (MDAPs) and OSD T&E oversight programs will include representatives from all three SCAs. Grouping of several smaller, similar systems into a single AST is acceptable and encouraged. However, every system requires separate planning and reporting documents to support the T&E program. ATEC Regulation March

22 d. The AST will identify and resolve system T&E issues. Issues will be resolved at the appropriate organizational level. When required, issues will be rapidly raised to the ATEC CG for resolution. Differences will be resolved prior to attendance at T&E WIPTs. The AST will present a single coordinated ATEC position at the T&E WIPT meetings. e. The AST will be responsible for the synchronization and integration of ATEC efforts for the assigned system. f. The AST will speak with one voice when interacting with organizations outside ATEC. g. The AST will review the capabilities stated in the ICD, CDD, CPD, and the critical operational issues and criteria (COIC) to ensure that they are reasonable and achievable, measurable and testable, and consistent with the initial capabilities document. h. The AST will review the request for proposal (RFP) and statement of work (SOW) for the system to ensure that they include adequate testing and information sharing to support the system T&E. However, ATEC personnel are not normally voting source selection members; the approval authority for participation as a voting member is the ATEC CG. ATEC personnel may provide technical or T&E expertise to source selection, to include Source Selection Evaluation Boards (SSEBs), when requested. Personnel who are source selection voting members should avoid participation in future evaluations of that system. i. The AST will keep ATEC and SCA leadership informed by continually updating program status and milestones in ADSS Combined Test Organization/Combined Test Team (CTO/CTT) ATEC will participate in CTO/CTTs on selected major programs to support integrated test operations. The CTO/CTT consists of representatives from the PMO, prime contractor or lead system integrator (LSI), and ATEC. The organization integrates DT and OT planning, shares test resources, and minimizes the duplication of test resources and the time required to execute combined testing, while preserving OT independence. Each CTO/CTT is unique and is defined by its negotiated charter. A sample charter is provided in ATEC Pamphlet a. ATEC will facilitate test planning for selected systems by forming a CTO or CTT with the PM and the prime contractor. Because a CTO/CTT requires considerable full-time commitment of manpower, this structure will be the exception rather than the rule. b. The ATEC CG, in coordination with the Program Executive Officer (PEO), will determine when a CTO/CTT is appropriate. c. Each CTO/CTT will have a charter that defines the organization and functions and must be approved by the PM; the ATEC CG; and the contractor. It will define both the personnel and resources that each of the three organizations agrees to provide to the CTO/CTT. The charter will also describe the authority and responsibility of the CTO/CTT and must represent an equal partnership among the three organizations that sign it. The CTO/CTT will operate with three co- Chairs representing the PM, ATEC, and the prime development contractor. The PM representative is normally designated as the managing partner and will generally run the day-today operations. The three Chairs will make key decisions jointly. If they cannot resolve an issue, then it will be elevated to the PM, contractor, and ATEC CG to be resolved. 18 ATEC Regulation March 2006

23 d. ATEC representation on the CTO/CTT will come from ATEC test commands (OTC and DTC) and will include members from the AST. e. A senior tester from either DTC or OTC will lead the ATEC members in the CTO/CTT and will speak for ATEC in the three-person CTO/CTT decision-making process. f. In order to maintain the independence required by Title 10, USC, the evaluator will not be a member of the CTO/CTT. Since close coordination is required between the evaluation team and the testers, AEC will normally identify a liaison to work with the CTO/CTT. g. The IOT will be coordinated with the CTO/CTT, but will be executed and evaluated by ATEC without direct PM or contractor involvement. h. The ATEC chain of command will rate designated representatives to the CTO/CTT with letter input provided by the CTO/CTT supervisor(s). i. Once a CTO/CTT is created, it will remain in place throughout the acquisition process, unless disbanded by the ATEC CG and the respective PEO. j. Once designated by the ATEC CG and the PEO, the CTO/CTT will act as the single integration team for component, system-level, and SoS testing. k. ATEC CTO/CTT representatives must work with other CTO/CTT members to ensure the execution of a T&E program at minimal costs and duplication of effort in order to meet both PM/contractor and evaluator data requirements. l. ATEC CTO/CTT members work with other CTO/CTT members for the overall good of the program. However, they are expected to support the ATEC position as developed by the AST and approved by the ATEC chain of command. m. ATEC test organization personnel operating within the CTO/CTT are also members of the AST. Thus the AST overlaps with the CTO/CTT, ensuring that there are no misunderstandings between the CTO/CTT and the AST position. ATEC members of the CTO/CTT will reflect and support the AST position during internal CTO/CTT discussions Rapid Acquisition Initiatives (RAI) a. The RAI process applies to ATEC s involvement in the Army s RAI process and Urgent Materiel Release (UMR) of equipment in support of the rapid equipping of warfighting commanders. This effort is coordinated by the ATEC DCSOPS Rapid Response Division. RAI systems are not part of the Defense Acquisition Program system, but follow an outline of the ATEC T&E process. b. DTC develops a safety confirmation prior to the RAI system being released to troops. c. The AST develops an RAI T&E Concept that is approved by ATEC CG or his designee. At the T&E Concept briefing, the CG will determine whether a C&L Report is required in addition to the Safety Confirmation. (1) The T&E Concept serves the purpose of the ESR/CIPR processes that are used for a formal acquisition program, and will provide direction to the AST. All AST members will be involved in the development and presentation of the AST s proposed T&E concept to ATEC leadership, regardless of the format that is used. ATEC Regulation March