DEPARTMENT OF THE ARMY UNITED STATES ARMY TEST AND EVALUATION COMMAND 4501 FORD AVENUE ALEXANDRIA VA 22302-1458 CSTE-TTP 4 April 2008 MEMORANDUM FOR SEE DISTRIBUTION 1. References: a. ATEC Regulation 73-1, System Test and Evaluation Policy, 16 Mar 06. b. Memorandum of Agreement (MOA) on Multi-Service Operational Test and Evaluation (MOT&E) and Operational Suitability Terminology and Definitions, Oct 07. 2. Purpose. The purpose of this IPG is to provide policy and guidance on the implementation of the standard test and evaluation (T&E) document names agreed to by all the Operational Test Agency (OTA) Commanders and the Director, Operational Test and Evaluation (DOT&E) at the 2007 Fall OTA Commanders Conference. These names are documented in the Memorandum of Agreement (MOA) on Multi-Service Operational Test and Evaluation (MOT&E) and Operational Suitability Terminology and Definitions, October 2007. ATEC Regulation 73-1 as well as ATEC Pamphlet 73-1 will be updated to reflect these changes. 3. Policy. a. This policy does not affect the rapid initiative programs. ATEC will continue to write Capabilities and Limitations reports. b. ATEC will use the document names approved by all the OTA Commanders for all programs that ATEC assesses and/or evaluates, whether single-service or multi-service programs. c. Not all the names of ATEC planning documents were changed. The planning documents are listed below: (1) System Evaluation Plans (SEPs) and SEP Updates (SEP-Us) will continue to be used by ATEC. (2) Event Design Plans (EDPs) are replaced with OTA Test Plans (OTA TPs). (3) Outline Test Plans (OTPs) are replaced with Test Resource Plans (TRPs). (4) Detailed Test Plans (DTPs) will continue to be used when appropriate.
(5) Army Input to the OTA Test Plan (AI OTA TP) will be used on multi-service programs when ATEC is not the lead Service. d. ATEC has changed the names of all assessment and evaluation reports. ATEC will not use the terms System Assessment (SA), System Evaluation Report (SER), or Milestone (MS) B SER. e. The name of the reporting document is tied to when the document is prepared in the acquisition cycle. For example, all documents prior to the Full-Rate Production (FRP) decision are called assessments, while documents for the FRP and later are called evaluations. Assessment and evaluation are defined below: (1) Assessment is defined slightly differently by each Service; however, the basis is the same assessing risk/progress towards meeting system requirements and assessing risk/progress towards a determination of effectiveness, suitability, and survivability. (2) Evaluation is the determination of the system s effectiveness, suitability, and survivability. f. ATEC will use the document names as outlined below. Table 1 provides an overview of the documents. Enclosure 1 provides samples of the new OTA Test Plan, OTA Assessment Report, OTA Milestone x Assessment Report, OTA Evaluation Report, and OTA Follow-On Evaluation Report cover sheets. (1) OTA Assessment Report (OAR) is used to document the capabilities, limitations, and progress of the acquisition program in meeting its requirements based on test results. An OAR will not be used for reporting acquisition program effectiveness, suitability, and survivability to DOT&E for an acquisition decision. It replaces the System Assessment. (2) OTA Milestone x Assessment Report (OMAR) is used to document the acquisition program s risk and progress towards meeting effectiveness, suitability, and survivability. It replaces the MS A SER, the MS B SER, and the MS C SER (unless MS C is for an FRP decision, in which case an OTA Evaluation Report will be used instead). (3) OTA Evaluation Report (OER) is used to document the acquisition program s demonstration of its effectiveness, suitability, and survivability. It replaces the FRP SER. (4) OTA Follow-on Evaluation Report (OFER) is used to document the acquisition program s risk and progress towards meeting effectiveness, suitability, and survivability criteria beyond FRP. It replaces both the SA and SER. (5) Operational Test Report (OTR) or Abbreviated Operational Test Report (AOTR) will be used by the Operational Test Command (OTC) to document test results from all 2
operational test events. The AOTR is in a Technical Document format. The OTR will be the document of record for operational tests that require an in-depth test report. (6) Army Input to the OTA Report (AIOR) will continue to be used on multi-service programs when ATEC is not the lead Service. The actual name of the report depends on what the report is supporting; milestone decision is an OMAR, and full-rate production decision is an OER. 3
Table 1: Report Products New Name (Old Name) Purpose Signature Goes To Test Plan Supports OAR (SA) Risk assessment/ progress towards IOT&E Oversight ATEC CG All others AEC Director Accommodate all Service OTA desires Non MS program decisions/status/ as required OMAR (MS A/B/C SER) Risk assessment/ progress towards IOT&E Oversight ATEC CG All others AEC Director DOT&E-Oversight, MDA, Service Chief, PEO, PM, User, others as desired MS A/B/C decisions OER (SER) Evaluation of effectiveness, suitability, and survivability Oversight ATEC CG All others AEC Director DOT&E-Oversight, MDA, Service Chief, PEO, PM, User, others as desired Full Rate Production (FRP) Decision Review OFER (SA/SER) Evaluation of effectiveness, suitability, and survivability Oversight ATEC CG All others AEC Director DOT&E-Oversight, MDA, Service Chief, PEO, PM, User, others as desired Evaluation of Follow-on Test AOTR (TDR) Provide test findings, test data, and test team observations Test Org CDR Evaluators, PM OAR, OMAR, OER, OFER OTR (TR) Provide detailed test report with supporting analysis of test issues and criteria based on data collected during the OT Test Org CDR Evaluators, PM OAR, OMAR, OER, OFER 4
4. Procedures. a. Plans. (1) The SEP or SEP Update (SEP-U) will continue to describe the strategy for assessing effectiveness, suitability, and survivability and evaluating the contribution of the system to overall mission capability. The SEP/SEP-U also describes the strategy for identifying system capability limitations and assessing risks and the potential impact on mission capability. (2) The OTA Test Plan (OTA TP) will document the test design, supporting methodology, and analytic details required for the specific test for each operational test (OT), live fire test (LFT), and selected developmental test (DT). The OTA TPs for acquisition categories (ACATs) I and II and OSD Oversight list programs for OT and LFT are provided to DOT&E for approval prior to the test. The content and format of the OTA TP has not changed from the current EDP format. (a) OTA TPs are structured, as necessary, to focus on the priority measures that are the focus of the particular test event. Not all of the priority measures have to be addressed in a single test and/or single OTA TP. (b) All OTA TPs must provide a clear indication of how the test will be executed. (c) All OTA TPs will continue to be signed by the US Army Evaluation Center (AEC) Director and the Test Command s commander. (3) The TRP is prepared for all tests that require Army or other Service personnel or resources that ATEC cannot provide. Examples include Soldier support, training ranges, instrumentation, flying hours, standard ammunition, or training devices. TRPs are developed, updated, and staffed (internally and externally) using the ATEC Decision Support System (ADSS). The current OTP format will be used for the TRP. b. Reports. (1) The OTA Assessment Report (OAR) may be written at any time and will address the technical maturity of the system throughout development and demonstration, production, and post deployment. The OAR may address the required capabilities of the system, the status of corrective actions, readiness for OT, or potential impact on mission capability and sustainment. The OAR may identify needed modifications; provide information on techniques, tactics, doctrine, organizations, and personnel requirements; or evaluate the logistic supportability of the system. The scope of issues to be addressed is flexible and may or may not cover all aspects of effectiveness, suitability, and survivability (ESS). The OAR is not tied to a materiel decision review. The current SA format will be used for the OAR. 5
(2) The OTA Milestone Assessment Report (OMAR) is a summary report focused on a specific acquisition milestone, either A, B, or C. The content of an OMAR will vary based on the MS it is supporting. The OMAR assesses risk/progress towards meeting system requirements and assesses risk/progress towards a determination of effectiveness, suitability, and survivability. For example, a MS A OMAR will contain very little test information while a MS B OMAR will basically be a risk assessment. The current format used for the MS B Risk Assessment will be used for the OMAR. (3) The OTA Evaluation Report (OER) is the summary report provided to the Milestone Decision Authority (MDA) at the FRP decision review. Its purpose is to report on the effectiveness, suitability, and survivability and evaluate the contribution of the system to overall mission capability. The current SER format will be used for the OER. (4) The OTA Follow-on Evaluation Report (OFER) will document the results for post- FRP decision OT evaluations. The current SER format will be used for the OFER. c. The following decision flow chart (figure 1) depicts what document to use and when it is needed. 6
Acquisition Program? No Capability & Limitation Report (CLR) Where in Acquisition Process? Prior to Full-Rate Production (FRP) Full-Rate Production (FRP) and Beyond Report Purpose? Report Purpose? Non-Milestone Milestone FRP? Beyond OTA Assessment Report Full-Rate Production? OTA Evaluation Report OTA Follow-on Evaluation Report No OTA Milestone x Assessment Report OTA Evaluation Report Figure 1. Document Decision Tree 7
5. Effective date. This policy becomes effective upon signature. This policy affects documents that are in draft as well as new documents. Only documents that are currently in the ATEC Command Group for signature will not be changed. 6. The ATEC Decision Support System (ADSS) is being modified to reflect the new document name changes. The AST Chairs will ensure the information for their programs in ADSS is correct. The expected effective date for the ADSS modification is 2 June 2008. 7. Point of contact. The POC for this action is Mrs. Dianne Luna, CSTE-TTP, 703-681-2747, DSN 761. FOR THE COMMANDER: Encl Sample Cover Sheets THOMAS R. FAUPEL COL, GS Chief of Staff Distribution: BE 8
Sample Cover Sheets January 2008 2008-OTC-1234 OTA Test Plan Produced as input to support a Milestone C Production Decision Manpack Radio Limited User Test Test plan produced by U.S. Army Operational Test Command Distribution limited to U.S. Government agencies only, test and evaluation, month year. Other requests for this document must be referred to: Commander, USAOTC (TEOT-OP), 91012 Station Avenue, Fort Hood, TX 76544-5068. DESTRUCTION NOTICE: Destroy by any method that will prevent disclosure or reconstruction of the document. This document contains information EXEMPT FROM MANDATORY DISCLOSURE under the Freedom of Information Act. Exemption 5 (predecisional materials) applies. FOR OFFICIAL USE ONLY Figure 2. OTA Test Plan 9
January 2008 OTA Assessment Report Produced as input to support a Materiel Release Decision Manpack Radio Report produced by U.S. Army Evaluation Center Distribution limited to U.S. Government agencies only, test and evaluation, month year. Other requests for this document must be referred to: Director, USAEC (TEAE-XX), 4501 Ford Avenue, Alexandria, VA 22302-1458. DESTRUCTION NOTICE: Destroy by any method that will prevent disclosure or reconstruction of the document. This document contains information EXEMPT FROM MANDATORY DISCLOSURE under the Freedom of Information Act. Exemption 5 (predecisional materials) applies. FOR OFFICIAL USE ONLY Figure 3. OTA Assessment Report 10
January 2008 OTA Milestone C Assessment Report Produced as input to support a Milestone C Production Decision Manpack Radio Report produced by U.S. Army Evaluation Center Distribution limited to U.S. Government agencies only, test and evaluation, month year. Other requests for this document must be referred to: Director, USAEC (TEAE-XX), 4501 Ford Avenue, Alexandria, VA 22302-1458. DESTRUCTION NOTICE: Destroy by any method that will prevent disclosure or reconstruction of the document. This document contains information EXEMPT FROM MANDATORY DISCLOSURE under the Freedom of Information Act. Exemption 5 (predecisional materials) applies. FOR OFFICIAL USE ONLY Figure 4. OTA Milestone X Assessment Report 11
January 2008 OTA Evaluation Report Produced for the support of the Full Rate Production Decision Manpack Radio Report produced by U.S. Army Evaluation Center Distribution limited to U.S. Government agencies only, test and evaluation, month year. Other requests for this document must be referred to: Director, USAEC (TEAE-XX), 4501 Ford Avenue, Alexandria, VA 22302-1458. DESTRUCTION NOTICE: Destroy by any method that will prevent disclosure or reconstruction of the document. This document contains information EXEMPT FROM MANDATORY DISCLOSURE under the Freedom of Information Act. Exemption 5 (predecisional materials) FOR OFFICIAL USE ONLY Figure 5. OTA Evaluation Report 12
January 2008 OTA Follow-on Evaluation Report Produced as additional input after the Full Rate Production Decision Manpack Radio Report produced by U.S. Army Evaluation Center Distribution limited to U.S. Government agencies only, test and evaluation, month year. Other requests for this document must be referred to: Director, USAEC (TEAE-XX), 4501 Ford Avenue, Alexandria, VA 22302-1458. DESTRUCTION NOTICE: Destroy by any method that will prevent disclosure or reconstruction of the document. This document contains information EXEMPT FROM MANDATORY DISCLOSURE under the Freedom of Information Act. Exemption 5 (predecisional materials) applies. FOR OFFICIAL USE ONLY Figure 6. OTA Follow-on Evaluation Report 13