Department of the Army *ATEC Regulation 73-21 United States Army Test and Evaluation Command 4501 Ford Avenue Alexandria, VA 22302-1458 23 August 2004 Test and Evaluation ACCREDITATION OF MODELS AND SIMULATIONS FOR TEST AND EVALUATION *History. This is a new ATEC regulation. Summary. This regulation provides policy and guidance regarding the verification, validation, and accreditation of models and simulations for use by the U.S. Army Test and Evaluation Command. Applicability. This regulation applies to all personnel attached to ATEC Headquarters (HQ) and its subordinate command activities. Supplementation. Supplementation of this regulation is prohibited without the prior approval of ATEC HQ (CSTE-TT). When supplements are approved and issued, ensure one copy of each supplement is furnished to ATEC HQ (CSTE-TT). Suggested improvements. The proponent for this regulation is the Director of Test Technology, ATEC. Users are invited to send comments and suggested improvements on DA Form 2028 (Recommended Changes to Publications and Blank Forms) through the chain of command to Commander, ATEC, ATTN: CSTE-TT-PMD, 4501 Ford Avenue, Alexandria, VA 22302-1458. Distribution: BE FOR THE COMMANDER: OFFICIAL: RICK A. DORSEY COL, GS Chief of Staff CYNTHIA A. CASSIL Deputy Chief of Staff for Information Management *This regulation supersedes OPTEC Memorandum 73-21, dated 15 February 1996. 1
Contents 1. Purpose... 2 2. References... 2 3. Explanation of abbreviations and terms... 2 4. Scope... 2 5. Definitions... 3 6. Policy... 3 7. Responsibilities... 4 8. Procedures... 5 9. Accreditation process... 7 Appendix A References... 9 Appendix B Sample Accreditation Plan Memorandum... 10 Appendix C Sample Accreditation Request Memorandum... 11 Glossary... 12 1. Purpose This regulation provides policy and guidance regarding the verification, validation, and accreditation (VV&A) of models and simulations (M&S) for use by the U.S. Army Test and Evaluation Command (ATEC). 2. References References are listed in appendix A. 3. Explanation of abbreviations and terms Abbreviations and terms used in this regulation are explained in the glossary. 4. Scope a. This policy applies to M&S used to support test and evaluation (T&E), whether provided to, sponsored by, or developed within ATEC. Models and simulations include, but are not limited to, hardware-in-the-loop facilities, digital simulations, system stimulators, emulators, and synthetic environments. b. This policy does not apply to (1) Training simulations used by test personnel before testing, when competent authority has certified the training. (2) Threat simulators that are surrogates for live threat equipment (see DA Pam 73-1, Appendix Z). (3) Simulations that are the systems/items under acquisition T&E. 2
(4) Chemical and biological agent simulants (see Code of Federal Regulations, Title 32, Chapter V, Part 626, Subpart B--Biological Defense Safety Program). (5) Analytical tools used only to study results and not used to produce those results. Analytical tools include commercially available and well-documented analytical products such as linear programming solvers, spreadsheet, or database applications. 5. Definitions a. Verification is the process of determining that the M&S accurately represents the developer s conceptual description and specifications. Verification evaluates the extent to which the M&S has been developed using sound and established software engineering techniques. b. Validation is the process of determining the extent to which the M&S is an accurate representation of the real-world from the perspective of the intended use of the M&S. Validation methods include expert consensus, comparison with historical results, comparison with test data, peer review, and independent review. c. Accreditation is the official determination that a model, simulation, or federation of M&S is acceptable for use for a specific purpose. 6. Policy a. ATEC will integrate M&S into system test and evaluation to support requirements definition, system design and development, and system assessment. M&S can assist in predicting performance throughout the mission space; support test planning and control; support range safety; enhance the operational realism of the test environment; support more economical, timely, and controlled test execution; add to understanding in interpreting collected data; or demonstrate the significance of conclusions. By law, operational T&E of major programs may not be based exclusively on modeling, simulation, or analysis. b. ATEC will accredit models and simulators when their use may affect the evaluation of the performance or military utility of the system under test. An ATEC accreditation authorizes M&S implementation in test and evaluation. c. The proposed use of M&S for T&E is developed early in the program life cycle and briefed in conjunction with the system concept in-process review (CIPR). Approval of the M&S use begins the accreditation process. d. Accreditation of M&S that support the T&E mission (e.g., M&S for test planning, range safety, and test visualization) but do not support a specific system acquisition evaluation is the responsibility of the M&S Sponsor. e. For multi-service operational test and evaluation, M&S accreditation is the responsibility of the lead operational test agency. 3
7. Responsibilities a. Commander, ATEC. The ATEC Commander accredits the M&S selected for T&E and may delegate the accreditation authority (AA) to a designee. b. Accreditation Authority (AA). The AA is the approval authority for accreditation of models and simulations. The ATEC Commander or his/her designee serves as the AA. In general, the ATEC Technical Director (TD) accredits M&S in support of evaluation for Acquisition Category (ACAT) I and Department of Defense (DOD) oversight programs, and the Director, Army Evaluation Center (AEC) accredits M&S in support of evaluation for other programs. The AA reviews and approves the accreditation plan prior to execution and the accreditation request prior to use of the M&S to support T&E. c. Accreditation Action Officer (AAO). The AAO oversees the development and implementation of an independent ATEC accreditation. The ATEC System Team (AST) identifies the appropriate subordinate command activity (SCA) AAO to lead the accreditation for each model and simulation supporting a T&E program. In general, the Developmental Test Command (DTC) or Operational Test Command (OTC) will provide the AAO when the primary use of the M&S is test support, and AEC will provide the AAO when the primary use of the M&S is evaluation. The AAO will (1) Develop, with the AST/SCAs/proponent organization, coordinated accreditation plans and acceptability criteria. (2) Prepare and submit documentation (accreditation plan, report, and request) for AA approval. (3) Coordinate accreditation efforts and provide oversight of verification and validation (V&V) activities as required. (4) In the case of instrumentation-simulation hybrids, ensure that requirements for accreditation of M&S and for certification of instrumentation are satisfied. d. ATEC Test Technology Directorate (TTD). The ATEC TTD provides support to and assists the ASTs. Once a decision is made to incorporate M&S as an element of the T&E strategy, ATEC TTD assists in developing accreditation plans and requests. The ATEC TTD will (1) Provide ATEC policy and guidance regarding the selection, use, and accreditation of M&S supporting T&E. For instrumentation having an M&S component, ensures that ATEC VV&A concerns are addressed (e.g., in the requirements for items being developed for ATEC by the Project Manager Instrumentation Targets and Threat Simulators). (2) Provide and update document examples (found at ATEC TTD intranet web page) for accreditation requirements to assist the ASTs in planning, conducting, and reporting results of VV&A activities. (3) Advise on threat representation in M&S. (4) Review accreditation plans and requests for ACAT I and DOD oversight programs. 4
(5) Maintain a library of approved VV&A plans, accreditation requests, and lessons learned. e. Subordinate Command Activities. The SCAs will (1) Provide AAO as required. (2) Review accreditation plans and reports as part of the document staffing process. For M&S in support of ACAT I and DOD oversight programs, submit accreditation plans and reports (signed by the Commander/Director of the AAO) to the ATEC CG with a short memorandum requesting approval. (3) Resource, or identify for customer reimbursement, requirements for VV&A activities and M&S implementation. (4) Develop implementation policy and guidance as appropriate and manage the support infrastructure necessary to ensure successful employment of M&S in support of T&E. (5) Support the AST efforts to resource and verify, validate, and accredit M&S internally or through customer reimbursement as appropriate to support T&E. (6) Plan, develop, sponsor, and implement test stimulation, including the preparation of resourcing estimates for the required test stimulation. (7) Review formal support packages to ensure adequacy for use in support of system T&E and document findings in the appropriate test report. f. Test and Evaluation Working-level Integrated Product Team (T&E WIPT). The T&E WIPT will (1) Relate the planned use of the M&S to system requirements and evaluation issues and criteria. (2) Identify the T&E goals and limitations that prompt the use of M&S. (3) Identify the M&S resources required. Resources include software, hardware, labor, temporary duty (TDY), contract costs, facilities, VV&A, and any other projected requirements. 8. Procedures a. Accreditation. (1) M&S developed or sponsored within ATEC that directly or indirectly influences evaluation must undergo formal accreditation. Formal accreditation addresses intended uses, acceptability criteria, and other requirements with adequate documentation upon which to base an accreditation decision. The AAO leads the process of determining the accreditation requirements, and execution of accreditation activities and reporting. (2) M&S developed or sponsored outside ATEC is accredited by ATEC HQ before the M&S is used to support test and/or evaluation. Simulations that are embedded in the system under test must undergo formal accreditation before they are used as a source of data for T&E. Adequate V&V documentation is obtained from the model developer (whether conducted by 5
that developer or an independent V&V agent) to support accreditation. ATEC will not accredit the simulation for use in support of T&E without documented V&V. (3) When the AA is the ATEC TD or CG, the accreditation plan and request are submitted through the appropriate SCAs and the ATEC Test Technology Directorate to the AA for approval. (4) For M&S applications used in other than T&E, the proponent will conduct V&V, and the primary user (M&S Sponsor) will accredit the M&S. (5) For simulations used in post-test analysis and evaluation, pre-test simulation results are compared to field test results to validate the simulation application. If significant differences are found, the model is calibrated (adjusted), but only after careful analysis discloses clear reasons why, and what calibration is appropriate; e.g., the atmospheric and environmental test conditions accounted for differences in test performance. In cases where model results fall outside of acceptable tolerances for the output, the M&S proponent must provide ATEC with details of the planned modifications of the M&S. b. Documentation. (1) Test and Evaluation Master Plan (TEMP) and Simulation Support Plan (SSP). Planned M&S applications and projected resource requirements (including VV&A) are documented in the system TEMP and SSP. The M&S requirements and applications, and the accreditation timeline are updated in each SSP and TEMP update. (2) System Evaluation Plan (SEP). The overall concept for employment of M&S to enhance system evaluation is described in the SEP. Through the SEP, the AST identifies each measure of performance, and to what degree it is likely to be affected by the use of the simulation. (3) Event Design Plan (EDP). A detailed description of the M&S environment and implementation in the test is provided in the EDP. (4) System Evaluation Report (SER). When M&S is used to address the specific measures of effectiveness or measures of performance, the details are documented in the SER. (5) Support Packages. Simulation results may be part of formal support packages officially approved and provided to ATEC by another command for test planning, in accordance with its normal mission functions. Examples of formal support packages that might be based on simulation include probability of kill (P(k)) tables from Army Materiel Systems Analysis Activity (AMSAA), doctrinal and organizational training support packages (DOTSP), threat test support packages (TTSP), and training support packages. SCAs will review formal support packages to ensure adequacy for use in support of system T&E. The review must be documented. When formal support package is accepted by ATEC, any M&S results obtained are considered accredited. (6) Accreditation Plan. The accreditation plan will identify the evaluation issues driving the requirement for the M&S, describe the planned M&S application, and identify data requirements and acceptability criteria for accreditation of the M&S. 6
(7) Accreditation Request. The accreditation request will provide an accreditation report that includes documentation identifying whether or not the acceptability criteria were met and a short accreditation request memorandum requesting approval. 9. Accreditation process The critical documents required in accrediting M&S for use in support of system T&E are the accreditation plan and the accreditation request. a. Accreditation Plan. (1) The acceptability criteria identify quantitative and qualitative metrics based on the intended use. Qualitative and quantitative acceptability criteria are based on evaluation criteria, intended uses of M&S, and an understanding of M&S outputs. (2) V&V planning and execution can be conducted either within ATEC or by an outside V&V agent, in which case the AAO should designate a V&V agent to monitor the V&V planning and execution. In either case, the accreditation plan may be used to describe fully the V&V event, to verify the simulation functionality, and to determine its degree of validity prior to its use in testing or evaluation. Such events may include testing, scientific or statistical analysis, search of information on previous VV&A results, subject matter expert review, and review of prior use. At the discretion of the AAO, the V&V plan and accreditation plan are consolidated into a single document. (3) The AAO, in conjunction with the AST, will determine when to submit the accreditation plan to support the planning and development of V&V activities. The SCA Commander/Director of the AAO signs the accreditation plan and submits to the AA with a short accreditation plan memorandum requesting approval. The AA initials the memorandum to approve. (Note: The AA does not sign the plan.) A sample memorandum is provided in appendix B. (4) The accreditation plan is approved prior to conducting each V&V event(s). b. Accreditation Request. (1) The request will recommend accreditation with any limitations on the use of the M&S in testing or in the subsequent evaluation. (2) The AAO will submit the accreditation request through the ATEC SCAs. For ACAT I and DOD oversight programs, the AAO will forward the SCA-coordinated accreditation request through ATEC TTD to the ATEC CG for approval. The AAO will forward other programs V&V and accreditation planning documents to the appropriate SCA to the Director, AEC for approval. (3) The accreditation request is submitted no later than 30 days prior to M&S test event or other events that may produce data for the evaluation. The SCA Commander/Director of the AAO signs the accreditation report and submits to the AA with a short accreditation request memorandum requesting approval. The AA initials the memorandum to approve. (Note: The AA does not sign the report.) A sample memorandum is provided in appendix C. 7
(4) An accreditation is required before the M&S is used to support T&E. Pending accreditation is sufficient cause to delay entry into testing. c. The accreditation steps and decision process diagram are shown in figures 1 and 2. Documentation Development Steps Responsible Agent Approval Agent Step 1. Accreditation Plan AAO AA - defines accreditation criteria - describes V&V event(s) Step 2. Accreditation Request AAO AA - results of V&V event(s) - criteria assessment Figure 1. VV&A Steps M&S Concept Approved at ESR Develop Accreditation Plan (AAO) Submit Accreditation Plan (AAO) Plan Approved? No Yes V&V Results and Written Report Yes Add Testing No Submit Accreditation Request Request Approved? No M&S Discontinued or Terminated Yes M&S Accredited Figure 2. Accreditation Decision Process Diagram 8
Appendix A References AR 5-11 Management of Army Models and Simulations AR 70-1 Army Acquisition Policy AR 73-1 Test and Evaluation Policy ATEC Regulation 37-3 Unfinanced Requirement (UFR) Process ATEC Regulation 73-1 System Test and Evaluation Policy DA Pam 5-11 Verification, Validation, and Accreditation of Army Models and Simulations DA Pam 73-1 Test and Evaluation in Support of Systems Acquisition Simulation and Modeling for Acquisition, Requirements and Training (SMART) Execution Plan, 6 November 2000 Memorandum, Director of Operational Test and Evaluation, subject: Modeling and Simulation, 4 June 2002 Memorandum, Deputy Secretary of Defense, subject: Defense Acquisition, 30 October 2002 9
Appendix B Sample Accreditation Plan Memorandum For ACAT I and DOD oversight programs, the signed accreditation plan is submitted with the accreditation plan memorandum to the ATEC CG for approval. Figure B-1 is a sample format. (Office Symbol) (Date) MEMORANDUM THRU Director, Test Technology Directorate, U.S. Army Test and Evaluation Command (CSTE-TT), 4501 Ford Avenue, Alexandria, VA 22302-1458 MEMORANDUM FOR Commander, U.S. Army Test and Evaluation Command (CSTE), 4501 Ford Avenue, Alexandria, VA 22302-1458 SUBJECT: Accreditation Plan for the (name o f M&S) in Support of the (name of test) 1. Request approval of the enclosed Accreditation Plan for the (name of M&S) in support of (test name, test type, & test dates). 2. The (name of directorate) will use the (name of M&S) to (describe requirement for use of the M&S). Acceptability criteria have been established for this specific use of the M&S and are provided (either list on memorandum or make reference to location in plan). The (name of directorate) will conduct an accreditation event (provide dates or time frame) to validate this requirement. 3. The point of contact at (name of subordinate command/activity) is (name/title/phone number/email address). Encl Accreditation Plan Signature Block of Requesting Director/Commander Figure B-1. Sample Accreditation Plan Memorandum 10
Appendix C Sample Accreditation Request Memorandum The accreditation request consists of the accreditation report and the accreditation request memorandum. Figure C-1 is a sample format. (Office Symbol) (Date) MEMORANDUM THRU Director, Test Technology Directorate, U.S. Army Test and Evaluation Command (CSTE-TT), 4501 Ford Avenue, Alexandria, VA 22302-1458 MEMORANDUM FOR Commander, U.S. Army Test and Evaluation Command (CSTE), 4501 Ford Avenue, Alexandria, VA 22302-1458 SUBJECT: Accreditation Request for the (name of M&S) in Support of the (name of test) 1. Request approval of the enclosed Accreditation Report for the (name of M&S) in support of (test name, test type, & test dates). 2. The enclosed report publishes the findings o f the Verification and Validation (V&V) of the (name of M&S) completed on (date). The system has no limitations on the specific use of the M&S for this test or in the subsequent evaluation. All criteria set forth in the accreditation plan were met during V&V. (NOTE: If there are limitations and/or criteria were not met, then briefly describe). 3. The point of contact at (name of subordinate command/activity) is (name/title/phone number/email address). Encl Signature Block of Requesting Accreditation Report Director/Commander Figure C-1. Sample Accreditation Request Memorandum 11
Glossary Section I Acronyms AA Accreditation Authority AAO Accreditation Action Officer ACAT Acquisition Category AEC Army Evaluation Center AMSAA Army Materiel Systems Analysis Activity AST ATEC System Team ATEC Army Test and Evaluation Command CIPR Concept In Process Review DOD Department of Defense DOTSP Doctrinal and Organizational Training Support Packages DTC Developmental Test Command EDP Event Design Plan ESR Early Strategy Review HQ Headquarters 12
M&S Models and Simulations OTC Operational Test Command P(k) Probability of Kill SCA Subordinate Command Activity SEP System Evaluation Plan SER System Evaluation Report SSP Simulation Support Plan T&E Test and Evaluation T&E WIPT Test and Evaluation Working-level Integrated Product Team TD Technical Director TDY Temporary Duty TEMP Test and Evaluation Master Plan TTD Test Technology Directorate TTSP Threat Test Support Packages V&V Verification and Validation 13
VV&A Verification, Validation, and Accreditation Section II Terms Accreditation The official certification that a model, simulation, or federation of models and simulations is acceptable for use for a specific purpose. M&S Sponsor Owner or operator of the subject model or simulation. Model A physical, mathematical, or otherwise logical representation of a system, entity, phenomenon, or process. Modeling and Simulation The development and use of live, virtual, and constructive models including simulators, stimulators, emulators, and prototypes to investigate, understand, or provide experiential stimulus to either (1) conceptual systems that do not exist or (2) real life systems that cannot accept experimentation or observation because of resource, range, security, or safety limitations. Simulation A method for implementing a model over time. Validation The process of determining the degree to which M&S is an accurate representation of the real world from the perspective of the intended use of the M&S. Verification The process of determining that the M&S implementation accurately represents the developer s conceptual description and specifications. 14