Institutionalizing a Culture of Statistical Thinking in DoD Testing
|
|
- Coleen Dalton
- 6 years ago
- Views:
Transcription
1 Institutionalizing a Culture of Statistical Thinking in DoD Testing Dr. Catherine Warner Science Advisor Statistical Engineering Leadership Webinar 25 September 2017
2 Outline Overview of DoD Testing Improving Operational Testing Statistical Analysis Methods for Improving Mission Characterization Continuing the Path Forward Bayesian Methods for Maximizing Information Defensible Surveys Capturing Human Interactions Improving Modeling and Simulation Looking to the Future 2
3 Goal of Operational Test: Evaluate Operational Effectiveness and Suitability Operational Environment Representative Users Real Threats Conducting Missions 3
4 DoD Test Paradigm Contractor Testing Developmental Testing Operational Testing Test Timeline Tend to be requirements driven 4
5 Requirements documents are often missing important mission considerations
6 OT characterizes mission capability Contractor Testing Developmental Testing Operational Testing Test Timeline Poor Acoustic Propagation; High Ambient Noise, High Density Traffic Difficulty of Environment Virginia vs. Georgia (ASW-3) ARCI APB-03 OT 688I vs. Gotland ARCI APB-06 OT 688I vs. Todaro Favorable Acoustic Propagation; Low Traffic; Low Ambient Noise Virginia vs. Albany (ASW-2) Snorkeling Diesel (includes most older SSK threats) Fast/Noisy SSN Slow SSN or SSBN (SSN threat equivalent) Target Source Level (Decreasing) Gotland (SSK threat equivalent) Todaro (Quietest SSK threats) 6
7 By the early 1980s, Congress concerns were growing
8 Congress established DOT&E separate from the Services operational testing agencies Department of Defense Office of the Secretary of Defense Director, Operational Test and Evaluation (DOT&E)
9 Congress established DOT&E separate from the Services operational testing agencies Department of Defense Office of the Secretary of Defense Army Navy & Marines Air Force Director, Operational Test and Evaluation (DOT&E) Service Operational Testing Agencies
10 Congress established DOT&E separate from the Services operational testing agencies Congress Department of Defense Office of the Secretary of Defense Army Navy & Marines Air Force Director, Operational Test and Evaluation (DOT&E) Service Operational Testing Agencies
11 Operational testing provides critical information to warfighters about new systems Before warfighters lives and missions depend on them
12 Operational testing provide critical information to warfighters about new systems Before warfighters lives and missions depend on them Time to correct problems Time to restrict missions
13 Improving Operational Testing
14 Why did we need to improve test methods? Figure Figure from DOT&E from DOT&E EA-18G EA-18G BLRIP BLRIP Percent Success Percent Success
15 DOT&E Sets Policy and Guidance for Conducting Operational Testing The goal of the experiment. This should reflect evaluation of end-to-end mission effectiveness in an operationally realistic environment. Quantitative mission-oriented response variables for effectiveness and suitability. (These could be Key Performance Parameters but most likely there will be others.) Factors that affect those measures of effectiveness and suitability. Systematically, in a rigorous and structured way, develop a test plan that provides good breadth of coverage of those factors across the applicable levels of the factors, taking into account known information in order to concentrate on the factors of most interest. A method for strategically varying factors across both developmental and operational testing with respect to responses of interest. Statistical measures of merit (power and confidence) on the relevant response variables for which it makes sense. These statistical measures are important to understanding "how much testing is enough?" and can be evaluated by decision makers on a quantitative basis so they can trade off test resources for desired confidence in results.
16 Laying the foundations for statistical methods in T&E Research Consortium Offsite Meeting Charter Statistical Engineering with NASA AO Training, OTA Training
17 Puzzled??
18 Sharing lessons learned advanced our mutual understanding
19 Without a destination, any path will do Institutionalizing Statistical Thinking in Test and Evaluation National Research Council Study Design of Experiments endorsed as a sound methodology for OT&E OTA MOA on DOE DOT&E Initiatives Guidance on DOE in TEMPs DOT&E Policy Issued OTA Test Design Processes Updated DOT&E Science Advisor Established Test Science Roadmap effort DOT&E/ TRMC funded Science of Test Research Consortium DOT&E TEMP Guide Published DASD (DT&E) STAT Implement ation Plan STAT COE DOT&E Roadmap Report Two Additional DOT&E Guidance memos on Application of DOE to OT&E Survey Best Practices Memo Cybersecurity Procedures Additional Survey and cyber work Modeling and simulation validation guidance Cyber priorities Updated TEMP Guidance M&S Guidance
20 Lessons Learned from Implementing DOE Strong leadership Communicate, communicate, communicate Find partners Compromise Be open to new ideas Create quick successes and highlight them Support the workforce 20
21 Statistical Analysis Methods for Improving Mission Characterization
22 Statistical analyses maximize information 8 Non-DOE approach - data in bins segregated DOE method - data in all bins used to construct model 7 6 Response Variable A B C D Rollup
23 Statistical models capture important interactions Apache FOT&E 80% confidence intervals shown
24 Continuing the Path Forward
25 Bayesian methods provide flexibility in combining information Stryker Family of Vehicles Reliability Miles Between System Abort Traditional Analysis Bayesian Combining Information 0
26 Sometimes mission outcome is subjective Survey regarding improved situational awareness Strongly agree Agree Slightly agree Slightly disagree Disagree Strongly disagree
27 Guidance highlighted key concepts for improving surveys Surveys are appropriate for quantitatively measuring operator and maintainer thoughts and opinions Have an administration plan for surveys and only use surveys when appropriate Use the right survey Empirically vetted surveys should be used to measure known constructs (e.g., workload, usability, trust) Custom surveys should be used appropriately Follow best practices for writing questions Always pre-test Avoid asking questions without a clear analysis plan Use interviews and focus groups for problem identification and general context Do not develop lengthy exhaustive surveys about every problem that could occur
28 Live-Virtual Constructive simulations can help us learn more but it needs better validation Range of threat types Operational Space Threat lethality
29 Live-Virtual Constructive simulations can help us learn more but it needs better validation Range of threat types Operational Space Modeling and simulation Threat lethality
30 Live-Virtual Constructive simulations can help us learn more but it needs better validation Range of threat types Operational Space Modeling and simulation live testing Threat lethality
31 We continue to increase the statistical defensibility of DoD Test and Evaluation National Research Council Study Design of Experiments endorsed as a sound methodology for OT&E OTA MOA on DOE DOT&E Initiatives Guidance on DOE in TEMPs DOT&E Policy Issued OTA Test Design Processes Updated DOT&E Science Advisor Established Test Science Roadmap effort DOT&E/ TRMC funded Science of Test Research Consortium DOT&E TEMP Guide Published DASD (DT&E) STAT Implement ation Plan STAT COE DOT&E Roadmap Report Two Additional DOT&E Guidance memos on Application of DOE to OT&E Survey Best Practices Memo Cybersecurity Procedures Additional Survey and cyber work Modeling and simulation validation guidance Cyber priorities Updated TEMP Guidance M&S Guidance
32 Future Test Challenges
33 We need to think carefully about the challenges we face in the future Space Cyber Autonomy Big data Workforce
UNCLASSIFIED R-1 ITEM NOMENCLATURE
Exhibit R-2, RDT&E Budget Item Justification: PB 2014 Office of Secretary Of Defense DATE: April 2013 0400: Research, Development, Test &, Defense-Wide COST ($ in Millions) All Prior FY 2014 Years FY 2012
More informationUNCLASSIFIED. UNCLASSIFIED Office of Secretary Of Defense Page 1 of 8 R-1 Line #163
Exhibit R-2, RDT&E Budget Item Justification: PB 2015 Office of Secretary Of Defense Date: March 2014 0400: Research, Development, Test &, Defense-Wide / BA 6: RDT&E Management Support COST ($ in Millions)
More informationDepartment of Defense DIRECTIVE
Department of Defense DIRECTIVE NUMBER 5141.02 February 2, 2009 DA&M SUBJECT: Director of Operational Test and Evaluation (DOT&E) References: See Enclosure 1 1. PURPOSE. This Directive: a. Reissues DoD
More informationI n t r o d u c t i o n
The President and the Congress have given me the opportunity to serve as Director, Operational Test and Evaluation for these last two and a half years. I have been honored and humbled to serve in this
More informationTest and Evaluation of Highly Complex Systems
Guest Editorial ITEA Journal 2009; 30: 3 6 Copyright 2009 by the International Test and Evaluation Association Test and Evaluation of Highly Complex Systems James J. Streilein, Ph.D. U.S. Army Test and
More informationTest and Evaluation (T&E) is essential to successful system
Test and Evaluation Myths and Misconceptions Steve Hutchison, Ph.D. Test and Evaluation (T&E) is essential to successful system acquisition. For the last 43 years, the Office of the Secretary of Defense
More informationI n t r o d u c t i o n
I was confirmed by the Senate on September 21, 2009, as the Director, Operational Test and Evaluation, and sworn in on September 23. It is a privilege to serve in this position. I will work to assure that
More informationTest and Evaluation and the ABCs: It s All about Speed
Invited Article ITEA Journal 2009; 30: 7 10 Copyright 2009 by the International Test and Evaluation Association Test and Evaluation and the ABCs: It s All about Speed Steven J. Hutchison, Ph.D. Defense
More informationOSD RDT&E BUDGET ITEM JUSTIFICATION (R2 Exhibit)
Exhibit R-2 0605804D8Z OSD RDT&E BUDGET ITEM JUSTIFICATION (R2 Exhibit) COST ($ in Millions) FY 2007 FY 2008 FY 2009 FY 2010 FY 2011 FY 2012 FY 2013 Total Program Element (PE) Cost 9.155 18.550 20.396
More informationTesting in a Joint Environment. Janet Garber Director Test and Evaluation Office Office of the Deputy Under Secretary of the Army
Testing in a Joint Environment Value Added and Considerations Janet Garber Director Test and Evaluation Office Office of the Deputy Under Secretary of the Army June 2008 UNCLASSIFIED 1 Why do we test?
More informationMission Based T&E Progress
U.S. Army Evaluation Center Mission Based T&E Progress Christopher Wilcox Deputy/Technical Director Fires Evaluation Directorate, US AEC 15 Mar 11 2 Purpose and Agenda Purpose: To review the status of
More informationUNCLASSIFIED OSD RDT&E BUDGET ITEM JUSTIFICATION (R2 Exhibit)
Budget Item Justification Exhibit R-2 0605804D8Z OSD RDT&E BUDGET ITEM JUSTIFICATION (R2 Exhibit) Cost ($ in Millions) FY 2008 FY 2009 FY 2010 FY 2011 FY 2012 FY 2013 Actual Total Program Element (PE)
More informationDepartment of Defense Fiscal Year (FY) 2013 President's Budget Submission
Department of Defense Fiscal Year (FY) 2013 President's Budget Submission February 2012 Operational Test and Evaluation, Defense Justification Book Operational Test and Evaluation, Defense OT&E THIS PAGE
More informationDevelopmental Test & Evaluation OUSD(AT&L)/DDR&E
Developmental Test & Evaluation OUSD(AT&L)/DDR&E Chris DiPetto 12 th Annual NDIA Systems Engineering Conference Agenda DT&E Title 10 USC overview Organization DDR&E imperatives What Title 10 means for
More informationCOMPLIANCE WITH THIS PUBLICATION IS MANDATORY
BY ORDER OF THE SECRETARY OF THE AIR FORCE AIR FORCE POLICY DIRECTIVE 99-1 3 JUNE 2014 Test and Evaluation TEST AND EVALUATION COMPLIANCE WITH THIS PUBLICATION IS MANDATORY ACCESSIBILITY: Publications
More informationUNCLASSIFIED. UNCLASSIFIED R-1 Line Item No. 4 Page 1 of 6
Exhibit R-2, RDT&E Project Justification February 2007 OPERATIONAL TEST AND EVALUATION, DEFENSE (0460) BUDGET ACTIVITY SIX LIVE FIRE TEST AND EVALUATION (LFT&E) PROGRAM ELEMENT (PE) 0605131OTE Cost ($
More informationDevelopmental Test and Evaluation Is Back
Guest Editorial ITEA Journal 2010; 31: 309 312 Developmental Test and Evaluation Is Back Edward R. Greer Director, Developmental Test and Evaluation, Washington, D.C. W ith the Weapon Systems Acquisition
More informationFY 2016 Annual Report
FY 2016 Annual Report I have served as the Director, Operational Test and Evaluation at the request of the President and Congress since September 2009. It has been an honor and a privilege to serve in
More informationInside the Beltway ITEA Journal 2008; 29: Copyright 2008 by the International Test and Evaluation Association
Inside the Beltway ITEA Journal 2008; 29: 121 124 Copyright 2008 by the International Test and Evaluation Association Enhancing Operational Realism in Test & Evaluation Ernest Seglie, Ph.D. Office of the
More informationJOINT TEST AND EVALUATION METHODOLOGY (JTEM) PROGRAM MANAGER S HANDBOOK FOR TESTING IN A JOINT ENVIRONMENT
JOINT TEST AND EVALUATION METHODOLOGY (JTEM) PROGRAM MANAGER S HANDBOOK FOR TESTING IN A JOINT ENVIRONMENT Approved By: Maximo Lorenzo Joint Test Director JTEM JT&E APRIL 17, 2009 DISTRIBUTION STATEMENT
More informationFY 2015 Annual Report
FY 2015 Annual Report American national security is based on preparedness. By ensuring our armed forces ability to deal with any extant challenge, we disincentivize threats to our interests and mitigate
More informationAMRDEC. Core Technical Competencies (CTC)
AMRDEC Core Technical Competencies (CTC) AMRDEC PAMPHLET 10-01 15 May 2015 The Aviation and Missile Research Development and Engineering Center The U. S. Army Aviation and Missile Research Development
More informationDoDI ,Operation of the Defense Acquisition System Change 1 & 2
DoDI 5000.02,Operation of the Defense Acquisition System Change 1 & 2 26 January & 2 February 2017 (Key Changes from DoDI 5000.02, 7 Jan 2015) Presented By: T.R. Randy Pilling Center Director Acquisition
More informationFY 2010 Annual Report
FY 2010 Annual Report In my first report to you last year, I discussed four initiatives I was undertaking as Director of Operational Test and Evaluation. In this Introduction, I describe the progress I
More informationUNCLASSIFIED FY 2016 OCO. FY 2016 Base
Exhibit R-2, RDT&E Budget Item Justification: PB 2016 Office of the Secretary Of Defense Date: February 2015 0400: Research, Development, Test & Evaluation, Defense-Wide / BA 6: RDT&E Management Support
More informationDefense Science Board Task Force Developmental Test and Evaluation Study Results
Invited Article ITEA Journal 2008; 29: 215 221 Copyright 2008 by the International Test and Evaluation Association Defense Science Board Task Force Developmental Test and Evaluation Study Results Pete
More informationSubj: DEPARTMENT OF THE NAVY CYBERSECURITY/INFORMATION ASSURANCE WORKFORCE MANAGEMENT, OVERSIGHT, AND COMPLIANCE
DEPARTMENT OF THE NAVY OFFICE OF THE SECRETARY 1000 NAVY PENTAGON WASHINGTON DC 20350 1000 SECNAVINST 5239.20 DON CIO SECNAV INSTRUCTION 5239.20 From: Secretary of the Navy Subj: DEPARTMENT OF THE NAVY
More informationGUARDING THE INTENT OF THE REQUIREMENT. Stephen J Scukanec. Eric N Kaplan
GUARDING THE INTENT OF THE REQUIREMENT 13th Annual Systems Engineering Conference Hyatt Regency Mission Bay San Diego October 25-28, 2010 Stephen J Scukanec Flight Test and Evaluation Aerospace Systems
More informationLongbow Apache and GMLRS had Nunn McCurdy but did not have any delays
1 2 3 4 The canceled programs are: EFV VH 71 SADARM Comanche Armed Recon Helo EIBCT DDG 1000 had MS B rescinded (47 programs had performance in DT problems, 35 programs had performance in OT problems,
More informationCybersecurity TEMP Body Example
ybersecurity TEMP Body Example 1.3. System Description (...) A unit equipped with TGVS performs armed reconnaissance missions and provides operators with sensors and weapons to observe and engage enemies.
More informationUNCLASSIFIED. Cost To Complete Total Program Element JA6: Joint Air-To-Ground Missile (JAGM)
Exhibit R-2, RDT&E Budget Item Justification: PB 2016 Army : February 2015 2040: Research, Development, Test & Evaluation, Army / BA 5: System Development & Demonstration (SDD) COST ($ in Millions) Years
More informationDEPARTMENT OF DEFENSE Developmental Test and Evaluation FY 2013 Annual Report
DEPARTMENT OF DEFENSE Developmental Test and Evaluation FY 2013 Annual Report MARCH 2014 The estimated cost of this report for the Department of Defense is approximately $521,000 in Fiscal Years 2013 2014.
More informationMission-Based Test & Evaluation Strategy: Creating Linkages between Technology Development and Mission Capability
U.S. Army Research, Development and Engineering Command Mission-Based Test & Evaluation Strategy: Creating Linkages between Technology Development and Mission Capability NDIA Systems Engineering Conference
More informationFiscal Year 2009 National Defense Authorization Act, Section 322. Study of Future DoD Depot Capabilities
Fiscal Year 2009 National Defense Authorization Act, Section 322 Study of Future DoD Depot Capabilities Update for the DoD Maintenance Symposium Monday October 26, 2009 Phoenix, Arizona Goals For Today
More informationUNCLASSIFIED FY 2016 OCO. FY 2016 Base
Exhibit R-2, RDT&E Budget Item Justification: PB 2016 Air Force Date: February 2015 3600: Research, Development, Test & Evaluation, Air Force / BA 6: RDT&E Management Support COST ($ in Millions) Prior
More informationJoint Test & Evaluation Program
Joint Test & Evaluation Program Program Overview Mr. Mike Crisp Deputy Director Air Warfare DOT&E March 22, 2005 Mr. Jim Thompson Joint Test and Evaluation Program Manager 1 What is the JT&E Program? DOT&E
More informationReport to Congress on Recommendations and Actions Taken to Advance the Role of the Chief of Naval Operations in the Development of Requirements, Acquisition Processes and Associated Budget Practices. The
More informationPrepared for Milestone A Decision
Test and Evaluation Master Plan For the Self-Propelled Artillery Weapon (SPAW) Prepared for Milestone A Decision Approval Authority: ATEC, TACOM, DASD(DT&E), DOT&E Milestone Decision Authority: US Army
More informationUNCLASSIFIED. R-1 ITEM NOMENCLATURE PE D8Z: Central Test and Evaluation Investment Program (CTEIP) FY 2013 OCO
COST ($ in Millions) FY 2011 FY 2012 FY 2013 Base FY 2013 OCO FY 2013 Total FY 2014 FY 2015 FY 2016 FY 2017 Cost To Complete Total Cost Total Program Element 157.971 156.297 144.109-144.109 140.097 141.038
More informationCOMPLIANCE WITH THIS PUBLICATION IS MANDATORY
BY ORDER OF THE SECRETARY OF THE AIR FORCE AIR FORCE POLICY DIRECTIVE 90-16 31 AUGUST 2011 Special Management STUDIES AND ANALYSES, ASSESSMENTS AND LESSONS LEARNED COMPLIANCE WITH THIS PUBLICATION IS MANDATORY
More informationUNCLASSIFIED. UNCLASSIFIED Air Force Page 1 of 9 R-1 Line #44
Exhibit R-2, RDT&E Budget Item Justification: PB 2015 Air Force Date: March 2014 3600: Research, Development, Test & Evaluation, Air Force / BA 4: Advanced Component Development & Prototypes (ACD&P) COST
More informationOpera onal Test & Evalua on Manual
MARINE CORPS OPERATIONAL TEST AND EVALUATION ACTIVITY Opera onal Test & Evalua on Manual 4th Edi on, Rev 1 THIS PAGE INTENTIONALLY LEFT BLANK MARINE CORPS OPERATIONAL TEST AND EVALUATION ACTIVITY Operational
More informationAcquisition. Air Force Procurement of 60K Tunner Cargo Loader Contractor Logistics Support (D ) March 3, 2006
March 3, 2006 Acquisition Air Force Procurement of 60K Tunner Cargo Loader Contractor Logistics Support (D-2006-059) Department of Defense Office of Inspector General Quality Integrity Accountability Report
More informationThe Role of T&E in the Systems Engineering Process Keynote Address
The Role of T&E in the Systems Engineering Process Keynote Address August 17, 2004 Glenn F. Lamartin Director, Defense Systems Top Priorities 1. 1. Successfully Successfully Pursue Pursue the the Global
More informationNATIONAL AIRSPACE SYSTEM (NAS)
NATIONAL AIRSPACE SYSTEM (NAS) Air Force/FAA ACAT IC Program Prime Contractor Air Traffic Control and Landing System Raytheon Corp. (Radar/Automation) Total Number of Systems: 92 sites Denro (Voice Switches)
More informationOUTSOURCING TRENDS THAT WILL HELP YOU PREPARE FOR 2017
OUTSOURCING TRENDS THAT WILL HELP YOU PREPARE FOR 2017 January 18 2017 Introduction Varun Chandrasekaran Head of Managed Services The Red Flag Group About The Red Flag Group Christopher Sindik Director
More informationFY 2017 Annual Report on Cost Assessment Activities. February 2018
A FY 2017 Annual Report on Cost Assessment Activities February 2018 This page intentionally left blank. FY 2017 Annual Report on Cost Assessment Activities Director, Cost Assessment and Program Evaluation
More informationDepartment of Defense Investment Review Board and Investment Management Process for Defense Business Systems
Department of Defense Investment Review Board and Investment Management Process for Defense Business Systems Report to Congress March 2012 Pursuant to Section 901 of the National Defense Authorization
More informationThe current Army operating concept is to Win in a complex
Army Expansibility Mobilization: The State of the Field Ken S. Gilliam and Barrett K. Parker ABSTRACT: This article provides an overview of key definitions and themes related to mobilization, especially
More informationInspector General FOR OFFICIAL USE ONLY
Report No. DODIG-2016-107 Inspector General U.S. Department of Defense JULY 5, 2016 Advanced Arresting Gear Program Exceeded Cost and Schedule Baselines INTEGRITY EFFICIENCY ACCOUNTABILITY EXCELLENCE The
More informationDepartment of Defense Developmental Test and Evaluation and Systems Engineering FY 2011 Annual Report. Washington, DC: DASD(DT&E) and DASD(SE), 2012.
Department of Defense Developmental Test and Evaluation and Systems Engineering FY 2011 Annual Report. Washington, DC: DASD(DT&E) and DASD(SE), 2012. Deputy Assistant Secretary of Defense for Developmental
More informationTest and Evaluation Resources
Test and Evaluation Resources Public law requires DOT&E to assess the adequacy of test and evaluation resources and facilities for operational and live fire testing. DOT&E monitors and reviews DOD- and
More informationTest and Evaluation Strategies for Network-Enabled Systems
ITEA Journal 2009; 30: 111 116 Copyright 2009 by the International Test and Evaluation Association Test and Evaluation Strategies for Network-Enabled Systems Stephen F. Conley U.S. Army Evaluation Center,
More informationDepartment of Defense INSTRUCTION
Department of Defense INSTRUCTION NUMBER 8330.01 May 21, 2014 Incorporating Change 1, December 18, 2017 DoD CIO SUBJECT: Interoperability of Information Technology (IT), Including National Security Systems
More informationCode 85 Weapons Analysis Facility (WAF) Technical Engineering Services Pre-Solicitation Conference
Code 85 Weapons Analysis Facility (WAF) Technical Engineering Services Pre-Solicitation Conference NUWC Division Newport Undersea Collaboration & Technology Outreach Center (UCTOC) June 12, 2014 Agenda
More informationARMY TACTICAL MISSILE SYSTEM (ATACMS) BLOCK II
ARMY TACTICAL MISSILE SYSTEM (ATACMS) BLOCK II Army ACAT ID Program Total Number of BATs: (3,487 BAT + 8,478 P3I BAT) Total Number of Missiles: Total Program Cost (TY$): Average Unit Cost (TY$): Full-rate
More informationStatement of Vice Admiral Albert H. Konetzni, Jr. USN (Retired) Before the Projection Forces Subcommittee of the House Armed Services Committee
Statement of Vice Admiral Albert H. Konetzni, Jr. USN (Retired) Before the Projection Forces Subcommittee of the House Armed Services Committee Chairman Bartlett and members of the committee, thank you
More informationCOMPLIANCE WITH THIS INSTRUCTION IS MANDATORY (AETC)
BY ORDER OF THE SECRETARY OF THE AIR FORCE AIR FORCE INSTRUCTION 99-103 16 OCTOBER 2013 AIR EDUCATION AND TRAINING COMMAND Supplement 6 APRIL 2015 Test and Evaluation CAPABILITIES-BASED TEST AND EVALUATION
More informationDepartment of Defense DIRECTIVE
Department of Defense DIRECTIVE NUMBER 3200.11 May 1, 2002 Certified Current as of December 1, 2003 SUBJECT: Major Range and Test Facility Base (MRTFB) DOT&E References: (a) DoD Directive 3200.11, "Major
More informationHQMC 7 Jul 00 E R R A T U M. MCO dtd 9 Jun 00 MARINE CORPS POLICY ON DEPOT MAINTENANCE CORE CAPABILITIES
HQMC 7 Jul 00 E R R A T U M TO MCO 4000.56 dtd MARINE CORPS POLICY ON DEPOT MAINTENANCE CORE CAPABILITIES 1. Please insert enclosure (1) pages 1 thru 7, pages were inadvertently left out during the printing
More informationDepartment of Defense DIRECTIVE. DoD Modeling and Simulation (M&S) Management
Department of Defense DIRECTIVE NUMBER 5000.59 August 8, 2007 USD(AT&L) SUBJECT: DoD Modeling and Simulation (M&S) Management References: (a) DoD Directive 5000.59, DoD Modeling and Simulation (M&S) Management,
More informationMILITARY STRATEGIC AND TACTICAL RELAY (MILSTAR) SATELLITE SYSTEM
MILITARY STRATEGIC AND TACTICAL RELAY (MILSTAR) SATELLITE SYSTEM Air Force ACAT ID Program Prime Contractor Total Number of Systems: 6 satellites Lockheed Martin Total Program Cost (TY$): N/A Average Unit
More informationGerry Christeson Test Resource Management Center 20 October 2010
Building Next Generation Range Capabilities Central Test and Evaluation Investment Program (CTEIP) Gerry Christeson Test Resource Management Center 20 October 2010 1 Test Resource Management Center (TRMC)
More informationTest and Evaluation Policy
Army Regulation 73 1 Test and Evaluation Test and Evaluation Policy UNCLASSIFIED Headquarters Department of the Army Washington, DC 16 November 2016 SUMMARY of CHANGE AR 73 1 Test and Evaluation Policy
More informationAIRBORNE LASER (ABL)
AIRBORNE LASER (ABL) Air Force ACAT ID Program Prime Contractor Total Number of Systems: 7 aircraft Boeing Total Program Cost (TY$): $6335M Average Unit Cost (TY$): $528M Full-rate production: FY06 SYSTEM
More informationOverview of the Chemical and Biological Defense Program Requirements Process
Overview of the Chemical and Biological Defense Program Requirements Process 14 March 2012 Director, Joint Requirements Office for Chemical, Biological, Radiological, and Nuclear Defense J-8, The Joint
More informationJAVELIN ANTITANK MISSILE
JAVELIN ANTITANK MISSILE Army ACAT ID Program Total Number of Systems: Total Program Cost (TY$): Average CLU Cost (TY$): Average Missile Cost (TY$): Full-rate production: 4,348 CLUs 28,453 missiles $3618M
More informationDevelopment Planning Working Group Update
Development Planning Working Group Update Ms. Aileen Sedmak Office of the Deputy Assistant Secretary of Defense for Systems Engineering 16th Annual NDIA Systems Engineering Conference Arlington, VA October
More informationDoD M-4, August 1988
1 2 FOREWORD TABLE OF CONTENTS Page FOREWORD 2 TABLE OF CONTENTS 3 CHAPTER 1 - OVERVIEW OF THE JOINT TEST AND EVALUATION PROGRAM 4 C1.1. PROGRAM DESCRIPTION 4 C1.2. NOMINATION AND SELECTION PROCESS 5 CHAPTER
More information(111) VerDate Sep :55 Jun 27, 2017 Jkt PO Frm Fmt 6601 Sfmt 6601 E:\HR\OC\A910.XXX A910
TITLE III PROCUREMENT The fiscal year 2018 Department of Defense procurement budget request totals $113,906,877,000. The Committee recommendation provides $132,501,445,000 for the procurement accounts.
More informationTRAINEE CLINICAL PSYCHOLOGIST GENERIC JOB DESCRIPTION
TRAINEE CLINICAL PSYCHOLOGIST GENERIC JOB DESCRIPTION This is a generic job description provided as a guide to applicants for clinical psychology training. Actual Trainee Clinical Psychologist job descriptions
More informationCybersecurity FY16 CYBERSECURITY. Cybersecurity 441
Cybersecurity SUMMARY DOT&E provides cybersecurity evaluations of DOD acquisition programs as part of the programs operational test and evaluation. In addition, Congress directed DOT&E to perform cybersecurity
More informationSubj: NAVY ENTERPRISE TEST AND EVALUATION BOARD OF DIRECTORS
D E PAR TME NT OF THE N A VY OFFICE OF T HE SECRET ARY 1000 NAVY PENT AGON WASHINGT ON D C 20350-1000 SECNAVINST 3900.44 ASN(RD&A) SECNAV INSTRUCTION 3900.44 From: Secretary of the Navy Subj: NAVY ENTERPRISE
More informationUnited States Government Accountability Office GAO. Report to Congressional Committees
GAO United States Government Accountability Office Report to Congressional Committees February 2005 MILITARY PERSONNEL DOD Needs to Conduct a Data- Driven Analysis of Active Military Personnel Levels Required
More informationSYSTEM DESCRIPTION & CONTRIBUTION TO JOINT VISION
F-22 RAPTOR (ATF) Air Force ACAT ID Program Prime Contractor Total Number of Systems: 339 Lockheed Martin, Boeing, Pratt &Whitney Total Program Cost (TY$): $62.5B Average Flyaway Cost (TY$): $97.9M Full-rate
More informationMerging Operational Realism with DOE Methods in Operational Testing NDIA Presentation on 13 March 2012
U.S. Merging Operational Realism with DOE Methods in Operational Testing NDIA Presentation on 13 March 2012 Nancy Dunn, DA Civilian Chief, Editorial & Statistics/DOE Division, US nancy.dunn@us.army.mil
More informationMaking the Case for Distributed Testing
ITEA Journal 2010; 31: 347 354 Making the Case for Distributed Testing Bernard Chip Ferguson Test Resource Management Center, Joint Mission Environment Test Capability (JMETC) Program, Arlington, Virginia
More informationThis is definitely another document that needs to have lots of HSI language in it!
1 The Capability Production Document (or CPD) is one of the most important things to come out of the Engineering and Manufacturing Development phase. It defines an increment of militarily useful, logistically
More informationExperimenting into the future Mr Ed Gough Deputy Commander Naval Meteorology and Oceanography Command
Naval Oceanography Experimenting into the future Mr Ed Gough Deputy Commander Naval Meteorology and Oceanography Command After Before Meteorology & Oceanography Strategic Plan Meteorology & Oceanography
More informationUNCLASSIFIED UNCLASSIFIED
: February 216 Exhibit R2, RDT&E Budget Item Justification: PB 217 2: Research, Development, Test & Evaluation, / BA 5: System Development & Demonstration (SDD) COST ($ in Millions) FY 215 FY 216 R1 Program
More informationUNCLASSIFIED R-1 ITEM NOMENCLATURE FY 2013 OCO
Exhibit R-2, RDT&E Budget Item Justification: PB 2013 Defense Information Systems Agency DATE: February 2012 COST ($ in Millions) FY 2011 FY 2012 Total FY 2014 FY 2015 FY 2016 FY 2017 To Complete Total
More information2014 MASTER PROJECT LIST
Promoting Integrated Care for Dual Eligibles (PRIDE) This project addressed a set of organizational challenges that high performing plans must resolve in order to scale up to serve larger numbers of dual
More informationMANAGING LARGE DISTRIBUTED DATA SETS FOR TESTING IN A JOINT ENVIRONMENT
MANAGING LARGE DISTRIBUTED DATA SETS FOR TESTING IN A JOINT ENVIRONMENT Thomas Bock thomas.bock@jte.osd.mil Tonya Easley tonya.easley@jte.osd.mil John Hoot Gibson john.gibson@jte.osd.mil Joint Test and
More informationGlossary. Approval for Full Production (AFP). The decision for full production of a system. Normally occurs at the final MS-C.
Glossary Acquisition Category (ACAT). Categories established to facilitate decentralized decisionmaking and execution and compliance with statutorily imposed requirements. The categories determine the
More informationMILITARY STRATEGIC AND TACTICAL RELAY (MILSTAR) SATELLITE SYSTEM
MILITARY STRATEGIC AND TACTICAL RELAY (MILSTAR) SATELLITE SYSTEM Air Force ACAT ID Program Prime Contractor Total Number of Satellites: 6 Lockheed Martin Total Program Cost (TY$): N/A Average Unit Cost
More informationSTATEMENT OF THE HONORABLE PETER B. TEETS, UNDERSECRETARY OF THE AIR FORCE, SPACE
STATEMENT OF THE HONORABLE PETER B. TEETS, UNDERSECRETARY OF THE AIR FORCE, SPACE BEFORE THE HOUSE ARMED SERVICES COMMITTEE STRATEGIC FORCES SUBCOMMITTEE UNITED STATES HOUSE OF REPRESENTATIVES ON JULY
More informationUS Special Operations Command
US Special Operations Command Operational Test & Evaluation Overview HQ USSOCOM LTC Kevin Vanyo 16 March 2011 The overall classification of this briefing is: Agenda OT&E Authority Mission and Tenants Responsibilities
More informationMCO D C Sep 2008
C 19 MARINE CORPS ORDER 3902.1D From: Commandant of the Marine Corps To: Distribution List Subj: MARINE CORPS STUDIES SYSTEM Ref: (a) SECNAVINST 5223.1C (b) SECNAV M-5214.1 Encl: (1) The Marine Corps Studies
More information17 th ITEA Engineering Workshop: System-of-Systems in a 3rd Offset Environment: Way Forward
17 th ITEA Engineering Workshop: System-of-Systems in a 3rd Offset Environment: Way Forward Mr. Paul D. Mann (Acting) Principal Deputy Director Test Resource Management Center January 26, 2017 1 2 TRMC
More informationMiddle Tier Acquisition and Other Rapid Acquisition Pathways
Middle Tier Acquisition and Other Rapid Acquisition Pathways Pete Modigliani Su Chang Dan Ward Contact us at accelerate@mitre.org Approved for public release. Distribution unlimited 17-3828-2. 2 Purpose
More informationDepartment of Defense INSTRUCTION. Non-Lethal Weapons (NLW) Human Effects Characterization
Department of Defense INSTRUCTION NUMBER 3200.19 May 17, 2012 Incorporating Change 1, September 13, 2017 USD(AT&L) SUBJECT: Non-Lethal Weapons (NLW) Human Effects Characterization References: See Enclosure
More informationDEPARTMENT OF DEFENSE Developmental Test and Evaluation FY 2016 Annual Report
DEPARTMENT OF DEFENSE Developmental Test and Evaluation FY 2016 Annual Report MARCH 2017 Dr. J. Brian Hall Acting Deputy Assistant Secretary of Defense Developmental Test and Evaluation The estimated cost
More informationDEPARTMENT OF THE NAVY COMMANDER OPERATIONAL TEST AND EVALUATION FORCE 7970 DIVEN STREET NORFOLK, VIRGINIA
DEPARTMENT OF THE NAVY COMMANDER OPERATIONAL TEST AND EVALUATION FORCE 7970 DIVEN STREET NORFOLK, VIRGINIA 23505-1498 COMOPTEVFOR INSTRUCTION 3980.2H Subj: OPERATIONAL TEST DIRECTOR'S MANUAL Encl: (1)
More informationDepartment of Defense INSTRUCTION
Department of Defense INSTRUCTION NUMBER 8260.04 December 18, 2009 USD(P&R) SUBJECT: Military Health System (MHS) Support to DoD Strategic Analysis References: (a) DoD Directive 5124.02, Under Secretary
More informationNew DoD Approaches on the Cyber Survivability of Weapon Systems
New DoD Approaches on the Cyber Survivability of Weapon Systems Colonel Dean Data Clothier Chief, Cyberspace Division Joint Staff/J-6 CSE is the Critical Foundation for Ensuring Cyber Survivability is
More informationTask Force Innovation Working Groups
Task Force Innovation Working Groups Emerging Operational Capabilities Adaptive Workforce Information EMERGING OPERATIONAL CAPABILITIES (EOC) WORKING GROUP VISION Accelerate Delivery of Emerging Operational
More informationWe Produce the Future. Air Force Doctrine
We Produce the Future Air Force Doctrine The Role of Doctrine At the very heart of warfare lies doctrine. It represents the central beliefs for waging war in order to achieve victory. Doctrine is of the
More information1.0 Executive Summary
1.0 Executive Summary On 9 October 2007, the Chief of Staff of the Air Force (CSAF) appointed Major General Polly A. Peyer to chair an Air Force blue ribbon review (BRR) of nuclear weapons policies and
More informationSUBJECT: Army Directive (Implementation of Acquisition Reform Initiatives 1 and 2)
S E C R E T A R Y O F T H E A R M Y W A S H I N G T O N MEMORANDUM FOR SEE DISTRIBUTION SUBJECT: Army Directive 2017-22 (Implementation of Acquisition Reform Initiatives 1 and 2) 1. References. A complete
More informationInsider Threat Webinar Series Defense Personnel Security and Research Center
Insider Threat Webinar Series Defense Personnel Security and Research Center Today s Session: Host: Rebecca Morgan, CDSE Insider Threat Guests: Dr. Eric Lang, Director, PERSEREC Callie Chandler, Deputy
More information