An Analysis of An Integrated Systems Engineering & Test & Evaluation Approach for Collaborative System of Systems

An Analysis of An Integrated Systems Engineering & Test & Evaluation Approach for Collaborative System of Systems Mr. Walter Ott Ph.D. Candidate The George Washington University Washington, D.C. wott@gwmail.gwu.edu Bill A. Olson, Ph.D. CSEP-Acq Professorial Lecturer The George Washington University Washington, D.C. bolson@gwu.edu Paul Blessner, Ph.D. Professorial Lecturer The George Washington University Washington, D.C. pbless@gwu.edu 1

2 Disclaimer The research, results and conclusions, and recommendations are the opinion of the author and should not be construed as an official position of the United States Department of Defense, Joint Staff or Science Applications International Corporation (SAIC).

Agenda Introduction The Problem Methodology Summary 3

4 Introduction

Close Air Support Mission Close Air Support (CAS): air action by fixed-wing and rotarywing aircraft against hostile targets that are in close proximity to friendly forces, and requires detailed integration of each air mission with the fire and movement of those forces. Immediate CAS: occurs outside the planning cycle; may involve reallocating airborne aircraft. (U.S. Air Force photo by Tech. Sgt. Michael R. Holzworth) 5 Source: JP 3-09.3 Close Air Support. (2009). Chairman Joint Chiefs of Staff

Close Air Support Mission Release Number: 370910 9/17/2010 - SOUTHWEST ASIA -- Coalition airpower integrated with Operation New Dawn partners in Iraq and the International Security Assistance Force in Afghanistan in the following operations September 16, according to Combined Air and Space Operations Center officials here. Close Air Support (CAS): air action by fixed-wing and rotarywing aircraft against hostile Air Operations in Afghanistan targets September that are 16, in 2010: close proximity to friendly Balocan, forces, RC-W and requires detailed integration of each air mission with the fire and movement of those forces. A U.S. Air Force B-1B aircrew, U.S. Navy F/A-18C pilots and U.S. Air Force A-10 Thunderbolt II pilots provided armed overwatch for friendly forces. F/A-18C pilots conducted a show of force. The B-1B aircrew release precision-guided munitions over an enemy target to deter enemy activity. A-10 pilots conducted a show of force, performed multiple strafing runs, and release multiple precision-guided munitions over multiple enemy positions to deter enemy activity, Joint terminal attack controllers deemed the engagements successful. Source: http://www.afcent.af.mil/news/story_print.asp?id=123222782 Immediate CAS: occurs outside the planning cycle; may involve reallocating airborne aircraft. (U.S. Air Force photo by Tech. Sgt. Michael R. Holzworth) 6 Source: JP 3-09.3 Close Air Support. (2009). Chairman Joint Chiefs of Staff

Systems Engineering Historical emphasis on system(s) Within the last 10-15 years, rise in: 7 Family of Systems (FOS) Systems of Systems (SoS) Enterprise Architecture (EA) New emphasis on User capabilities Resolution of capability gaps Integration of systems End-to-end process Net-centric versus platform centric

System of Systems: set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities Source: Defense Acquisition Guidebook. (2013). Retrieved from https://acc.dau.mil/docs/dag_pdf/dag_complete.pdf. 8 Types of SoS - Directed: SoS objectives, management, funding, and authority; systems are subordinated to a SoS. - Acknowledged: SoS objectives, management, funding, and authority; however, systems retain their own management, funding, and authority in parallel with the SoS. - Collaborative: No top down objectives, management authority, responsibility, or funding at the SoS level; systems voluntarily work together to address shared or common interests. - Virtual: Like collaborative but systems do not know each other. Definition Example Systems Stakeholder Involvement Source: Office of the Deputy Under Secretary of Defense for Acquisition and Technology, Systems and Software Engineering. Systems Engineering Guide for Systems of Systems, Version 1.0. Washington, DC: ODUSD(A&T)SSE, 2008. System versus Collaborative SoS System Collaborative SoS A functionally, physically, and/or behaviorally related group of regularly interacting or interdependent elements; that group of elements forming a unified whole. (3) The component systems interact more or less voluntarily to fulfill agreed upon central purposes. The Internet is a collaborative system. The Internet Engineering Task Force works out standards but has no power to enforce them. The central players collectively decide how to provide or deny service, thereby providing some means of enforcing and maintaining standards. (5) May include communities of practice (COPs). Aegis Weapon System Digitally-Aided Close Air Support (DACAS); Digitally- Aided Fire Support (DAFS); Joint Personnel Recovery (JPR) Clearer set of stakeholders. Stakeholders committed to that system and play specific roles in the SE of the system.(11) Multiple levels of stakeholders. Often competing interests between individual system level, and the SoS level.(12) Governance (Management) Aligned PM and funding. Governance of the SE process is usually hierarchical. (11) Added levels of complexity due to management and funding for both SoS and systems; SoS has no control over constituent systems. Constituent system s program managers and systems engineers collaborative at the SoS level to influence system development that meets SoS needs. Governance of SoS SE process will necessarily take on a collaborative nature. (12) Operational Focus (Operational Environment) Designed and developed to meet operational objectives. Mission objectives are established on a structured requirements or capability development process along with defined concepts of operation and priorities for development.(13) Changes in systems to meet SoS requirements are based on agreements and collaboration. SoS needs are based on agreement and collaboration, not top-down authority from the SoS manager. (6) Acquisition Test & Evaluation Boundaries & Interfaces Performance & Behavior Aligned to ACAT Milestones, documented requirements, SE with a Systems Engineering Plan (SEP). Cross multiple system lifecycles, involving legacy systems, developmental systems, and technology insertion; collaboration on development of SoS requirements. Test and Evaluation of the [entire] system, or subsystems, is possible. Multiple systems at different stages of life cycle. Need to expand or redefine existing SE processes to accommodate unique considerations of individual systems to address the overall SoS needs.(13) Focuses on boundaries and interfaces for the single system. Focus on identifying contributing systems, gain their participation, and enable data flow, understanding, and functionality across the SoS based on participating systems concurrence. Performance of the system to meet specified performance objectives. Performance across the SoS that meets SoS user capability needs as agreed upon by the participating systems.

9 The Problem Set

The Problem Set Services [continue to] independently develop, field, and manage systems that are not interoperable in the Systems-of-Systems (SoS) environment 10 Stove-piped solutions, lack interoperability Require workarounds or extensive resources to mitigate interoperability shortfalls Problem set magnified when operating in the Coalition/multi-national environment Question: How does the SE community manage the SoS Engineering challenge to achieve an interoperable SoS? SoS challenge magnified by Systems at or approaching end of life-cycle Independently evolving legacy systems New and emerging systems COTs and proprietary systems/software Rate of technological change Asynchronous Life Cycles

Interoperability The interoperability problem is not new However, the pace of technology and the need to operate in a SoS environment have increased its impact 1986 1987 1993 1996 1998 2003 2003 2005 2008 2003 Source: Military Readiness: Lingering Training and Equipment Issues Hamper Air Support of Ground Forces. (2003). Retrieved from http://www.gao.gov/assets/240/238142.pdf. 11

Interoperability Military services have not yet achieved DoD s goal The for ensuring interoperability that equipment problem acquired is not for close new air support missions is interoperable and cost-effective. However, the pace of technology and the need to operate in a SoS environment have increased its impact The digital systems are not interoperable across the services The lack of interoperability across the services reduces the equipment s effectiveness and limits its usefulness. 1986 1987 1993 1996 1998 2003 2003 2005 2008 2003 Source: Military Readiness: Lingering Training and Equipment Issues Hamper Air Support of Ground Forces. (2003). Retrieved from http://www.gao.gov/assets/240/238142.pdf. 12

Interoperability Interoperability: The ability of systems, units, or forces to provide data, information, materiel, and services to and accept the same from other systems, units, or forces AND to use the data, information, materiel, and services so exchanged to enable them to operate effectively together. Interoperability is more than just information exchange. It includes systems, processes, procedures, organizations and missions over the life cycle and must be balanced with information assurance. (DoDD 4630.5) Functional Interoperability: Also known as syntactic interoperability and technical interoperability. It implies coordination across organizational lines. The higher the number of stakeholders and their dependencies, the stronger the need to define common interface specifications. All parties must agree to implement a common profile or interface template (Sherif, 2009) Semantic Interoperability: focused on the consistent definition of data elements, ensuring consistency from system to system and application to application. Achieved through the development and use of standards for data and their definitions, minimizing interpretation errors. (Sherif, 2009) 13 Source: (a) DoDD 4630.5 Interoperability and Supportability of Information Technology (IT) and National Security Systems (NSS). (2007). Retrieved from http://www.dtic.mil/whs/directives/corres/pdf/463005p.pdf. (b) Sherif, M. H. (2010). Handbook of Enterprise Integration. Boca Raton, Florida: Auerbach Publications; 1 edition (November 9, 2009).

The Complexity of Interoperability Interoperability Procedures Equipment Hardware and software Standards Combination of the above Source Digital Interoperability Optimizing the Kill Chain. (2012) Precision Strike Technology Symposium. Source Digital Interoperability Optimizing the Kill Chain. (2012) Precision Strike Technology Symposium. Source: USAF AFLCM/HNB (2013) Cursor On Target, COT-101. 14

15 Standards: Panacae or Pandora s Box? 3 elements to VMF: Message Bearer(s) Combat Net Radio protocols Message Header(s) Required for message transfer Data not contained in message Message Format(s) Catalogue of messages MIL-STD 188-220 MIL-STD 2045-47001 MIL-STD 6017 MIL-STD 188-220 CNR Protocols (Bearer Protocols) 188-220A 1st 188-220B 188-220C 188-220D 188-220D Ch1 Source: Robinson, T. (2011). International Data Link Society's Guide to Variable Message Format (VMF) International Data Link Society (IDLS) (1.0 ed., pp. 2): IDLS. Header (Message Header) Message 2045-47001 Order of Transmission TIDP-R2 6017 2045-47001A R3 6017A 2045-47001B R4 6017B 2045-47001CVMF Data R5 Stream 6017C 2045-47001D R6 2045-47001D Ch1 R6+ CNR Protocols (Bearer Protocols) 188-220A Last 188-220B 188-220C 188-220D 188-220D Ch1 To be interoperable, systems must use the same Message Bearer, Message Header, and Message Format standards

CAS Mission Performance Performance factors: Accuracy Flexibility Timeliness Situational Awareness (U.S. Air Force photo by Tech. Sgt. Michael R. Holzworth) 16

CAS Today: Non-standard, Non-Interoperable F18A+/C/D/E/F F15E Link 16 AFAPD F16B40/50 B52 F16B40/50 AC130 MTS AV8B AC130 CoT A10C VMF F18A+/C/D/E/F F16B30 A10C SADL ASOC Gateway JSOTF (SATCOM) JTAC AFATDS DASC AFAPD Air Force Applications Program Development AFATDS Advanced Field Artillery Tactical Data System ASOC Air Support Operations Center CoT Cursor on Target Source Digital Interoperability Optimizing the Kill Chain. (2012) Precision Strike Technology Symposium. DASC JSOTF MTS SADL Direct Air Support Center Joint Special Operations Task Force Marine Tactical System Situational Awareness Data Link --------------- SADL --------------- Link-16 --------------- CoT --------------- AFAPD --------------- VMF ---------------- MTS

DACAS CI Vision F15E F18A+/C/D/E/F Link 16 Airborne Gateway F18A+/C/D/E/F F35 F16B40/50 AC130 AC130 CoT A10C B52 VMF AC130 AV8B AH64 AH1 F16B30 SADL A10C JSOTF DASC JFO (SATCOM) Ground Commander ASOC Gateway JTAC Reduces number of legacy digital communication capabilities JTAC responsible for by 43% ASOC CoT DASC Air Support Operations Center Cursor on Target Direct Air Support Center JSOTF SADL VMF Joint Special Operations Task Force Situational Awareness Data Link Variable Message Format --------------- SADL --------------- Link-16 --------------- CoT --------------- VMF

Basic Research Question Does an Integrated SE and T&E Process improve a SoS s Performance? Hypothesis Research Questions H1a Use of an Integrated SE & T&E Process contributes significantly and positively to reducing SoS interoperability risk over time H1b Use of an Integrated SE & T&E Process has no significant impact on SoS interoperability risk over time H1c Use of an Integrated SE & T&E Process contributes significantly and negatively to SoS interoperability risk over time 19

End-to-End Process Comparison Immediate CAS Mission Process Today, limited cross-service digital interoperability Reliant on voice The To Be Process Cross-Service interoperability Leverage digital M2M data exchanges The As Is versus the To Be Source: Robinson, T. (2011). International Data Link Society s guide to: Variable Message Format. Source: JP 3-09.3 Close Air Support. (2009). Chairman Joint Chiefs of Staff 20

21 Methodology

Methodology Utilizing an Integrated Systems Engineering and Test & Evaluation Approach, conduct an analysis of the Systems Engineering process for a collaborative SoS Validate the applicability of the approach for collaborative SoS by 22 Analysis of Risk Reduction and Test events results Improved Syntactic SoS interoperability Quantitative analysis of Semantic interoperability Analysis of As Is vs. To Be performance

Data sources Interoperability Test and Risk Reduction Event Reports Population Constituent aircraft systems and ground kits (JTAC/JFO) Dynamic population due to asynchronous life cycles, availability of resources at System, and nature of the collaborative SoS Validation of Research Data and Validation Provide quantitative methodology for semantic interoperability 23

Examined SoS Models DoD SoS Wave Model SoS Engineering Process 1, SoS Needs & Objectives 2. SoS Needs & Objectives 2. SoS Capstone Requirements Allocation 5. SoS 1. SoS Enterprise 3. SoS Conops 4. SoS Scenario Operational Strategic Plan Architecture 6. SoS Project 2. Integrated SoS Acquisition Strategy 3. Control System Programs 1. Capability Requirements 2. SoS Systems Architecture. Control 1. Technology Investment & 4. Solution 9. SoS Analysis Development Design & Assessment Plan 5. System Enterprise Activities Implementation Development Activities 8. System 6. System Reengineering Operational Activities Verification and/or Disposal Technical Activities 10. SoS Verification 7. System Capability Development 11. SoS Validation 3. SoS Operations SoS Engineering Process Model (Source: Sage, Biemer 2007) Sources: Dahmann, J., Rebovich, G., Lane, J., Lowry, R., & Baldwin, K. (2012). An implemented view of systems engineering for systems of systems. Aerospace and Electronic Systems Magazine, IEEE, 27(5), 11-16. doi: 10.1109/MAES.2012.6226689 Sage, A. P., & Biemer, S. M. (2007). Processes for System Family Architecting, Design, and Integration. Systems Journal, IEEE, 1(1), 5-16. doi: 10.1109/JSYST.2007.900240

Tx/Rx Quantifying Semantic Interoperability No Impact Interoperability Broken Interoperability Broken No Impact Interoperability Impacted Bearer Protocols Match? No Not Interoperable No No No No Are There Discretionary Data Authentication Code (DAC) Category (CAT) Fields? Are Discretionary Data Authentication Code (DAC) Category (CAT) Fields implemented? Are There Special Considerations Yes Are Special Considerations Implemented? No No Yes No Start Message Header Match? Are Required Protocols Matching? (Syntatic) Yes Examine Data Field Element Minimum Implementation Examine Data Authentication Code (DAC) Column Are There Mandatory Data Authentication Code (DAC) Category (CAT) Fields? Yes Are Mandatory Display Column Fields Implemented? Examine Display Column Fields Examine Special Consderations Yes Display Not Prohibited Interoperability Impacted No No Are There Display Field Requirements? Are Display Requirements Implemented? Message Standard Match? Yes Yes Matrix tables limited to qualitative depiction of interoperability Message 5 Rx/Tx System A System B System C System D System E System F System G System H Non-compliance impacts with data field, display and special considerations requirements Mandatory with Display Mandatory, no display with Special considerations Discretionary with Display Interoperability Maintained Method & Location System Specific Tabular Display Graphical Display Interoperability Maintained

Expected Outcomes Improved Syntactic Interoperability between SoS constituent systems Improved Semantic interoperability between SoS constituent systems Improved SoS Mission performance (timeliness) 26

Summary Challenges to SoSE are unique Disparate asynchronous lifecycles No single approach can solve these emerging problems, and thus no strategy is best for all projects (Gorod, Sauser, and Boardman, 2008) An Integrated Systems Engineering Development and Test & Evaluation Approach is valid for a collaborative SoS SoS approach must ensure synchronization of upgrades 27

References (a) Dahmann, J. (2012, 19-22 March 2012). Integrating systems engineering and test & evaluation in system of systems development. Paper presented at the Systems Conference (SysCon), 2012 IEEE International. (b) Sage, A. P., & Biemer, S. M. (2007). Processes for System Family Architecting, Design, and Integration. Systems Journal, IEEE, 1(1), 5-16. doi: 10.1109/JSYST.2007.900240 (c) Choi, D., & Sage, A. P. (2012). A framework for interoperability assessments in Systems of Systems and Families of Systems. Information, Knowledge, Systems Management, 11(3), 275-295. (d) Maier, M. W. (1998). Architecting principles for systems-of-systems. Systems Engineering, 1(4), 267-284. (e) Chen, P., & Clothier, J. (2003). Advancing systems engineering for systems of systems challenges. Systems Engineering, 6(3), 170-183. 28

References (f) Keating, C., Rogers, R., Unal, R., Dryer, D., Sousa-Poza, A., Safford, R.,.Rabadi, G. (2008). System of systems engineering. Engineering Management Review, IEEE, 36(4), 62-62. (g) Gorod, A., Sauser, B., & Boardman, J. (2008). System-of-systems engineering management: a review of modern history and a path forward. Systems Journal, IEEE, 2(4), 484-499 (h) Dahmann, J., Rebovich, G., Lane, J., Lowry, R., & Baldwin, K. (2012). An implemented view of systems engineering for systems of systems. Aerospace and Electronic Systems Magazine, IEEE, 27(5). (i) Joint Publication (JP) 3-09.3, Close Air Support. (2009). Chairman Joint Chiefs of Staff (j) Sherif, M. H. (2010). Handbook of Enterprise Integration. Boca Raton, Florida: Auerbach Publications; 1 edition (November 9, 2009). (k) Hura, M., McLeod, G., Larson, E., Schneider, J., & Gonzales, D. (2000). Interoperability: A Continuing Challenge in Coalition Air Operations Santa Monica, California: Rand Corporation. 29

30 Back-Up Slides

The CAS Challenge http://www.youtube.com/results?search_query=500lb%20bomb%20dropped%20 on%20us%20soldiers%20by%20mistake&sm=1 Watch timeframe 0:00-0:25 Warning: Explicit language http://www.youtube.com/watch?v=wwrguzbuhiw 31