SoS Systems Engineering (SE) and Test & Evaluation (T&E) Final Report of the NDIA SE Division SoS SE and T&E Committees

Transcription

1 Systems Engineering (SE) and Test & Evaluation (T&E) Final Report of the NDIA SE Division SE and T&E Committees March

2 Abstract This paper presents an approach to integrated systems engineering (SE) and test and evaluation (T&E) for based on work underway by the National Defense Industry Association Systems Engineering Division Systems of Systems and mental Test and Evaluation Committees. The paper focuses on how to approach T&E for given the challenges of large scale development as a continuous improvement process that provides information on capabilities and limitations for end users and feedback to the and system SE teams toward evolution. 2

3 Facility Schedule Dependencies Security Integration Test Signal cables wrong length or incorrect Power/Cooling connections Safety interconnect Test equipment Equipment in NFR Signal cable delivery RE/NBDC/SDP availability RE array availability Test results prior to string integration Detailed tasks for integration/prep Classified shipment Classified workstations Classified field returns Procedures incomplete or incorrect SDP emulator Work-arounds required Cable Tasks Setup Testing Security Task NDIA Strategic Initiative: Best Practices Model for T&E Product of one-day facilitated and T&E Workshop sponsored by NDIA SE and DTE committees, held August 17, 2010, MITRE, McLean VA Adopted by NDIA SE Committee to work with T&E Committee to address this as a 2011 action Purpose: Outline the fundamentals of the model of T&E as a: Continuous improvement process supporting capabilities and limitations information for end users and feedback to the and system SE teams toward evolution of the 2010 IEEE E Conference NDIA Test and Evaluation Conference Paper #11651 Test and Evaluation Issues for Systems of Systems: Creating Sleep Aids for Those Sleepless Nights Beth Wilson, Raytheon Tom Wissink, Lockheed Martin Darlene Mosser-Kerner, OSD DT&E NDIA T&E Division, mental Test and Evaluation Committee Judith Dahmann, MITRE John Palmer, Boeing NDIA SE Division, Systems of Systems Committee Data Collection: White Paper Facilitated Workshop: The Technique SE Conference Papers Rob Heilman, TRMC Bob Aaron, ATEC Strategic Initiative Coleads NDIA T&E Conference Mar Improvement Areas: Strategic Initiatives Collaborative Go-Do NDIA T&E Conference Mar 2011 Potential Problem Areas 1) Future T&E for Systems brought together as 2) Requirements 3) Metrics 4) Systems Changes 5) End to End Testing with systems not yet available Potential Causes If we could only fix one thing, it would be Leverage Matrix Map Causes to problem areas Transition from Problem Space to Solution Space 6 3

4 Definition, Types and Domains : A set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities Types of Directed: objectives, management, funding and authority; systems are subordinated to Acknowledged: objectives, management, funding and authority; however systems retain their own management, funding and authority in parallel with the Collaborative: top down objectives, management, authority, responsibility, or funding at the level; Systems voluntarily work together to address shared or common interest Virtual: Like collaborative, but systems don t know about each other Domains Sets of systems working together to provide a broader capability or mission A military platform (e.g. ship, aircraft, satellite, ground vehicle) equipped with additional independent systems (e.g. sensor, weapons communication) needed to meet platform objectives Platform IT-Based Networked information systems to support operations within or across platforms or systems to meet mission or platform objectives Model focuses on Acknowledged Level 4

5 Comparing Systems and Stakeholder Involvement System Clearer set of stakeholders Acknowledged System of Systems Management & Oversight Two levels of stakeholders with mixed possibly competing interests Governance Aligned PM and funding Added levels of complexity due to management and funding for both and systems; does not have control over over all constituent systems Operational Focus Acquisition Test & Evaluation Boundaries & Interfaces Performance & Behavior Designed and developed to meet operational objectives Aligned to established acquisition processes Test and evaluation the system is possible Operational Environment Called upon to meet operational objectives using systems whose objectives may or may not align with the system s objectives ation Cross multiple system lifecycles across acquisition programs, involving legacy systems, developmental systems, and technology insertion; Capability objectives but may not have formal requirements Testing more challenging due systems asynchronous life cycles and given the complexity of all the moving parts Engineering & Design Considerations Focuses on boundaries and interfaces Performance of the system to meet performance objectives Focus on identifying systems contributing to objectives and enabling the flow of data, control and functionality across the while balancing needs of the systems Performance across the that satisfies user capability needs while balancing needs of the systems Reference: US DoD Guide for Systems Engineering of Systems of Systems T&E Implications Validation criteria more difficult to establish Cannot explicitly impose conditions on system T&E System level operational objectives may not have clear analog in conditions that need T&E Depends on constituent system test of requirements as well as level Difficult to bring multiple systems together for T&E in synchrony with capability evolution Additional test points needed to confirm behavior Increased subjectivity in assessing behavior, given challenges of system alignment 5

6 SE as the Framework for T&E Effective application of SE at the level provides a structured framework to address T&E challenges Approaches to managing asynchronous system development and test itecture approaches which shelter the from changes in systems Effective T&E is grounded in a clear understanding of objectives and requirements of the test item The value of an is accrued from the collective behavior of the toward user capabilities Systems engineering conducted at the level provides the basis for T&E DoD SE Guide, SE artifacts and wave model provide fundamentals of SE for DoD SE and T&E share key common elements It can be difficult to tell where SE stops and T&E begins 6

7 Aspects of T&E for SE and T&E To Support Evolution Address risk of changes in constituent systems including this planned for support of the objectives and those planned by the constituent systems independently from the Can be viewed as Verification and Validation (V&V) Done as part of the evolution, not as a separate process T&E Feedback on Fielded overall performance and identify operational problems and emergent (unexpected) behavior Typically draws on data from operationally realistic environments May be periodic and not tied directly to update cycles T&E is also needed when testing individual systems in an context This approach may apply there as well, but the focus at the level Focus here is on T&E Support to Evolution 7

8 Wave Model: Framework for Model Analysis Analysis Analysis Presented at IEEE Systems Conference April 2011 [1] An implementer s view of SE More familiar and intuitive time-sequenced wave model representation Information is thus rendered in a form more readily usable by SE practitioners in the field Representation that corresponds with incremental development approaches that are the norm for capability evolution [1] An ers View of Systems of Systems Dahmann, Baldwin, Rebovich, Lane and Lowry Concept of Wave ning was developed by Dr. David Dombkins See Complex Project Management Booksurge Publishing, South 8 Carolina: 2007.

9 SE and T&E Evolution at Each Step Recognize T&E constraints Full T&E to address changes in constituent systems is not feasible given the size and complexity of many and the dynamic nature of constituent systems Includes conventional live testing and approaches using various forms of virtual and constructive simulation Focus T&E specifically on areas of risk Begin with the changes which have been made in the Identify where changes could have adverse impacts on the user missions the risk using evidence from a range of sources including live test Evidence can be based on activity at the level, as well as roll-ups of system level activity and can be explicit verification testing, results of models and simulations, use of linked integration facilities, and results of system level operational test and evaluation Results Continuous improvement feedback to End users in the form of capabilities and limitations rather than as test criteria for deployment SE teams of both the and systems on progress and issues Analysis External Environment Analysis Analysis 9

10 External Environment Analysis Analysis Analysis Approach Assumes Initiation of an Acknowledged Decision has been made to establish an SE organization An entity is responsible for the with SE support to the As an acknowledged, the systems which constitute the maintain operational and management independence At the initiation of an, the information typically available includes initial or first order Statement of top-level objectives for the ( capability objectives) Description of how systems in the will be employed in an operational setting ( CONOPS) and Programmatic and technical information about systems that affect capability objectives (systems information) Risks are identified when an is launched and mitigation actions are tracked and updated throughout each cycle, along with new risks (Risks and Mitigations) 10

11 JFC s Guidance and Intentions JP3-60 JP3-60 JP3-60 JP3-60 JP3-09 ned (known) Scheduled Is Target a TST? Combat ment On-Call J P 3-30 ALSA TST TTP Manual Time-Sensitive Target Categories ISR /TST Criteria Achieved ISR Options Desired Environment Available Assets Hard/Soft Kill Soft Immediate Unplanned (Known) Hard Engage Target ALSA Unanticipated (Unknown) Locate Identify Classify Characterize Remove From Target List Joint TST Commander Target? Schedule BDA Option Generate/ to Track de Track Mgr de Target Mgr de Mgr de Commander JFC s Guidance and Intentions Is Target a TST? Combat ment ISR /TST Criteria Achieved ISR Options Desired Environment Available Assets Hard/Soft Kill Soft Hard Engage Target Locate Identify Classify Characterize Remove From Target List ISR /TST Criteria Target? Schedule BDA Option ISR Locate Identify Classify Characterize Generate/ to Track Options Desired Environment Available Assets Schedule BDA Option Generate/ to Track External Environment SE: Analysis Provides analysis of the as is and basis for evolution Analysis Analysis Analysis Multiple Options CONOPs Thread Selection Selected FA FCB TST depiction Based on current joint analysis (FA FCB), maps to ALSA, and general acceptance PLAN FIND FI TRACK t Achieved ASSESS TARGET Joint TST Commander ENGAGE de Track Mgr de Target Mgr de Mgr de Commander Force Application itecture ment Final Report Understand operational context and developing a CONOPS -- Includes key steps in process and constraints on those steps; may be a set of mission threads, conditions, players and performance objectives Requirements Space Thread definition and depiction in the JBMC2 Roadmap A time-sensitive target (TST) is defined as those targets requiring immediate response because they pose (or will soon pose) a danger to friendly forces or are highly lucrative, fleeting targets of opportunity. Also called TST. (JP 1-02) PLAN FIND FI TRACK t Achieved ASSESS TARGET ENGAGE Thread components Find Fix Track Target Engage Activities associated with each component FIND FI TRACK TARGET ENGAGE Search for Humans Search for Vehicles Collect Surface Fixed Contact Data Collect Communications Signals Data Collect n- Communications Signals Data Collect Directed Energy Data Functional Baseline Translate Information Correlate Information Fuse Information Classify Humans Classify Vehicles Provide Positive Identification of Friendly Forces TST Thread Find Fix Track Target Engage Track Humans Track Vehicles Maintain Order of Battle (OB) Perform Munitions ment Perform Weapons Systems ment Perform Tactics ment an functional architecture for the by looking at the key functions to be supported across the thread or activity sequence, including performance objectives Engage Land Surface Destroy Fixed Surface Destroy Moving Surface Create Lethal Incapacitate Human Neutralize or Disrupt Vehicular Surface Layout the specific tasks for each component of the activity sequence to further delineate the functionality supporting the E2E capability objective Analysis Links Current System Baseline ISR tasking systems / platforms: ADOCS BANDSAW DCGS FoS GCCS-A GCCS-M JTT TBMCS WEEMC ASAS Find Fix Track Target Sensor platforms: ACS AEGIS AESA E-2C EP-3 Global Hawk JSTARS MP-RTIP Predator PROPHET RAH-66 RC-135 SHARP TUAV U2 MTI TACELINT TBMCS LAN ADOCS LAN Systems for Detecting, Fixing and Tracking (1): A2IPB ADOCS BANDSAW ASAS DCGS FoS DTSS EPPIC GALE ISR-M ISR-W IWS JSIPS-N JSWS JTT IPL MIDB 5D GBS Wideband Gapfiller Win-T Systems for Detecting, Fixing and Tracking (2): MATRE MTI PTW Raindrop TEG TES TES-N TBMCS WEEMC Waterfall Targeting platforms: AEGIS AWACS E-2C JSTARS MC2A Targeting systems: ADOCS AFATDS APS ATWCS DJC2 EMT IFSAS ITO MCS TADIL-J,-A,-B AMPS JTIDS MIDS Link 16 SADL IBS Targeting systems (2): GCCS-A GCCS-J GCCS-M IMTDS ISR-M IWS JFCOM JFI JC2 JMPS JTT N PTW TBMCS TAMPS TTWCS TPS WTP WEEMC ADSI RWS IVS MTS TACFIRE Execution systems: AWACS IMTDS JSTARS Weapon platforms (1): A-10 AC-130 AH-1 ALCM AV-8B B1-B B-2 B-52 CG/CG() Link 11B AFADP Link 11 Engage Weapon platforms (2): DDG/DD() F/A-22 F-15 F-16 F-18 F-117 JSF KC-135 Land Warrior MC-130/M MLRS RAH-66 S-3B Stryker Tomahawk UA Howitzer Link 4A JTRS (all clusters) Same systems list as Find, Fix, Track (Tasking and ISR systems employed to conduct BDA) Systems in Blue were Increment 1 Pathfinder systems; Systems in Red are Increment 2 Pathfinder systems; Systems in Brown are to be added to the Roadmap Source: JBMC2 Roadmap, V1.0 Identify systems supporting the capability objectives and align them to the components and functionality needs, with data on current performance Results provide basis for architecture development and planning for updates SE Artifacts Characterize Capability objectives CONOPs Constituent system info Technical Baselines Performance Measures & Methods Performance Data Requirement Space Risks & Mitigations for SE SE ning Elements Master Agreements 11

12 JP3-60 JP3-60 JP3-60 JP3-60 JP3-09 J P 3-30 ALSA TST TTP Manual ALSA JFC s Guidance and Intentions Is Target a TST? Combat ment ISR /TST Criteria Achieved ISR Options Desired Environment Available Assets Hard/Soft Kill Soft Hard Engage Target Locate Identify Classify Characterize Remove From Target List Joint TST Commander Target? Schedule BDA Option Generate/ to Track de Track Mgr de Target Mgr de Mgr de Commander JFC s Guidance and Intentions ned (known) Scheduled Is Target a TST? Combat ment Time-Sensitive Target Categories ISR /TST Criteria Achieved On-Call ISR Options Desired Environment Available Assets Hard/Soft Kill Soft Immediate Unplanned (Known) Hard Engage Target Unanticipated (Unknown) Locate Identify Classify Characterize Remove From Target List Joint TST Commander Target? Schedule BDA Option Generate/ to Track de Track Mgr de Target Mgr de Mgr de Commander ISR /TST Criteria ISR Locate Identify Classify Characterize Generate/ to Track Options Desired Environment Available Assets Schedule BDA Option External Environment T&E Component of Analysis Multiple Options CONOPs Thread Selection CONOPs Selected FA FCB TST depiction Based on current joint analysis (FA FCB), maps to ALSA, and general acceptance PLAN FIND FI TRACK t Achieved ASSESS TARGET ENGAGE Force Application itecture ment Final Report Understand operational context and developing a CONOPS -- Includes key steps in process and constraints on those steps; may be a set of mission threads, conditions, players erstand performance operational objectives context and eloping a CONOPS -- Includes steps in process and constraints hose steps; may be a set of sion threads, conditions, players performance objectives quirements Space Search for Humans Search for Vehicles Collect Surface Fixed Contact Data Collect Communications Signals Data Collect n- Communications Signals Data Collect Directed Energy Data Functional Baseline Translate Information Correlate Information Fuse Information Classify Humans Classify Vehicles Provide Positive Identification of Friendly Forces TST Thread Find Fix Track Target Engage Track Humans Track Vehicles Maintain Order of Battle (OB) Perform Munitions ment Perform Weapons Systems ment Perform Tactics ment Engage Land Surface Destroy Fixed Surface Destroy Moving Surface Create Lethal Incapacitate Human Neutralize or Disrupt Vehicular Surface Layout the specific tasks for each component of the activity sequence to further delineate the functionality supporting the E2E capability objective Current System Baseline Requirements Space Thread definition and depiction in the JBMC2 Roadmap A time-sensitive target (TST) is defined as those targets requiring immediate response because they pose (or will soon pose) a danger to friendly forces or are highly lucrative, fleeting targets of opportunity. Also called TST. (JP 1-02) PLAN FIND FI TRACK t Achieved TARGET ENGAGE Thread components Activities associated with each component Engage ASSESS Identify systems supporting the Find Fix Track Target FIND FI TRACK TARGET ENGAGE capability objectives and align them to the components and Analysis Layout the specific tasks for each component of the activity sequence to further delineate the functionality supporting the E2E capability objective Analysis an functional architecture for the by looking at the key functions to be supported across the thread or activity sequence, including performance objectives CONOPs, mission threads functionality needs, with data on Analysis and tasks are all needed Capability current performance and performance elements for structuring test objectives provide a foundation an functional architecture for the by for T&E looking at the key functions to Systematic be development supported across the and analysis of this data is core thread or activity sequence, to analysis including and performance supports the development of the architecture, objectives planning of updates Cases where more data is needed (and testing may be required) are identified T&E foundations are established in analysis which draws on T&E of fielded systems Links ISR tasking systems / platforms: ADOCS BANDSAW DCGS FoS GCCS-A GCCS-M JTT TBMCS WEEMC ASAS Analysis Sensor platforms: ACS AEGIS AESA E-2C EP-3 Global Hawk JSTARS MP-RTIP Predator PROPHET RAH-66 RC-135 SHARP TUAV U2 MTI TACELINT TBMCS LAN ADOCS LAN Systems for Detecting, Fixing and Tracking (1): A2IPB ADOCS BANDSAW ASAS DCGS FoS DTSS EPPIC GALE ISR-M ISR-W IWS JSIPS-N JSWS JTT IPL MIDB 5D GBS Wideband Gapfiller Win-T Systems for Detecting, Fixing and Tracking (2): MATRE MTI PTW Raindrop TEG TES TES-N TBMCS WEEMC Waterfall Find Fix Track Target Targeting platforms: AEGIS AWACS E-2C JSTARS MC2A Targeting systems: ADOCS AFATDS APS ATWCS DJC2 EMT IFSAS ITO MCS TADIL-J,-A,-B AMPS JTIDS MIDS Link 16 SADL IBS Targeting systems (2): GCCS-A GCCS-J GCCS-M IMTDS ISR-M IWS JFCOM JFI JC2 JMPS JTT N PTW TBMCS TAMPS TTWCS TPS WTP WEEMC ADSI RWS IVS MTS TACFIRE Analysis Execution systems: AWACS IMTDS JSTARS Weapon platforms (1): A-10 AC-130 AH-1 ALCM AV-8B B1-B B-2 B-52 CG/CG() Link 11B AFADP Link 11 Engage Weapon platforms (2): DDG/DD() F/A-22 F-15 F-16 F-18 F-117 JSF KC-135 Land Warrior MC-130/M MLRS RAH-66 S-3B Stryker Tomahawk UA Howitzer Link 4A JTRS (all clusters) Identify systems supporting th capability objectives and align them to the components and functionality needs, with data current performance Current System Baseline Same systems list as Find, Fix, Track (Tasking and ISR systems employed to conduct BDA) Systems in Blue were Increment 1 Pathfinder systems; Systems in Red are Increment 2 Pathfinder systems; Systems in Brown are to be added to the Roadmap Source: JBMC2 Roadmap, V1.0 Identify systems supporting the capability objectives and align them to the components and functionality needs, with data on current performance Understanding current system performance draws on available evidence from various sources, including systems T&E 12

13 External Environment Analysis Analysis Analysis SE: itecture s and evolves the persistent technical framework for addressing evolution SE Artifact itecture Defines the way in which the constituent systems work together Includes systems, functions, relationships and dependencies, as well as end-to-end functionality, data flow & communications 13

14 Observe Observe Battlespace Battlespace / Threat Data Command & Control Manned - Unmanned Forces ( ning) Battlespace / Threats / Requirements Prepare COAs s / Orders Command & Control Manned - Unmanned Forces ( / Escort) Maneuver Orders Manned Maneuver ( / Escort) Unmanned Maneuver ( / Escort) Cooperative Unmanned Maneuver ( / Escort) Cooperative Manned-Unmanned Maneuver ( / Escort) Manned Maneuver (ISR) Unmanned Maneuver (ISR) Cooperative Unmanned Maneuver (ISR) Cooperative Manned-Unmanned Maneuver (ISR) Command & Control Manned - Unmanned Forces () Detect / ID / Surface Threats Orders Manned n-lethal of Unmanned n-lethal of Cooperative Unmanned n- Lethal of Cooperative Manned- Unmanned n-lethal of Manned Lethal of Unmanned Lethal of Cooperative Unmanned Lethal of Cooperative Manned- Unmanned Lethal of Sustain Command & Control Manned - Unmanned Forces (Recover / Reconstitute) Recovery / Reconstitution Orders Recover / Reconsitute Observe Observe Battlespace Battlespace / Threat Data Command & Control Manned - Unmanned Forces ( ning) Battlespace / Threats / Requirements Prepare COAs s / Orders Command & Control Manned - Unmanned Forces ( / Escort) Maneuver Orders Manned Maneuver ( / Escort) Unmanned Maneuver ( / Escort) Cooperative Unmanned Maneuver ( / Escort) Cooperative Manned-Unmanned Maneuver ( / Escort) Manned Maneuver (ISR) Unmanned Maneuver (ISR) Cooperative Unmanned Maneuver (ISR) Cooperative Manned-Unmanned Maneuver (ISR) Command & Control Manned - Unmanned Forces () Detect / ID / Surface Threats Orders Manned n-lethal of Unmanned n-lethal of Cooperative Unmanned n- Lethal of Cooperative Manned- Unmanned n-lethal of Manned Lethal of Unmanned Lethal of Cooperative Unmanned Lethal of Cooperative Manned- Unmanned Lethal of Sustain Command & Control Manned - Unmanned Forces (Recover / Reconstitute) Recovery / Reconstitution Orders Recover / Reconsitute MESF Manned-Unmanned Cooperative, Semi-Autonomous GOPLAT Defense Phases (/ISR) Units / Platforms / Systems GOPLAT(s) MESF Craft#1 (Lead) (SURC / RAC / SOC-R) MESF Craft#2 (SURC / RAC / SOC-R) TUSV#1 TUSV#2 MESF TOC TUAV#1 (FireScout / ScanEagle / STUAS NTA Defense NTA Maritime Law Enforcement NTA Protect/Secure Operationally Critical Installations, Facilities, and Systems NTA Provide Harbor Defense and Port Security Distribute / Disseminate Intentions / Orders NTA Tactical Operations s Distribute / Disseminate Orders NTA 5.3 Determine and Actions and Operations Distribute / Disseminate Intentions / Orders NTA 5.3 Determine and Actions and Operations NTA Tactical Operations NTA Tactical Operations Brief Forces: 2xMESF Craft, standard loadout 2xTUSVs, sensors, non-lethal + lethal weapons payload LCS in the AOO, operating 1xTUAV, sensors CONOPS: MESF Craft patrol + Auto/Semi-Auto cooperative TUSV patrol TUAV provides surveillance and combat ID TUSVs extend MESF sensors, provide combat ID and engagement MESF Manned-Unmanned Cooperative, Semi-Autonomous GOPLAT Defense Phases (/ISR) Units / Platforms / Systems GOPLAT(s) MESF Craft#1 (Lead) (SURC / RAC / SOC-R) MESF Craft#2 (SURC / RAC / SOC-R) TUSV#1 TUSV#2 MESF TOC TUAV#1 (FireScout / ScanEagle / STUAS / MRUAS) (TBD) LCS (TBD) CTF-56 (Constructive) JFMCC (C2F MOC) NTA Defense NTA Maritime Law Enforcement NTA Protect/Secure Operationally Critical Installations, Facilities, and Systems NTA Provide Harbor Defense and Port Security Distribute / Disseminate Intentions / Orders NTA Tactical Operations s Distribute / Disseminate Orders NTA 5.3 Determine and Actions and Operations Distribute / Disseminate Intentions / Orders NTA 5.3 Determine and Actions and Operations NTA Tactical Operations NTA Tactical Operations Brief Forces: 2xMESF Craft, standard loadout 2xTUSVs, sensors, non-lethal + lethal weapons payload LCS in the AOO, operating 1xTUAV, sensors CONOPS: MESF Craft patrol + Auto/Semi-Auto cooperative TUSV patrol TUAV provides surveillance and combat ID TUSVs extend MESF sensors, provide combat ID and engagement? Preflight TUAV Prepare USV for Prepare USV for Preflight / MRUAS) (TBD) LCS (TBD) CTF-56 (Constructive) JFMCC (C2F MOC) Launch TUAV Launch Auto- Surveillance Area USVs Monitor TUAV Surveillance Data NTA Tactical Reconnaissance and Surveillance Semi-Auto- Monitor s Monitor s Transmit Distribute TUAV Surveillance Data? NTA Maneuver in Formation Cooperatively USV USV NTA Maintain and Display Tactical Picture Monitor COP Monitor COP Preflight TUAV Prepare USV for Prepare USV for Preflight NTA Provide Area Security NTA s 2 Coop Coop Transmit Transmit Launch TUAV Monitor USV Monitor USV Monitor USV Launch USV Sensors Auto- Surveillance Area USV Redirect Sensors Redirect Sensors USVs LCS MESF Semi-Auto- Monitor s Monitor s Transmit Monitor TUAV Surveillance Data NTA Tactical Reconnaissance and Surveillance MESF Coord with GOPLATs Situation Situation CTF Monitor / Intentions Distribute TUAV Surveillance Data TUAV TOC NTA Maneuver in Formation Monitor COP Monitor COP Monitor / Intentions Maneuver Cooperatively USV USV NTA Maintain and Display Tactical Picture CTF Monitor / Intentions NTA Provide Area Security NTA s 2 Coop Coop Transmit Transmit Monitor USV Monitor USV Monitor USV USV Sensors USV Redirect Sensors Redirect Sensors LCS MESF MESF Coord with GOPLATs Situation Situation CTF Monitor / Intentions TUAV TOC Monitor / Intentions Maneuver CTF Monitor / Intentions T&E Component of itecture Each mission plan / CONOP / COA describes a path through the various steps in the mission Identify and evaluate alternative approaches to organizing and augmenting systems to meet needs itecture Identify Systems Contributing to Capability Objectives ROC 01 Lanes des Systems ROC 05 ROC 05 ROC 05 ROC 02 ROC 02 ROC 01 ROC 05 ROC 03 ROC 03 Specify how current systems support the capability objectives Identify specific systems supporting the capability objectives and align them to functionality needs Analysis External Environment Analysis Analysis Data on attributes and performance of systems (typically drawn from system T&E) is key to identification and analysis of architecture approaches T&E contributes to the assessment of alternative architectures through application of various approaches including LVC environments to assess alternatives against desired architecture objectives 20 14

15 SE: Evaluates the priorities, options and backlogs to define the plan for the next upgrade cycle. Analysis External Environment Analysis Analysis Artifacts An allocated baseline Risks and mitigations Agreements ation, integration & test plans An integrated master schedule (IMS) d Master, Technical baselines Requirements space 15

16 MESF Manned-Unmanned Cooperative, Semi-Autonomous GOPLAT Defense M is s ion Phases Ex ec ute(/isr) Units / Platforms / Systems GOPLAT(s ) MESF Craft#2 (SURC / RAC / SOC-R) TUSV#1 TUSV#2 MESF TOC TUAV#1 (FireSc out/ ScanEagle / STUAS / M RUAS) (TBD) LCS (TBD) CTF-56 (Cons tructive) JFMCC (C2F MOC) NTA Defens e NTA Maritime Law Enforcement NTA Protect/Secure Operationally Critic al Installations,Facilities,and Sys tems NTA Provide Harbor Defense and Port Security Intentions / Orders NTA Tactical Operations s Orders 2x TUSVs, sensors, non -le t ha l + l et ha l we ap on s pa yl oa d LCS in the AOO, operating 1xTUAV, sensors NTA 5.3 Determine and Actions and Operations Dis tribute / Di s s e m i na te Intentions / Orders NTA 5.3 Determine and Actions and Operations NTA Tactical Operations CONOPS: M ESF Craft patrol +Auto/Sem i-auto c ooperativ e TUSV patrol TUAV prov ides surv eillanc e and combat ID TUSVs ex tend MESFsens ors, provide c ombat ID and engagement Prepare USV for NTA Tactical Operations? Do wn l o a d Preflight (M aneuver) Up l o a d (M aneuver) (M aneuver) Up l o a d (M aneuver) Up l o a d (Pay load) Do wn l o a d (M aneuver) Do wn l o a d (M aneuver) (Pay load) Launch Launch TUAV Do wn l o a d (Pay load) Do wn l o a d (Pay load) Auto-M aneuv er to Surv eillance Area USVs Trans m it M onitor / Dis tribute TUAV Surv eillance Data M aneuver to M onitor / Up d a te USV M aneuvers Cooperatively M aneuver to Auto-Co o p M aneuver to Auto-Co o p M aneuver to NTA Tac tical Reconnaissanc e and Surv eilance Sem i-auto- M onitor s NTA Provide Area Security NTA Maneuv er in Form ation NTA s M onitor COP M onitor COP M aneuvers NTA Maintain and Dis play Tactic al Picture 2 Coop Coop Auto-Co o p Auto-Co o p Trans m it Trans m it M onitor USV M onitor / Up d a te USV Sens ors M onitor USV Redirec t Sens ors Redirec t Sens ors M aneuvers LCS M ESF M ESF TOC M onitor / CTFUp d a te TUAV M aneuvers M onitor M is sion / Intentions Maneuver Up d a te CTF M onitor M is sion / Intentions T&E Component of Analysis External Environment Analysis Analysis MESF Craft#1 (Lead) (SURC / RAC / SOC-R) Dis tribute / Di s s e m i na te Dis tribute / Di s s e m i na te Brief Forc es : 2xMESF Craft, s tandard loadout Analysis of Impacts of Changes Changes in the are identified (both planned by the and planned independently by the constituents) What are the potential impacts of these changes? What are the risks? What evidence is there that these changes will not adversely impact other systems and mission objectives? What data is needed and how can this data be obtained? Can this be done as part of the system tests? Are added test events needed? How are these incorporated into the overall plan and IMS? What testing tools and environments are needed to address the specific challenges? Test drivers to address asynchronous development? Use of Live, Virtual and Constructive (LVC) environments to address specific risks? Prepare USV for Preflight TUAV Up l o a d Up l o a d M onitor s M onitor TUAV Surv eillance Data M onitor / Up d a te USV M onitor USV M onitor / Up d a te USV Coord with GOPLATs As s es s Situation As s es s Situation M onitor M is sion / Up d a te Intentio ns A critical part of planning an update is the analysis of changes and risks to identify the areas to be addressed by T&E 16

17 External Environment Analysis Analysis Analysis SE: Monitors implementations at the system level and plans and conducts level testing, resulting in a new product baseline Artifacts Test Report Technical s, Requirements Space, Performance Data System Test Reports IMS Technical Baselines 17

18 ISR tasking systems / platforms: ADOCS BANDSAW DCGS FoS GCCS-A GCCS-M JTT TBMCS WEEMC ASAS Sensor platforms: ACS AEGIS AESA E-2C EP-3 Global Hawk JSTARS MP-RTIP Predator PROPHET RAH-66 RC-135 SHARP TUAV U2 MTI TACELINT TBMCS LAN ADOCS LAN Systems for Detecting, Fixing and Tracking (1): A2IPB ADOCS BANDSAW ASAS DCGS FoS DTSS EPPIC GALE ISR-M ISR-W IWS JSIPS-N JSWS JTT IPL MIDB 5D GBS Wideband Gapfiller Win-T Systems for Detecting, Fixing and Tracking (2): MATRE MTI PTW Raindrop TEG TES TES-N TBMCS WEEMC Waterfall TADIL-J,-A,-B JTIDS Targeting platforms: AEGIS AWACS E-2C JSTARS MC2A Targeting systems: ADOCS AFATDS APS ATWCS DJC2 EMT IFSAS ITO MCS AMPS Targeting systems (2): GCCS-A GCCS-J GCCS-M IMTDS ISR-M IWS JFCOM JFI JC2 JMPS JTT N PTW TBMCS TAMPS TTWCS TPS WTP WEEMC ADSI RWS Execution systems: AWACS IMTDS JSTARS Weapon platforms (1): A-10 AC-130 AH-1 ALCM AV-8B B1-B B-2 B-52 CG/CG() Weapon platforms (2): DDG/DD() F/A-22 F-15 F-16 F-18 F-117 JSF KC-135 Land Warrior MC-130/M MLRS RAH-66 S-3B Stryker Tomahawk UA Howitzer Same systems list as Find, Fix, Track (Tasking and ISR systems employed to conduct BDA) MESF Manned-Unmanned Cooperative, Semi-Autonomous GOPLAT Defense Phases (/ISR) Units / Platforms / Systems GOPLAT(s) MESF Craft#1 (Lead) (SURC / RAC / SOC-R) MESF Craft#2 (SURC / RAC / SOC-R) TUSV#1 TUSV#2 MESF TOC TUAV#1 (FireScout / ScanEagle / STUAS / MRUAS) (TBD) LCS (TBD) CTF-56 (Constructive) JFMCC (C2F MOC) NTA Defense NTA Maritime Law Enforcement NTA Protect/Secure Operationally Critical Installations, Facilities, and Systems NTA Provide Harbor Defense and Port Security Distribute / Disseminate Intentions / Orders NTA Tactical Operations s Distribute / Disseminate Orders NTA 5.3 Determine and Actions and Operations Distribute / Disseminate Intentions / Orders NTA 5.3 Determine and Actions and Operations NTA Tactical Operations NTA Tactical Operations Brief Forces: 2xMESF Craft, standard loadout 2xTUSVs, sensors, non-lethal + lethal weapons payload LCS in the AOO, operating 1xTUAV, sensors CONOPS: MESF Craft patrol + Auto/Semi-Auto cooperative TUSV patrol TUAV provides surveillance and combat ID TUSVs extend MESF sensors, provide combat ID and engagement? Preflight TUAV Prepare USV for Prepare USV for Preflight Launch TUAV Launch SADL IBS Auto- Surveillance Area IVS MTS TACFIRE USVs Link 11B AFADP Link 11 Monitor TUAV Surveillance Data NTA Tactical Reconnaissance and Surveillance Semi-Auto- Monitor s Monitor s Transmit Distribute TUAV Surveillance Data Link 4A JTRS (all clusters) NTA Maneuver in Formation Cooperatively USV USV NTA Maintain and Display Tactical Picture Monitor COP Monitor COP NTA Provide Area Security NTA s 2 Coop Coop Transmit Transmit Monitor USV Monitor USV Monitor USV USV Sensors USV Redirect Sensors Redirect Sensors LCS MESF MESF Coord with GOPLATs Situation Situation CTF Monitor / Intentions TUAV TOC Monitor / Intentions Maneuver CTF Monitor / Intentions External Environment Monitor System and ment, Integration and Test Integrated Master Schedule (Key sync points (not aggregation of plans) Risks and Mitigation s changes and dependencies which drive testing Systems Additions to system plans for development and test System of Systems Program 1 Constituent Systems Program 3 T&E Component of Collect and assess data from system and development technical reviews and tests product baseline, architecture, performance assessments, and requirements space Provide input into Analysis Review Progress And Inform Users and SE Process Lanes des Systems Links Find Fix Track Target MIDS Link 16 Engage Systems in Blue were Increment 1 Pathfinder systems; Systems in Red are Increment 2 Pathfinder systems; Systems in Brown are to be added to the Roadmap Source: JBMC2 Roadmap, V1.0 Analysis Analysis Analysis T&E is a key part of implementation for both the and the systems Systems making updates conduct T&E at the system level Program 5 Program 6 27 level T&E activities include Monitoring implementation of system testing, conducting added testing to address risks, and evaluating the results, recommending changes in plans as needed Results of the capability are identified (both planned and unplanned) Does performance meet expectations for this increment? What are the potential impacts on the next increment? What are the risks? What evidence is there that these changes will need to be regression tested in the next increment? 18

19 Summary and Next Steps Key elements of the approach to SE and T&E Addresses the key challenges facing T&E in an environment complexity, system independence and asynchronous development Integrates T&E with SE throughout the evolution of an based on the wave model T&E contributes to all steps in the evolution Focuses T&E on risks to systems and recognizing full end to end testing with each system change is intractable Emphasizes use range of information types to address these risks Presentation is the product of the 2011 joint task of the NDIA SE and T&E committees Represents initial product in this area Open areas and considerations for next steps 19