CONTRACT-BASED AIR TRANSPORTATION SYSTEM (CATS) Operational Assessment A Human in the Loop experiment to assess the CoO concept from ATCO & Pilot point of view Presented by Sandrine Guibert EUROCONTROL Second CATS Workshop Geneva January 26th, 2010 Page: 1
Goal Proof of Concept Study, following E-OCVM Idea Implemented Concept V0 V1 V2 V3 V4 V 5 ATM Needs Scope Feasibility Integration Pre-operation Operation CATS Identify ATM performance needs & constraints Scope operational concepts and create validation strategy Iteratively develop and evaluate concept Integrate concept in wider context And confirm performance Industrialisation and procedure approval Implementation E-OCVM Concept Lifecycle Model The CATS operational assessment is focused on the En-route part and covers the V1 level (Scope) and V2 level (Feasibility) of the Level of Maturity. Page: 2
Goal Second operational assessment to evaluate the operational acceptability of the CoO and associated TWs concepts from the controllers' and pilots point of view. Operational acceptability will be evaluated in the context of the transfer of responsibility area between two ANSPs, and the relationship between ATCOs and Pilots. Evaluation environment is restricted to 4 2 en-route controller working positions (CWP: EXE + PLN) which manage the traffic and handover the aircraft 4 and two cockpit positions The objectives of the HIL2 experiment is twofold: Analyse the collaboration process between controllers and aircrew regarding the TW management. Analyse the impact of the TW management on the aircrew's activity in the cockpit. Page: 3
HIL2 Hypothesis CoO implementation allows safe operations. CoO is still manageable even with increase of traffic as foreseen in 2020. CoO implementation affects positively the flight within the sector (flight duration...). Implementation of TWs ensures the respect of schedule. TWs integrate flexibility to cope with uncertainty. The working methods offered to ATCOs and pilots, as a result of the CoO implementation, are feasible and acceptable (task sharing, role and responsibility, as well as the offered support tools). Implementation of CoO does not impose significant additional workload to ATCOs or pilots. Page: 4
Methodology Evaluation principles: SESAR KPAs (Key Performance Area): system performance 4 4 KPAs are applicable for CATS Safety Capacity Efficiency Predictability Objectives relating to human performance Page: 5
Operational issues evaluation Objectives relating to system performance Objectives relating to human performance SAFETY CAPACITY EFFICIENCY Feasibility and acceptability of the aircrew & ATCos' working methods due to the CoO execution Impact of CoO execution on aircrew & ATCOs' performance Impact of CoO on aircrew & ATCOs' activity PREDICTABILITY Indicators Indicators SAF.LOCAL.ER. PI (1, 2, 3, 5, 6 & 8) CAP.LOCAL.ER. PI (2, 8, 10, 11, 12 & 13) Workload: ISA, NASA-TLX, Interviews, Observations, Performance outcomes, Questionnaire Situation Awareness: SASHA_Q, Interviews, EFF.LOCAL.ER. PI (1, 7, 8, 9, 10, 11) Observations, Performance outcomes, questionnaire Error production and management: Observations, PRED.LOCAL.ER. PI (1 & 2) Questionnaire, Interviews, Performance outcomes Operator's activity: Cognitive processes, Decision making, Risk management, Constraints, etc. Collaborative and R/T activity: Communications (number, time, content, speaker and receiver, etc.) Number of TWs fulfilled Page: 6
Experimental environment Traffic: Real traffic Traffic load has been adapted to the airspace Current level of traffic (2008) 2020 traffic load (EUROCONTROL STATFOR) Simulation facilities SkySoft platform (rapid prototyping adapted from SkyGuide simulator) 2 sectors (MI1 & KL1) EXE and PLN on each CWP 2 cockpits (Microsoft Flight Simulator) No pseudo-pilot: Automatic order execution by aircraft Automatic hand over/ assume by feed sectors Page: 7
Experimental environment Measured Sectors ACC Sector Min FL Max FL KL1 FL275 FL345 LSAG LIRR LSAGKL1 LIRRMI1 FL275 FL275 FL345 FL345 MI1 FL275 FL345 Crew1 Pilot KL ATCos Crew2 Pilot MI Exe Pln Exe Pln Page: 8
Conduct of the experiment Duration: 10 days Dates: 19th. to the 23th. October 2009. 26th. to the 30th. October 2009. Location: Skysoft premises Geneva Four controllers experienced in En-route ATC Two experimented pilots One on site pre-training session at Roma ACC and in Airfrance Page: 9
Conduct of the experiment Timetable: Day 19 th. October Morning Simulation devices presentation Afternoon Familiarization 20 th. October 21 th. October 22 th. October 23 th. October Familiarization Operational training (session Operational training (session 2) 1) Operational training (session Experimental runs #1, 2 & 3 3) Experimental runs #4, 5 & 6 26 th. October Familiarization and Training Experimental runs #7 & 8 27 th. October 28 th. October 29 th. October 30 th. October Experimental runs #9, 10 & 11 Experimental runs #12, 13 & 14 Experimental runs #15 & 16 Final Debriefing Spare day Page: 10
Conduct of the experiment Daily Programme Daily Programme 0830 0845 0900 1045 1045 1100 1100 1245 1245 1415 SkySoft arrival Set-up in Operations Room Exercise 1 + Debriefing & Questionnaires Break Exercise 2 + Debriefing & Questionnaires Lunch 1415 1600 Exercise 3 + Debriefing & Questionnaires 1630 Leave Page: 11
Experimental plan (1) Independent variables: With and without TWs 2 levels of traffic (2008/expected in 2020) Disruptions (nominal & non nominal scenario) 4 similar traffic scenarios Independent variables TWs Traffic load disruptions ATCO X X X Pilot X X Geographic Sector Load 30 Geo Sect Load Geo Sect Load (Avg) 25 20 Nb of Aircraft 15 10 5 0 16:00 16:10 16:20 16:30 16:40 16:50 17:00 17:10 17:20 17:30 Simulation Time Page: 12
Experimental Plan (2) Dependent variables: Observations 4 ATCOs performance (Overt The Shoulder OTS rating scales, FAA) 4 Spontaneous verbalisations Performance measurements 4 STCA, Separation Performance Indicators (Intervention type to resolve predicted conflict, Time of intervention before potential LoS, Distance before potential LoS, Localisation of controllers interventions) 4 ATCO orders 4 Flight duration 4 Fulfilled TW Self assessment 4 Workload : ISA & NASA-TLX 4 Situation awareness : SASHA_Q 4 Safety feeling : questionnaire on risks & hazards Post run interviews & Post experimental questionnaires (concept, performance, cooperation, simulation) Page: 13
Experimental plan (3) 16 runs and 32 measures: 2 traffic loads per run 8 runs with TW for 2008 & 2020 traffic loads for ATCO 8 runs without TW for 2008 & 2020 traffic loads for ATCO The 4 experimental conditions were compared for each ATCOs team (EXE-PLN for MI1 & EXE-PLN for KL1 ) 16 x 4 measured flight segments for pilots Page: 14
ATCOs Workload (ISA) 5.0 ISAWorkLoad KL1_EXE 5.0 ISAWorkLoad MI1_EXE 4.0 4.0 3.0 3.0 2.0 2.0 1.0 1.0 5.0 2008/2008-TW -- No Impact -- p > 0.250 (z:0) 2020/2020-TW -- No Impact -- p = 0.148 (z:1.05) 2008/2020 -- Impacted -- p = 18 (z:-2.10) 2008TW/2020TW -- Impacted -- p = 47 (z:-1.68) ISAWorkLoad KL1_PLN 2008/2008-TW -- No Impact -- p > 0.250 (z:0.26) 2020/2020-TW -- No Impact -- p = 65 (z:-1.52) 2008/2020 -- Impacted -- p = 33 (z:-1.84) 2008TW/2020TW -- Impacted -- p = 02 (z:-2.84) ISAWorkLoad MI1_PLN 5.0 4.0 4.0 3.0 3.0 2.0 2.0 1.0 1.0 2008/2008-TW -- No Impact -- p > 0.250 (z:0.53) 2008/2008-TW -- No Impact -- p = 0.216 (z:-0.79) 2020/2020-TW -- No Impact -- p = 0.232 (z:-0.74) 2020/2020-TW -- No Impact -- p > 0.250 (z:-5) 2008/2020 -- Impacted -- p = 05 (z:-2.57) 2008/2020 -- Impacted -- p = 33 (z:-1.84) 2008TW/2020TW -- Impacted -- p = 21 (z:-2.05) 2008TW/2020TW -- Impacted -- p = 21 (z:-2.05) Not significant between with TW and without TW Significant between 2008 and 2020 traffic load Page: 15
ATCOs Workload self assessment (NASA TLX) 9 8 NASA TLX WorkLoad KL1_EXE 9 8 NASA TLX WorkLoad MI1_EXE 7 7 6 6 5 5 4 3 2 1 2008/2008-TW -- No Impact -- p = 0.174 (z:-0.95) 2020/2020-TW -- No Impact -- p > 0.250 (z:-0.32) 2008/2020 -- Impacted -- p < 02 (z:-3.36) 2008TW/2020TW -- Impacted -- p < 02 (z:-2.94) 4 3 2 1 2008/2008-TW -- No Impact -- p > 0.250 (z:0.32) 2020/2020-TW -- No Impact -- p > 0.250 (z:0.32) 2008/2020 -- Impacted -- p < 02 (z:-2.94) 2008TW/2020TW -- Impacted -- p < 02 (z:-3.36) 9 8 NASA TLX WorkLoad KL1_PLN 9 8 NASA TLX WorkLoad MI1_PLN 7 7 6 6 5 5 4 4 3 3 2 2 1 1 2008/2008-TW -- No Impact -- p > 0.250 (z:-0.63) 2020/2020-TW -- No Impact -- p = 0.174 (z:0.95) 2008/2020 -- Impacted -- p < 02 (z:-2.94) 2008TW/2020TW -- Impacted -- p = 23 (z:-2.00) 2008/2008-TW -- No Impact -- p > 0.250 (z:-0.63) 2020/2020-TW -- No Impact -- p = 0.114 (z:-1.21) 2008/2020 -- Impacted -- p = 06 (z:-2.52) 2008TW/2020TW -- Impacted -- p < 02 (z:-2.99) Not significant between with TW and without TW Significant between 2008 and 2020 traffic load Page: 16
Pilots Workload self assessment (NASA TLX) 40 35 NASA TLX Pilots Significant impact of TW implementation. 30 25 20 15 10 Pilot evaluated this impact as fully acceptable with the workload induced by the other cockpit tasks during the cruise phase. 5 0 Without TW With TW Wilcoxon test: Variable impact results if p<0,05 With-TW / Without- TW Impact P=0,0000 8 25%-75% Min-Max However, experiments should be done to validate the workload induced by the TWs in emergency situations or in high workload conditions. Page: 17
ATCOs Situation Awareness (SASHA- Q) 6.0 5.5 SASHA (Situational Awareness for SHAPE) KL1_EXE 6.0 5.5 SASHA (Situational Awareness for SHAPE) MI1_EXE 5.0 5.0 4.5 4.5 4.0 4.0 3.5 3.5 3.0 3.0 2.5 2.5 2.0 1.5 1.0 0.5 2.0 1.5 1.0 0.5 2008/2008-TW -- No Impact -- p > 0.250 (z:0) 2020/2020-TW -- No Impact -- p > 0.250 (z:-0.11) 2008/2020 -- Impacted -- p < 02 (z:3.15) 2008TW/2020TW -- Impacted -- p < 02 (z:3.05) 2008/2008-TW -- No Impact -- p > 0.250 (z:-5) 2020/2020-TW -- No Impact -- p > 0.250 (z:-5) 2008/2020 -- Impacted -- p = 04 (z:2.68) 2008TW/2020TW -- Impacted -- p = 07 (z:2.47) 6.0 5.5 SASHA (Situational Awareness for SHAPE) KL1_PLN 6.0 5.5 SASHA (Situational Awareness for SHAPE) MI1_PLN 5.0 5.0 4.5 4.5 4.0 4.0 3.5 3.5 3.0 3.0 2.5 2.5 2.0 2.0 1.5 1.5 1.0 1.0 0.5 0.5 2008/2008-TW -- No Impact -- p = 0.124 (z:-1.16) 2008/2008-TW -- Impacted -- p = 26 (z:1.94) 2020/2020-TW -- No Impact -- p = 0.187 (z:-0.89) 2020/2020-TW -- Impacted -- p = 06 (z:2.52) 2008/2020 -- Impacted -- p < 02 (z:3.26) 2008/2020 -- Impacted -- p = 18 (z:2.10) 2008TW/2020TW -- Impacted -- p = 02 (z:2.84) 2008TW/2020TW -- No Impact -- p = 65 (z:1.52) Not significant between with TW and without TW, except MI PLN. Significant between 2008 and 2020 traffic load Page: 18
Pilots Situation Awareness (SASHA- QP) 6,2 SASHA Pilots 6,0 5,8 5,6 5,4 5,2 5,0 4,8 4,6 4,4 4,2 Without TW With TW 25%-75% Min-Max Wilcoxon test: Variable impact results if p<0,05 With-TW / Without- TW No impact P=0,2425 Not significant between with TW and without TW Page: 19
OTS Performance OTS overall Performance 2020 (Over The sholders observations) KL1 OTS overall Performance 2020 (Over The sholders observations) MI1 6.0 5.5 6.0 5.5 5.0 5.0 4.5 4.5 4.0 4.0 3.5 3.5 3.0 3.0 2.5 2.5 2.0 2.0 1.5 1.5 1.0 1.0 0.5 0.5 2020 2020-TW 2020 2020-TW 2020/2020-TW -- No Impact -- p > 0.250 (z:-0.37) 2020/2020-TW -- No Impact -- p > 0.250 (z:0) Not significant between with TW and without TW Page: 20
Acceptability and Usability Concept has been described by pilots and ATCOs as: Feasible & Acceptable TW was manageable, for 2008 as well as 2020, even if TW management is an additional task Traffic volume constrained ATCOs more than the TW Training proposed provided sufficient knowledge on the concept, platform, HMI and tools Pilots and ATCOs find the concept easy to learn and to use Potential improvements: For ATCOs, some aspects regarding the TWs positions within the sector have to be changed in TWs calculation For Pilots, there is a need to more anticipate the following TWs on display. Page: 21
ATCOs Orders ATCO GLOBAL Orders/hours KL1_EXE ATCO GLOBAL Orders/hours MI1_EXE 14 12 14 12 10 10 8 8 6 6 4 4 2 2 2008/2008-TW -- No Impact -- p = 0.161 (z:1.00) 2020/2020-TW -- No Impact -- p = 0.201 (z:0.84) 2008/2020 -- Impacted -- p = 08 (z:-2.42) 2008TW/2020TW -- Impacted -- p = 37 (z:-1.79) 2008/2008-TW -- No Impact -- p = 0.124 (z:1.16) 2020/2020-TW -- No Impact -- p = 0.174 (z:0.95) 2008/2020 -- No Impact -- p = 0.148 (z:-1.05) 2008TW/2020TW -- No Impact -- p = 0.201 (z:-0.84) Not significant between with TW and without TW Significant between 2008 and 2020 traffic load for KL Detailed analysis shows: no impact of TW implementation on FL and speed orders impact on Go To orders, as expected, significant in 2008, not in 2020. Page: 22
RT Communication 2,2 Radio Communication 2,0 1,8 1,6 1,4 Controller communication duration with piloted aircraft shows no significant difference between with TW and without TW whatever the traffic load conditions 1,2 1,0 0,8 2008 2020 2008TW 2020TW 25%-75% Min-Max 2020 traffic condition generates longer communication duration, even if it is not significant. Wilcoxon test: Variable impact results if p<0,05 2008/2008-TW 2020-2020-TW 2008-2020 2008TW/2020TW No impact No impact No impact No impact P=0,0678 P=0,2733 P=0,1441 P=0,1441 Page: 23
Working Methods Pilots and ATCOs affirmed that TWs implementation induces no particular changes in their working methods TW management was consider as a task mainly for PLN, for traffic preparation. TW management was consider as a PF task 4 But a need to have party line and maybe TWs on FMS Safety remains the first priority Conflict resolution first then TW achievement Collaboration Aircraft s intentions more shared between operators not impacted between ATCOs 4 PLN able to help the EXE through collaborative tool pilots have the feeling TW implementation could increase slightly the communication between the aircrew. Page: 24
Safety: Aircraft Separation 2008 No loss of separation Majority of the traffic is maintained of more than 10Nm and 1000ft Page: 25
Safety: Aircraft Separation 2020 No loss of separation A high safety level is maintained whatever the traffic load conditions and/or the TWs conditions The TWs do not impact the controllers anticipation to manage safety Pilots and ATCOs reported in questionnaires that their safety feeling was not impaired by the TWs use Page: 26
Efficiency: Flight duration FlightDuration for Pseudo-piloted A/C KL1 FlightDuration for Pseudo-piloted A/C MI1 12 115.0 12 115.0 11 11 105.0 105.0 10 10 95.0 95.0 9 9 85.0 85.0 8 8 2008/2008-TW 2020/2020-TW 2008/2020 2008TW/2020TW -- Impacted -- -- No Impact -- -- No Impact -- -- No Impact -- p = 14 (z:2.21) p = 0.201 (z:0.84) p > 0.250 (z:-0.53) p = 0.174 (z:-0.95) 2008/2008-TW 2020/2020-TW 2008/2020 2008TW/2020TW -- Impacted -- -- No Impact -- -- No Impact -- -- Impacted -- p = 05 (z:2.63) p > 0.250 (z:0.63) p = 71 (z:1.47) p = 47 (z:1.68) Significant between with TW and without TW in both sectors in 2008 conditions Significant between with TW and without TW in both sectors in 2020 condition for Milano only The flight durations "with TW" are closer to the reference than "without TW" whatever the traffic load conditions. Page: 27
Efficiency: TWs OUT Target Windows Out (Percentage) KL1 Target Windows Out (Percentage) MI1 10 10 8 8 6 6 4 4 2 2 2008-TW 2020-TW 2008-TW 2020-TW 2008TW/2020TW -- No Impact -- p > 0.250 (z:-0.42) 2008TW/2020TW -- No Impact -- p > 0.250 (z:0.42) 10 Target Windows Out (Percentage) KL1 10 Target Windows Out (Percentage) MI1 8 8 6 6 4 4 2 2 2020-TW 2020EV-TW 2020-TW 2020EV-TW 2020-TW/2020EV-TW -- No Impact -- p > 0.250 (z:-0.42) 2020-TW/2020EV-TW -- No Impact -- p = 0.201 (z:-0.84) No significant difference (p<0,05) between the two traffic load conditions whatever the controlled sector or event- very low level of TWs OUT Page: 28
Efficiency: Fuel Burned Fuel burned FS1_PIL Fuel burned FS2_PIL 100 90 100 90 80 80 70 70 60 60 50 50 40 40 30 30 20 20 10 10 2008/2008-TW -- No Impact -- p = 0.248 (z:0.68) 2020/2020-TW -- No Impact -- p > 0.250 (z:-0.11) 2008/2020 -- No Impact -- p > 0.250 (z:-5) 2008-TW/2020-TW -- No Impact -- p > 0.250 (z:-0.42) 2008/2008-TW -- No Impact -- p > 0.250 (z:-0.53) 2020/2020-TW -- No Impact -- p > 0.250 (z:-0.58) 2008/2020 -- No Impact -- p > 0.250 (z:-0.21) 2008-TW/2020-TW -- No Impact -- p > 0.250 (z:0.11) No significant difference (p<0,05) for the fuel burned by the aircraft whatever the traffic load conditions and the TWs implementation Page: 29
Conclusions CoO concept is manageable and acceptable with the 2008 & 2020 traffic loads, without any impact on Safety. CoO improve the predictability in times and routing ATCOs and pilots agreed CoO could optimise ATM system. However, the forecasted increase of traffic remains a limitation. New tools are required to support the traffic load. The number of TWs not fulfilled predicts positive prospect for HIL3. Page: 30
Thank you for your attention! www.cats-fp6.aero/ www.eurocontrol.int/eec/cats Page: 31