DEFENSE ACQUISITION UNIVERSITY F-35 Mission Systems Software Case Study Professor Jan Kinner prepared this case, with support from the F-35 JPO s Stephanie Brinley, in January 2017 for class discussion rather than to illustrate either effective or ineffective handling of a program situation. DISCLAIMER This case was produced in the Department of Defense (DOD) school environment to acquaint students with defense acquisition issues and to provide a basis for classroom discussion of these issues. The views expressed in this case are those of the authors or the case participants and do not reflect the official position or policy of the DOD or those of the United States Government. References in this case to the DODI 5000 series and life cycle phases reflect the use of terminology at the time our research was conducted on the program in question. 1
F-35 Mission Systems Software 1 You can see from its angled lines, the F-35 is a stealth aircraft designed to evade enemy radars. What you can t see is the 24 million lines of code which turn it into a flying computer. That s what makes this plane such a big deal. Source: David Martin, Is the F-35 Worth It? 60 Minutes, 16 Feb 2014 It was a little after 7 in the morning on a July, 2012 day and Bob Church, the F-35 Joint Program Office s (JPO) Air System Software Director was reviewing various documents related to the F- 35 s Mission Systems software. 2 At a review the previous day one of the first meetings attended by Bob in his new job - Lockheed-Martin s (LM) representatives claimed they were meeting cost and schedule targets on the Mission Systems software (SW) while the JPO s representatives asserted LM s measures contain partial truths and that LM was significantly behind schedule. The Mission Systems software was one of the top risks on the F-35 program. The program had been restructured with a new performance baseline three months ago so the disagreement between the LM and JPO representatives about the status of the Mission System SW raised red flags in Bob s mind. As part of the rebase-lining activities the Initial Operational Capabilities (IOC) dates were changed to TBD because of concerns about the Mission Systems software. Who s right? thought Bob as he thumbed through another program document. What is the current status of the Mission Systems software? Is it behind schedule? Will the right software, containing the right set of mission system capabilities needed to declare IOC be delivered on time, to the right aircraft? Bob knew the JPO s leadership some of whom attended the same meeting as Bob - would be asking him these and other questions shortly and expecting answers. The Joint Strike Fighter (JSF) The JSF was conceived as a relatively affordable fifth-generation strike fighter. It is a singleseat, single-engine aircraft incorporating low-observable (stealth) technologies, defensive avionics, advanced sensor fusion 3, internal and external weapons, and advanced prognostic maintenance capability. Also called the Lightning II and designated the F-35, it is being procured for the AF, USMC, and USN and numerous international partners in a cooperative development program. 4 There are three variants (Figure 1), each with multi-mission capabilities (Table 1). F-35A: The conventional takeoff and landing (CTOL) variant is a multi-role, stealthy strike aircraft replacement for the Air Force s F-16 Falcon, A-10 Thunderbolt II, and will complement the F-22A Raptor. 1 This case refers only to the Mission System software on the F-35. It does not include the Vehicle System software, Autonomic Logistics Global Sustainment software, Simulation software, or Laboratory software. 2 Although based upon actual work at the F-35 JPO the names have been disguised. 3 Sensor fusion is the ability to take information from both multiple onboard and off-board aircraft sensors and display the information in an easy-to-use format for the single pilot. 4 As of Dec, 2011 there were eight international partners participating with the US under Memorandums of Understanding. 2
F-35B: The short takeoff and vertical landing (STOVL) variant is a multi-role, stealthy strike fighter which will replace the Marine Corps F/A-18C/D Hornet, AV-8B Harrier, the Italian Navy s AV-8 aircraft, and replace the UK s Harrier GR7/9 and Harrier T10 aircraft. F-35C: The carrier-suitable variant (CV) is a multi-role, stealthy strike aircraft to complement the Navy s F/A-18 E/F Super Hornet. Figure 1. The F-35 s Family of Aircraft 3
Table 1. Required Mission Categories by F-35 Variant 5 The final number of F-35s each Service will field has fluctuated over time due to budget constraints and Service strategies. As of March, 2012 the DOD plans call for acquiring a total of 2,456 F-35s (13 RDT&E aircraft and 2,443 production aircraft). Hundreds of additional F-35s are expected to be purchased by U.S. allies, eight of which are cost sharing partners in the program. The Lockheed-Martin Corporation Missions F-35A (USAF) F-35B (USMC) F-35C (USN) Air Operations Air Interdiction Offensive Counter Air Anti-Air Warfare Close Air Support Strategic Attack Suppression of Enemy Air Defenses Destruction of Enemy Air Defenses Defensive Counter Air Armed Reconnaissance Aerial Reconnaissance Forward Air Controller (Airborne) Tactical Air Controller (Airborne) Combat Search & Rescue Support of Tactical Recovery of Aircraft & Personnel Strike Coordination & Reconnaissance Attack of Maritime Surface Targets Mining and Reconnaisance Assault Support Escort Inherent Electronic Protection Electronic Attack Electronic Warfare Support The Lockheed-Martin (LM) Corporation was formed in 1995 when the Lockheed Corporation merged with the Martin Marietta Corporation. Today, LM is a global security and aerospace company principally engaged in the research, design, development, manufacture, integration, and sustainment of advanced technology systems and products. LM also provides a broad range of management, engineering, technical, scientific, logistic, and information services. While LM s products and services span the globe, LM s principal customers are agencies of the U.S. Government. LM has four business segments: Electronic Systems, Information Systems & Global Solutions (IS&GS), Space Systems and Aeronautics. LM s Aeronautics Division, often called Aero is located in Fort Worth, Texas and is responsible for the design, development, manufacture, integration, sustainment and support of the F-35. The Division s personnel have a wealth of talent and experience developing products and software. 5 Some of these categories are often combined reducing the number of mission categories reported for the F-35 4
Acquisition History The JSF ACAT I-D program began in November 1996 with a five-year competition between Lockheed Martin (LM) and Boeing. Following a down-select, Lockheed-Martin (LM) was awarded a Cost-Plus Award Fee contract on 26 Oct 2001 to develop the Joint Strike Fighter, later designated the F-35 and named the Lightning II. On that same date the F-35 Program was approved for entry into the System Development and Demonstration Phase (SDD) of the Defense Acquisition System (Figure 2). Lockheed s contract called for an initial 22 aircraft (14 flying test aircraft and 8 ground-test vehicles) to be built during SDD which was expected to last 126 months. Flight testing of those aircraft was scheduled for Edwards AFB, CA and Naval Air Station, Patuxent River, MD. Figure 2. The Defense Acquisition System at the time of contract award 6 Lockheed committed to delivering full mission capability, using a block approach, by the end of SDD in FY 2012. Low Rate Initial Production (LRIP 7 ) was schedule to start in 2006 with Initial Operational Capability (IOC) for each of the services occurring between 2010 and 2012 (Figure 3). The decision to overlap development, production, and testing was deliberate because all three US services and the international customers wanted IOC at or about the same time. 6 Note: Subsequent revisions to the DODI 5000.02 changed the SDD phase to Engineering Manufacturing and Development (EMD) 7 The Low-Rate Initial Production (LRIP) is conducted during the Production and Deployment (PD) Phase. LRIP is the point in time where manufacturing development is completed and the ability to produce a small-quantity set of articles to provide for representation at Initial Operational Test and Evaluation (IOT&E). It also establishes an initial production base and sets the stage for a gradual increase in production rate to allow for Full-Rate Production (FRP) upon completion of Operational Test and Evaluation (OT&E). 5
Figure 3. The Joint Strike Fighter (F-35) Development Schedule, 2003 Like many other DOD programs (e.g. F-18, F-22) that have utilized this approach, concurrency risks materialized during SDD causing significant cost growth, schedule slippage, and performance shortfalls. The program was restructured and rebase-lined in 2004 due to weight and performance problems and again in 2007 because of cost and schedule slips. Cost and schedule problems continued to plague the program which led to another restructuring and rebaselining of the program in March, 2012. The cumulative effects of the rebase-linings extended the SDD phase from 126 months (10.5 years) to 192 months (16 years); delayed the start of LRIP by 16 months; and added several additional lots to LRIP which extended the completion of LRIP by several years. This, in-turn, delayed the start of flight testing, changed the planned declaration of IOC dates for each of the Services to TBD (Table 2), and delayed the Full Rate Production decision by seven years (Figure 4). Events Development APB 26 Oct 2001 Revised Development APB 17 Mar 2004 Revised Development APB 30 Mar 2007 Revised Development APB 26 Mar 2012 Objective Threshold Objective Threshold Objective Threshold Objective Threshold USMC IOC (F-35B) Apr 2010 Sep 2010 Mar 2012 Sep 2012 Mar 2012 Sep 2012 TBD TBD USAF IOC (F-35A) Jun 2011 Dec 2011 Mar 2013 Sep 2013 Mar 2013 Sep 2013 TBD TBD USN IOC (F-35C) Apr 2012 Sep 2012 Mar 2013 Sep 2013 Mar 2015 Sep 2015 TBD TBD Table 2. Initial Operating Date Changes 2001-2012 6
Figure 4. Lockheed s 2001 Plan vs the 2012 Plan F-35 Software Development Plan Software is the secret sauce of the F-35. Its software enables almost 90 percent of the F-35 s air system functionality, operations, and support (Figure 5). The software for the F-35 consists of off-board and on-board software. The off-board software consists of the Off-Board Mission Support (OMS), Autonomic Logistics Information System (ALIS), and Training software. The on-board embedded software also called the Air Vehicle software, is further decomposed into Vehicle Systems, Prognostic and Health Management (PHM), Weapons Integration, and Mission Systems software. The Vehicle Systems software manages the functionality and status of the aircraft (e.g., flight controls, fuels, hydraulics, electrical power system, etc.). The PHM systems link the aircraft sensors which monitor the health of flight critical, propulsion, airframe, mission and vehicle systems to off-board systems for logistics purposes. The Weapons Integration software is self-explanatory. The Mission Systems software provides the F-35 s war fighting capabilities and includes such things as infrared, optical radar sensing, integrated cockpit controls, displays, radio communications, navigation, identification, weaponry, Electronic Warfare, and the Helmet Mounted Display capabilities. Figure 5. Functionality provided by software, USAF comparison 8 8 Source: Firesmith, et al, The Model Framework for Engineering System Architectures, with information from Etter, 2000 updated with data from F-35 JPO 7
In order to provide incremental F-35 capabilities and support each Service s IOC requirements Lockheed Martin developed an Air System Block Plan. The Air System Block Plan consists of an infrastructure development block (Block 0.1), a mission capability development block (Block 0.5) and three JCS Capability Blocks (Figure 6). The software development plans for each Figure 6. Lockheed s Air System Block Plan category of software mirrors and supports the Air System Block Plan. Each block built upon the previous block until the full set of capabilities in Block 3 - traceable to the Joint Contract Specification (JCS) and JORD 9 would be delivered early in LRIP. Initial Operational Test & Evaluation (IOT&E) would then commence. Following successful completion of IOT&E the F- 35 Program would enter Full Rate Production where each aircraft being produced would be fully mission capable when it rolled off the end of the production line. Although concurrent development of the Vehicle and Mission Systems software was occurring the primary emphasis in the early stages of the program was on developing and fielding the Vehicle Systems software (Blocks 0.1 and 0.5) in order to meet first flight and early factory deliveries. Development of Block 0.1 (often referred to as flight sciences software ), which provides basic vehicle management functions, started at contract award and was completed in 2007. The block 0.5 software which provides the initial mission system architecture and sensor infrastructure and enables communications, navigation, and limited radar functionality was released to flight test in April, 2010. Software problems occurring before and during flight test were not resolved in the block 0.5 configuration and the Program s leadership deemed block 0.5 unsuitable for initial training. Fixes were deferred to the initial release of block 1.0. The first block of the Mission Systems software to be installed on the operational aircraft is block 1.0. Aircraft equipped with the block 1.0 software will be fully flight-qualified with baseline air-to-air capability and air-to-ground war fighting capabilities. Release of the block 1.0 software will allow the services to start training their pilots and their maintenance personnel. 9 The Operational Requirements Document is still in effect for the F-35. The JCIDS process has been updated and the ORD has been replaced by such JCIDS documents as the ICD, CDD, and CPD. 8
The Mission Systems block 2.0 software, which builds upon the previous blocks of software, will allow the services to start planning deployments using the F-35. The capability for close air support and interdiction roles will be added along with the qualification of weapons used to perform those missions. Block 2.0 was scheduled to be the first block that would undergo Operational Test & Evaluation (OT&E) which was scheduled for the end of 2011. The block 3.0 software, which builds upon the previous blocks of software, adds additional capabilities (Suppression of Enemy Air Defenses) and weapons to the F-35. Completion of block 3.0 would allow the Services to declare Initial Operational Capability (IOC) for their specific variant of the F-35. Flight testing of block 3 was scheduled to last three years, followed by OT&E which was expected to be completed the third quarter of 2013. The number of source lines of codes (SLOC) for each block and the total number of source lines of code has varied over time due to unclear reporting, inaccurate estimates and required changes to the software due to the addition of new requirements and defect cleanup. The SLOC counts reported by the Government Accountability Office (GAO) often do not identify whether it is the total SLOC count for the F-35 or for just the on board or off board systems of just the Mission Systems. The total number of SLOCs estimated in 2006 was 15,315K with 4,879K identified as Mission Systems software. However, the total number did not include the re-hosting of the onboard software into the off board training system software. More recent estimates, which includes the re-hosting number, places the total number of SLOCs at slightly more than 25,000K SLOCs and the Mission Systems at slightly more than 6,271K SLOCs. The F-35A completed its first flight in December 2006 14 months later than originally projected. This was followed by the first flights of the F-35B in June 2008 and the F-35C in June 2010. Flight testing began in 2008, but fell behind schedule because of required modifications to the aircraft and changes to the Vehicle Systems software which, in turn, slowed development of the Mission Systems software. F-35 Mission System Software As described above, the Mission Systems software is being developed, integrated, tested, and delivered to the aircraft in three blocks of capabilities, using an Evolutionary Acquisition (EA) approach, which aligned with the air system block plan (Figure 6). Following air system requirements analysis each major block of the Mission Systems software is decomposed into smaller blocks for design, code, integration test, flight test and verification. Block 1 consisted of Blocks 1A and 1B; Block 2 consists of Blocks 2A and 2B; and Block 3 consists of Blocks 3i and 3F. The software designers would then break each of these smaller blocks into seven or eight builds (e.g., 2BR1, 2BR.1.1, 2BR2, etc.). An estimate of the number of SLOCs in the Mission Systems software is shown in Table 3. The same operational flight program (OFP) is used for all three variants, as the OFP is able to identify the aircraft s variant. 9
Table 3. 2006 estimate of percentage and number of SLOCs per block As LM continued to ramp up production concerns continued to be raised that the concurrent high production rates and relatively slow increase in flight test production would commit the DOD and Services to high risk test, training, and deployment plans. In 2009 the GAO reported the F- 35 s development will cost more and take longer to complete than reported to the Congress a year earlier, primarily because of contract cost overruns and the extended time needed to complete flight testing. In the same report the GAO wrote, The sheer complexity of the JSF program with 22.9 million lines of software code, three variants, and multi-mission development suggests that the aircraft will encounter many unforeseen problems during flight testing requiring additional time in the schedule for rework. Continuing delays and rising costs continued to plague the program increasing the chatter in the media and defense circles about a fighter gap and the future of the F-35. In February, 2010 the Secretary of Defense directed a comprehensive restructuring of the program. Shortly thereafter LM reported the development of all software (onboard and off board) was more than 80 percent complete, but the Mission Systems software was only 56 percent finished and running 6 months behind schedule. In April, as part of the restructuring of the program, developmental flight testing was extended by 13 months, the full rate production decision was moved to April 2016 and additional funding was provided to add software personnel and a third integration laboratory. The number of software-related defects continued to increase on the program, but fixes were often deferred to future blocks as were some planned block capabilities in attempts to meet schedule and mitigate risks (Figure 7). As a result, technical debt rapidly increased for each block and build of software. 10
Figure 7. LM s Mission Capability Block Development Plan Risks associated with concurrency continue to materialize. Completion of SDD had slipped a number of years while the production program had slipped less than a year. However, the only software that could be hosted on the LRIP aircraft was relatively immature 10 and certainly not mission capable. The Services, as part of the SecDef s restructuring of the Program were directed to re-scope their expectations for IOC and what capabilities they would need for IOC as it was certain by now that the original plan of delivering fully mission capable by 2010 was not going to happen. The Marines determined they needed the capabilities associated with Block 2B completed in order to declare IOC while the AF and USN were waiting on the capabilities provided by Block 3i and 3F software, respectively, to be completed before they would declare IOC. In its 2010 report on the F-35 the GAO noted: Currently, JSF engineers have written about three-fourths of the total lines of aircraft code expected and about 40 percent of the written code has been integrated and tested. The first increment, needed to enable basic aircraft flying characteristics, is the only one completed. All other software increments are behind schedule Team officials believe the most complex and troublesome work is still 10 The maturity level of the software based on a number of factors (software problems, functionality, management, configuration management and fixes completed) while being program dependent, is often used a criterion to move forward into testing. A DOT&E memo on the subject can be found here: http://www.dote.osd.mil/pub/policies/1994/19940531sw_maturitycriteria_fordedicatedoteofsw- IntensiveSystems.pdf 11
ahead, that dedicated resources to complete software development and integration are inadequate, and that the contractor will not be able to write code as fast and productively as earlier releases. The joint estimating team also predicts software requirements will continue to grow. We note the JSF software has grown 40 percent since the preliminary design review and 13 percent since the critical design review over initial program estimates. The first production F-35 conducted its first flight in February of 2011 with deliveries of the aircraft beginning that same year. Like other portions of the program the Mission Systems software effort grew in size and complexity, taking longer to complete than was planned (Figures 8 and 9). The number of defects continued to increase with a large number of repairs being deferred to future builds. Restructuring of the program, resulting from the USD(AT&L) s direction and an accompanying Nunn-McCurdy breach in the 2010-2011 timeframe resulted in additional slips to the Air System Block Plan and its associated Software Block Plan. Figure 8. Defect Status Figure 9. Vehicle Systems and Mission Systems source lines of code in thousands. 12
In 2011 the GAO reported: Software providing essential JSF capability is not mature and releases to the test program are behind schedule. Officials underestimated the time and effort needed to develop and integrate the software, substantially contributing to the program s overall cost and schedule problems and testing delays, and requiring the retention of engineers for longer periods. Several blocks have grown in size and taken longer to complete than planned (Figure 10). Delays in developing, integrating, and releasing software to the program have cascading effects hampering flight tests, training, and lab accreditation. While progress is being made, a substantial amount of software work remains before the program can demonstrate full warfighting capability, The program released its second block, or increment 0.5, to flight test nearly 2 years later than the plan set in 2006, largely due to integration problems. Defects and workload bottlenecks delayed the release of full block 1 capabilities; the initial limited release of block 1 software was flown for the first time in November 2010 11. In its annual (2011) Selected Acquisition Report to Congress the JPO s PM wrote: Software risk remains one of the top development issues for the program. Block 1B and 2A development/integration challenges are impacting capacity for delivery of Block 2B. The development program restructure built capacity for discovery in Blocks 2 and 3 development... By the end of 2011, 19 SDD (test) aircraft had been delivered along with 9 aircraft from LRIP lots 1, 2, and 3. Many of these were being used to support ground test and flight operations. 76 additional LRIP aircraft were already in various stages of fabrication and assembly and more were starting to enter the pipeline as Lockheed continued to ramp up its production capabilities toward its goal of 80 aircraft per year by FY17 even though the latest restructuring slipped the full rate production by three years to 2019 and the IOC dates were being reported as TBD. Of the 24 million source lines of code (MSLOCs) written to date approximately 9 MSLOCs were associated with the Air Vehicle software (Vehicle and Mission Systems software) while the remaining 15 MSLOCs supported off-aircraft systems such as the Autonomic Logistics Information System (ALIS), the F-35 s maintenance and support system, and the pilot and maintenance training systems. 12 Lockheed s schedule in January, 2012 is shown in Figure 11. 11 The 0.1, 0.5 and 1.0 blocks referred to by the GAO Vehicle Systems software not the Mission Systems software. 12 The on-aircraft Mission Systems software is re-hosted in the off-aircraft training systems is one reason there are ~15 MSLOC in the off-aircraft systems. 13
Figure 10: Mission Systems software growth KSLOCs, 2006 vs current estimate, by block Mission System Status Check Figure 11: F-35 Software Development Plan a/o January, 2012 Bob stood and stretched, his back cracking as he did so. He had to blink his eyes several times to focus on the clock that hung on the wall. He had been at it for almost four hours. His desk was littered with stacks of program documents, minutes of past meetings, pens of varying colors, numerous highlighters, Styrofoam cups stained with coffee, and a large notepad upon which he had written a number of short notes as he attempted to decipher the status of the mission system software. Next to four of the notes Bob had placed red stars to signify areas of concern. These notes were: Initial Operating Capability LM assumed and planned on SDD being complete before IOC was declared there was no thought about doing any partial capability IOCs to the program the plan was to complete SDD which is now six years longer than originally envisioned and give the Services fully spec compliant aircraft. 14
LM is producing LRIP aircraft that are not and will not be fully spec compliant for sometime How will the Services accept aircraft that are only partially spec compliant? Software Measures & Metrics LM s has an extensive list of SW measures and metrics which are described in the SW Quantitative Management Plan delivered to the JPO. These are reported monthly via the F- 35 s Air Vehicle Software Scorecard which is shared with the JPO The F-35 Air Vehicle Software Scorecard appears to accurately describe the current status of each block of software being delivered per the original plan (fully compliant by the end of Block 3). The JPO has its own set of software metrics. This might be the reason that at yesterday s meeting the LM folks asserted, We are doing great while the JPO response was, No you aren t - here is our evaluation and you are doing terrible. The Services focus is on a select set of capabilities within specific blocks needed to declare IOC (e.g., USMC needs three capabilities in block 2B). LM s build plan lists the capabilities for each build within each block of software, however this appears to be a wandering baseline. LM s monthly build plan shows they will deliver a certain set of capabilities in a specific build (example Capability X). However, the following month LM would come in and say, We had a problem with Capability X so we moved it to the next build. In the following month s build plan Capability X would be listed, but the updated plan didn t show Capability X as being moved the previous month so the software build looks like it is tracking to the current plan even though it had moved from one build to another. Result: What is the baseline? The maturity levels of the code isn t being tracked. LM takes credit for delivery of full capability even though the full capability isn t completed so the JPO gives them zero credit for delivery of that capability. Software Development Schedule All known major events for each Block are identified on the schedule, but they are not horizontally and vertically integrated making it impossible to do a critical path analysis or schedule risk assessment. The validity of the schedule is questionable because LM is moving capabilities, but not tracking the movement of those capabilities Earned Value Data Required and available, but data is questionable as LM s EVMS was decertified In October 2010 by DCMA because of ongoing (since 2007) problems with their business systems Bob was scheduled to meet with front office in the morning to discuss the status of the Mission Systems software. On the surface it appeared LM was getting there, but like other contractors he had been involved with in the past, he wasn t sure when they would complete their software builds. As he rubbed his forehead he kept wondering, What is the status of the software? When will the Services be able to declare IOC? How do I know? 15
Assignment Questions 1. Given what was provided in the case, what is the status of the Mission Systems Software (MSS)? 2. Bob noted there was an Integrated Master Schedule (IMS), but it was not horizontally or vertically integrated. What does it mean to be horizontally and vertically integrated? 3. What software measures and metrics would you have advised the JPO to use at the beginning of the program? Why? Are these the same ones the JPO should be using now? Why? 4. To mitigate risk and recover schedule LM deferred fixing defects to later software blocks as well as deferring select capability sets to future software blocks. One outcome of this approach is a tremendous increase in technical debt. What is technical debt? How would you propose the technical debt be worked off? 5. As Bob, what would you recommend as a way ahead to the front office in the morning? 16