I nspec tor Ge ne ral

Transcription

1 Report No. DODIG I nspec tor Ge ne ral U.S. Department of Defense J U LY 3, Procedures to Ensure Sufficient Rare Earth Elements for the Defense Industrial Base Need Improvement I N T E G R I T Y E F F I C I E N C Y A C C O U N TA B I L I T Y E X C E L L E N C E The document contains information that may be exempt from mandatory disclosure under the Freedom of Information Act.

2 INTEGRITY EFFICIENCY ACCOUNTABILITY EXCELLENCE Mission Our mission is to provide independent, relevant, and timely oversight of the Department of Defense that: supports the warfighter; promotes accountability, integrity, and efficiency; advises the Secretary of Defense and Congress; and informs the public. Vision Our vision is to be a model oversight organization in the federal government by leading change, speaking truth, and promoting excellence; a diverse organization, working together as one professional team, recognized as leaders in our field. Fraud, Waste & Abuse HOTLINE Department of Defense dodig.mil/hotline For more information about whistleblower protection, please see the inside back cover. DRAFT REPORT

3 Results in Brief Procedures to Ensure Sufficient Rare Earth Elements for the Defense Industrial Base Need Improvement July 3, 2014 Objective We determined whether DoD effectively planned for life-cycle sustainment of rare earth elements (REE) for the defense industrial base (DIB). Specifically, we determined whether DoD effectively implemented procedures to maintain a sufficient and available supply of REEs for the DIB. Finding DoD lacked a comprehensive and reliable process to assess REE supply and demand. Specifically, Defense Logistics Agency, Strategic Materials Division officials did not ensure that its modeling and simulation contractor used: REE supply forecasts that considered market and environmental risks; complete REE demand survey results; and verified economic consumption data to forecast REE demand. This occurred because the Defense Logistics Agency, Strategic Materials Division did not have adequate verification and validation procedures in place to ensure realistic supply and demand inputs and did not require that the contractor use an accredited model to forecast REE supply and demand. As a result, DoD may not have identified all REEs with expected shortfalls, increasing the risk that those shortfalls will adversely affect critical weapons systems production in the DIB, and overall DoD readiness. Recommendations We recommend that the Director, Defense Logistics Agency Strategic Materials Division: develop and implement a verification and validation plan for REE supply and demand forecasting model inputs; develop and implement procedures to ensure that future shortfall analyses compare DoD demand and supply for REEs under the same scenarios; develop and implement procedures for obtaining DoD REE consumption data by leveraging Service acquisition executive participation and other techniques as appropriate; develop and implement an accreditation plan for the forecasting model s intended use; and ensure that current and future contracts for models, simulations and associated data include verification, validation and accreditation procedures in the contract requirements. Management Comments and Our Response The Director, Defense Logistics Agency, Acquisition Directorate generally addressed the recommendations; however, comments on Recommendation 2 partially addressed the recommendation. Therefore, we are requesting additional comments on Recommendation 2 by August 4, see the Recommendation Table on the back of this page. Please Visit us on the web at DRAFT REPORT DODIG (Project No. D2013-D000LA ) i

4 Recommendations Table Management Director, Defense Logistics Agency Strategic Materials Division Please provide comments by August 4, Recommendations Requiring Comment No Additional Comments Required 2 1.a, 1.b, 3, 4, and 5. ii DODIG

5 INSPECTOR GENERAL DEPARTMENT OF DEFENSE 4800 MARK CENTER DRIVE ALEXANDRIA, VIRGINIA July 3, 2014 MEMORANDUM FOR UNDER SECRETARY OF DEFENSE FOR ACQUISITION, TECHNOLOGY, AND LOGISTICS DEPUTY ASSISTANT SECRETARY OF DEFENSE, MANUFACTURING AND INDUSTRIAL BASE POLICY DIRECTOR, DEFENSE LOGISTICS AGENCY SUBJECT: Procedures to Ensure Sufficient Rare Earth Elements for the Defense Industrial Base Need Improvement (DODIG ) We are providing this report for review and comment. DoD lacked a comprehensive and reliable approach to assess rare earth element supply and demand, and identify shortfalls that could impact the defense industrial base. We considered management comments on a draft of this report when preparing the final report. DoD Directive requires that recommendations be resolved promptly. The Director, Defense Logistics Agency, Acquisition Directorate, responded to the draft report and generally agreed with the recommendations; however, comments on Recommendation 2 partially addressed the recommendation. Therefore we are requesting additional comments on Recommendation 2 by August 4, Please send a PDF file containing your comments to audrco@dodig.mil. Copies of your comments must have the actual signature of the authorizing official for your organization. We cannot accept the /Signed/ symbol in place of the actual signature. If you arrange to send classified comments electronically, you must send them over the SECRET Internet Protocol Router Network (SIPRNET). We appreciate the courtesies extended to the staff. (703) (DSN ). Please direct questions to me at Carol N. Gorman Assistant Inspector General Readiness and Cyber Operations DODIG iii

6 Contents Introduction Objective 1 Background 1 Organizations With Roles or Responsibilities Related to REEs 4 Review of Internal Controls 5 Finding. REE Supply and Demand Assessment Process Not Comprehensive or Reliable 6 Critical Defense Materials Assessment Process 6 Market and Environmental Risks Were Not Considered in Supply Forecasts 7 Incomplete Survey Results Accepted 9 Unverified Consumption Data Used to Forecast Demand 10 Inadequate Procedures and Unaccredited Model Used to Forecast Supply and Demand 11 Shortfalls Could Adversely Affect DoD Readiness 12 Ongoing Management Initiatives 13 Recommendations, Management Comments, and Our Response 14 Appendixes Appendix A. Scope and Methodology 18 Use of Computer Processed Data 19 Use of Technical Assistance 20 Prior Coverage 20 Appendix B. Rare Earth Elements 21 Appendix C. Organizations With Roles or Responsibilities Related to REEs 22 Appendix D. Quantitative Methods Division Assessment of the RAMF SM Model 25 Management Comments Defense Logistics Agency 27 Acronyms and Abbreviations 30 iv DODIG

7 Introduction Introduction Objective Our audit objective was to determine whether DoD effectively planned for life cycle sustainment of rare earth elements (REE) for the defense industrial base (DIB). 1 Specifically, we determined whether DoD effectively implemented procedures to maintain a sufficient and available supply of REEs for the DIB. See Appendix A for a discussion of the scope and methodology and for prior audit coverage related to the objectives. Background REEs consist of 17 elements 15 that form the chemical series called the lanthanides, and 2 others scandium and yttrium that tend to occur in the same ore deposits and exhibit similar chemical properties as the lanthanides. Eight lanthanides with atomic numbers 57 through 64 are often defined as light REEs. Seven lanthanides with atomic numbers 65 through 71 and one non lanthanide, yttrium, with similar physical and chemical properties, are defined as heavy REEs. The other non lanthanide, scandium, does not exhibit sufficient similar physical and chemical properties to be defined as either a light or heavy REE. Although denoted as rare, REEs are relatively abundant in total quantity worldwide. However, REE deposits are seldom found in sufficient amounts to be extracted and processed economically (see Appendix B for a list of the 17 elements and their defense uses). REE Commercial and Defense Applications According to the Rare Earth Technology Alliance, REEs are vital to many modern technologies, including consumer electronics, computers and networks, communications, clean energy, advanced transportation, health care, environmental mitigation, and national defense. Because of their unique magnetic and electrochemical properties, REEs help DoD weapons systems perform with reduced weight and energy consumption; or give them greater efficiency, performance, miniaturization, durability, and thermal stability. 1 The DIB is the portion of the industrial complex responsible for the design, production, delivery, and maintenance of military weapons systems, subsystems, and components or parts that fulfill U.S. military requirements. DODIG

8 Introduction Joint Direct Attack Munition Source: For example, dysprosium and neodymium are used in the targeting capabilities of the Joint Direct Attack Munition. The Joint Direct Attack Munition is a low cost guidance kit that converts existing unguided dumb bombs into accurately guided, near precision, smart weapons. The munition s tail fin assembly control motor actuators contain neodymium iron boron magnets that direct the bomb precisely to its target. Dysprosium is added to enhance the ability of the magnets to maintain their magnetic properties at high temperatures. REE Supply Chains In general, the REE supply chain consists of ore mining, separating the ore into individual rare earth oxides (REOs), refining REOs into metals, forming the metals into alloys, and manufacturing the alloys into end use items. REE supply chains are spread around the world, and may have more than 10 steps from ore mining to final end use items. From the mid 1960s through the 1980s, the United States was the world s leader in REO production. However, since that time, China took advantage of lower labor costs and lower environmental standards to develop and expand its REO mining, refining, and manufacturing supply chain capabilities. According to the Government Accountability Office, 97 percent of the world s REO production came from China in By contrast, U.S. REO production and other supply chain capabilities had diminished to the point where it was virtually 100 percent dependent on REE imports. Congressional Directives to Address DoD REE Availability In FY 2011, Congress took action to address REE availability concerns, given China s dominance in REE supply chain capabilities. Specifically, in Public Law , The Ike Skelton National Defense Authorization Act for Fiscal Year 2011, Section 843, Assessment and Plan for Critical Rare Earth 2 DODIG

9 Introduction Materials in Defense Applications, January 7, 2011, Congress directed DoD to assess the supply and demand for REEs in defense applications and identify REEs that met both of the following criteria: the REE is critical to the production, sustainment, or operation of significant U.S. military equipment; and the REE is subject to interruption of supply, based on actions or events outside the control of the U.S. Government. For REEs that met the criteria, Congress directed DoD to develop a plan that would ensure a U.S. supply by December 31, Congress also directed that the plan include consideration of risk mitigation methods for those specific REEs, including stockpiling, substitution, and industry subsidies. In Public Law , The National Defense Authorization Act for Fiscal Year 2012, Section 853, Assessment of Feasibility and Advisability of Establishment of Rare Earth Material Inventory, December 31, 2011, Congress required DoD to assess the feasibility and advisability of establishing a REE inventory to ensure their long term availability. House of Representatives Conference Report, HR , December 12, 2011, accompanied Public Law , and required DoD to submit a report on the feasibility and desirability of recycling, recovering, and reprocessing REEs from fluorescent lighting in DoD facilities, batteries, certain magnets used in weapons systems, and commercial off the shelf items such as computer hard drives. DoD Response to Congressional Directives Addressing REE Availability In response to the FY 2011 and FY 2012 National Defense Authorization Acts, the Office of the Secretary of Defense submitted the following reports: Report to Congress on Rare Earth Materials in Defense Applications, March 2012 (843 Report), stated that by 2013, U.S. REE production could satisfy the level of consumption required to meet DoD procurement needs for six of the seven REEs that met the two congressionally mandated assessment criteria, 2 the single exception being yttrium. DoD also stated that it would pursue a three pronged 2 The seven REEs that met the criteria were dysprosium, erbium, europium, gadolinium, neodymium, praseodymium, and yttrium. DODIG

10 Introduction approach to addressing REE availability: diversification of supply, pursuit of substitutes, and reclamation of waste. Report to Congress on Assessment of Feasibility and Advisability of Establishment of Rare Earth Material Inventory, September 2012 (853 Report), stated that DoD did not identify rare earth material shortfalls at the mining and oxide production level. 3 However, gaps in the manufacturing supply chains for certain REEs prompted DoD to recommend solutions to address potential supply vulnerabilities for two REEs ultra pure yttrium oxide and dysprosium metal. Report on Feasibility and Desirability of Recycling, Recovery, and Reprocessing Rare Earth Elements, September 2012, stated that the recycling of linear fluorescent lighting (yttrium, europium, terbium) and nickel metal hydride batteries (neodymium) was technically feasible and desirable. The Defense Logistics Agency, Strategic Materials Division (DLA SM) was responsible for preparing assessments that supported the reports. For the 843 and 853 Reports, DLA SM used the same assessment process it used to support routine biennial reports to Congress covering strategic and critical materials, but expanded its coverage to include additional REEs. For example, in the report, Strategic and Critical Materials 2013 Report on Stockpile Requirements, January 2013, DLA SM included 16 REEs in their evaluation of 76 materials to determine whether the materials would exhibit shortfalls in the context of a congressionally mandated conflict scenario. As a result of DLA SM s work in supporting the 2013 report, DoD recommended that Congress authorize the FY 2014 acquisition of specific materials to mitigate the shortfalls. In the FY 2014 National Defense Authorization Act, Congress authorized the acquisition of dysprosium and yttrium. Organizations With Roles or Responsibilities Related to REEs Several DoD, U.S. Government, and private organizations have roles or responsibilities related to the assessment or production of REEs, including the Under Secretary of Defense for Acquisition, Technology and Logistics (USD[AT&L]); the Office of 3 The shortfall of yttrium identified in the 843 Report was not identified at the mining and oxide production level because supply sources were expanded to include foreign supply. 4 DODIG

11 Introduction the Deputy Assistant Secretary of Defense for Manufacturing and Industrial Base Policy (DASD[MIBP]); DLA SM; DLA SM s modeling and simulation contractor, the Institute for Defense Analyses (IDA); the U.S. Geological Survey (USGS); and Molycorp, Inc. (Molycorp). See Appendix C for more detailed information about these organizations. Review of Internal Controls DoD Instruction , Managers Internal Control Program Procedures, May 30, 2013, requires DoD organizations to implement a comprehensive system of internal controls that provides reasonable assurance that programs are operating as intended and to evaluate the effectiveness of the controls. We identified internal control weaknesses related to the assessment of REE supply and demand for defense applications. Specifically, DoD lacked a comprehensive and reliable approach to assess REE supply and demand. We will provide a copy of the report to the senior official responsible for internal controls in the Defense Logistics Agency. DODIG

12 Finding Finding REE Supply and Demand Assessment Process Not Comprehensive or Reliable DoD lacked a comprehensive and reliable process to assess REE supply and demand. Specifically, DLA SM officials did not ensure that its modeling and simulation contractor used: REE supply forecasts that considered market and environmental risks; complete REE demand survey results; and verified economic consumption data to forecast REE demand. This occurred because DLA SM did not have adequate verification and validation procedures in place to ensure realistic supply and demand inputs and did not require that the contractor use an accredited forecasting model to forecast REE supply and demand. As a result, DoD may not have identified all REEs with expected shortfalls, increasing the risk that those shortfalls will adversely affect critical weapons systems production in the DIB, and overall DoD readiness. Critical Defense Materials Assessment Process The Strategic and Critical Material Stock Piling Act (section 98, title 50, United States Code), as amended through Public Law , The National Defense Authorization Act for Fiscal Year 2013, January 2, 2013, requires the Secretary of Defense to submit to Congress, on a biennial basis, a report on stockpile requirements for strategic and critical materials. Specifically, the Secretary is to recommend what strategic and critical materials should be stockpiled by the DoD to preclude a costly and dangerous dependence upon foreign sources of supply of such items. Critical materials requirements are based on national emergency planning assumptions of a military conflict scenario, consistent with the scenario used by the Secretary of Defense for budgeting and defense planning purposes. USD(AT&L), as the National Defense Stockpile Manager, is the signatory authority for congressional reporting on strategic and critical materials. DLA SM, as the executive agent of the National Defense Stockpile Manager, performs critical 6 DODIG

13 Finding materials assessments in support of the biennial requirements reports. To support the FY 2011 and 2013 requirements reports, and the 843 and 853 Reports, DLA SM used the following process to assess critical materials: identify materials from a watch list for comprehensive assessment; estimate DoD supply and demand for identified materials to determine shortfalls; assess supply chains for impact on identified materials; identify and prioritize mitigation options for materials determined to have shortfalls; and recommend the most promising mitigation solutions. DLA SM contracted with IDA to perform analyses in support of these assessments, including IDA s use of various econometric 4 and scenario models in what it called the Risk Assessment and Mitigation Framework for Strategic Materials (RAMF SM) to estimate DoD supply, demand, and shortfalls for critical materials, including REEs. However, according to IDA, there was considerable uncertainty attached to REE shortfalls because available data on REE demand is inconsistent and REE supply is dynamic. Therefore, it is critical for DLA SM to ensure that IDA incorporates procedures to mitigate uncertainties with the supply and demand data IDA used in the RAMF SM to determine REE shortfalls. Market and Environmental Risks Were Not Considered in Supply Forecasts DLA SM s contractor, IDA, did not adequately assess market and environmental risks that could adversely affect REE production data forecasts used to estimate future REE supplies. DoD Instruction , DoD Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A), December 9, 2009, requires DoD Components to ensure that data used in models and simulations the DoD Component develops or modifies are validated throughout their lifecycle. The Instruction defines data validation as the process of validating that data accurately represents real world conditions. IDA based future REE supply estimates primarily on forecasts made by a U.S. producer, Molycorp. According to its 2010 Annual Report, Molycorp stated it was the only known U.S. producer and holder of REEs and forecasted that it would achieve 4 Econometrics is the use of statistical and mathematical models to quantify economic activity. DODIG

14 Finding full production capacities of REOs at its mining and separation facility in Mountain Pass, California, in two phases: Phase I 19,050 metric tons (MTs) per year by the end of 2012; and Phase II 40,000 MTs per year by the end of (FOUO) Molycorp also stated in its 2010 Annual Report that multiple real world market, environmental, and other risks could have a material impact on the company s financial condition or operations. For example, Molycorp stated that the actual funding required to expand and modernize its Mountain Pass facility might vary materially from estimates and that additional funding may not be available. As a result, delays in completing these improvements would have a material adverse effect on business and financial conditions, which in turn could adversely affect production capacity. According to the 843 Report working paper, IDA acknowledged supply risks associated with Molycorp s Mountain Pass REE output projections and the mid and long term viability of the Mountain Pass operation. However, IDA estimated REE production capacities based on the 2010 forecasts without considering the potential impact of these risks on REE supply availability. DLA SM expected that DoD would acquire REE supply inventories under the Defense Production Act of in the event of an emergency and allowed the DLA SM... allowed the use of production capacity versus actual production or other realistic means when estimating future REE supplies. use of production capacity versus actual production or other realistic means when estimating future REE supplies. The Defense Production Act allows for the use of production capacity when evaluating materials the President may consider diverting from ordinary use to national defense purposes, but best practices dictate that production capacity estimates should be realistic. Realistic estimates would consider the developmental status of the Molycorp production capacity and the volatility of the REE market. Therefore, DLA SM personnel should have ensured that IDA not only identified, but also analyzed, mitigated, and tracked the risks that could adversely affect production capacity forecasts for REOs. The Risk Management Guide for DoD Acquisition, Sixth Edition, August 2006, provides best practices 5 Public Law , codified with amendments at section 2061, title 50, United States Code. The Defense Production Act authorizes the President to divert certain materials and facilities from ordinary use to national defense purposes when national defense needs cannot otherwise be satisfied in a timely fashion. It also states that to ensure adequate production capacity and supply, Components responsible for defense acquisition should continually evaluate the availability of adequate production sources, including materials. 8 DODIG

15 Finding for assessing risks identified by decision makers. This applies to the RAMF SM model because shortfalls identified could be mitigated through the acquisition of REEs. The guide identifies the need for a realistic, achievable, and measurable risk management plan to address the root cause or risk to be considered in decision making. DLA SM should ensure its contractors incorporate practices in the Risk Management Guide when assessing production data subject to uncertainty. In addition, DLA SM should use probabilistic tools as part of risk analysis and risk mitigation procedures to account for supply data uncertainties. (FOUO) In addition, in supporting the 843 Report, IDA compared REE supply estimates based on production capacity (based on the Defense Production Act) to REE demand estimates based on Fiscal Years Defense Plan (peacetime) demand. This comparison resulted in underestimated REE shortfalls because IDA compared contingency supply to peacetime demand. DLA SM should ensure that future shortfall analyses compare DoD demand and supply for REEs using the same scenarios. Incomplete Survey Results Accepted DLA SM personnel did not adequately verify the completeness and consistency of REE demand data it received from the Services and provided to IDA. DoD Instruction requires DoD Components to ensure that data used in models and simulations are verified throughout their lifecycle. The Instruction defines data verification as the process of verifying the internal consistency and correctness of data. As part of its assessment of DoD REE demand, DLA SM surveyed the Services on the types and quantities of critical REEs they use in various defense system applications. In that survey, DLA SM requested that the Services focus on their top three systems for which the REEs were critical to production, sustainment, or operation of the system. However, the Services responses to nine questions covering specific REEs and their quantities were incomplete, or inconsistently reported. For example, the Navy s survey response to a question requesting both the number and the quantities of REEs in a defense system only listed the REEs but not the quantities. DLA SM provided the survey results to IDA and neither DLA SM nor IDA adequately followed up with the Services to obtain more accurate or complete data. DODIG

16 Finding DLA SM accepted incomplete REE survey results because it anticipated complete data would be too difficult or costly to retrieve. DLA SM personnel stated they used the best available REE usage data and that they were limited in obtaining more data on DoD REE usage because: the Services did not have sufficiently detailed information readily available concerning REE usage within their specific defense systems; changes in DoD acquisition strategy no longer require contractors to provide details on what materials are included in the weapon system components so the Services would have to pay contractors for this data; contractors in the REE supply chain are reluctant to provide what they see as proprietary data; and they did not have the resources (personnel, funding, and time) to obtain the REE usage for every weapon system. However, without adequate follow-up, DLA SM could not be sure which, if any, of these limitations applied to the inaccurate, incomplete, or inconsistent data reported. Implementing better methods, such as improving coordination with Service acquisition executives, or elevating data requests to the USD(AT&L) level, will help DLA SM to obtain more complete, consistent, and relevant DoD REE demand data for weapons systems. Unverified Consumption Data Used to Forecast Demand (FOUO) IDA did not verify the economic consumption data it used in the RAMF SM to estimate DoD REE demand, as required by DoD Instruction For example, IDA used USGS s estimate of total U.S. REE consumption by individual REEs 19,610 MTs in 2010 as the basis for estimating DoD REE demand in support of the 843 Report and to forecast DoD REE procurement demand for the years (see table below). (FOUO) Table. REE Procurement Demand Estimates* 2011 to 2015 (in MTs) (FOUO) (FOUO) (FOUO) (FOUO) (FOUO) *Estimates are RAMF SM model outputs for DoD REE consumption for purposes of procurement. (FOUO) 10 DODIG

17 Finding However, IDA did not verify the internal consistency and correctness of the USGS provided data. When asked, USGS could not provide information supporting the methodology used to calculate the individual REE U.S. consumption amounts for A USGS representative stated that the analyst who calculated DoD REE consumption for 2010 left the agency, but the representative stated that generally, an analyst uses professional judgment and reviews information obtained from several sources to determine the individual REE consumption amounts. Additionally, DLA SM did not ensure that IDA requested supporting documentation for USGS REE consumption data. DLA SM contracted with USGS to provide REE economic consumption data in support of DoD s biennial reporting requirement to Congress on strategic and critical materials requirements. According to DLA SM and USGS officials, the economic consumption data was to be given directly to IDA for use in the RAMF SM model. Although USGS is a widely recognized authority in matters pertaining to minerals, this did not preclude DLA SM or IDA from obtaining support to verify USGS data and the methodology used to develop the data. DLA SM should develop and implement procedures to verify consistency and correctness of the data used to estimate DoD REE consumption. Inadequate Procedures and Unaccredited Model Used to Forecast Supply and Demand As stated in the previous sections, DLA SM needs to improve its procedures for verifying and validating supply and demand inputs that are used in the RAMF SM model. Additionally, DLA SM should require the modeling and simulation contractor to use an accredited model to identify REE shortfalls. DoD Instruction states that the Defense Logistics Agency, as a DoD Component Head, shall assign responsibilities to ensure that models, simulations, and their associated data that are used by DLA are accredited for their intended use. According to the Instruction, accreditation is the official certification that a model or federation of models 6 and associated data are acceptable for a specific purpose. The accreditation process includes identifying model assumptions, capabilities, limitations, risks, and impacts. The process also includes an assessment that focuses on how well the model meets acceptability criteria. 7 The assessment includes the qualitative and quantitative metrics used to measure the 6 7 Military Standard 3022, Department of Defense Standard Practice: Documentation of Verification, Validation, and Accreditation (VV&A) for Models and Simulations, January 28, 2008, defines a federation of models and simulations as a system of interacting models, simulations, and a supporting infrastructure that are based on a common understanding of the objects portrayed in the system. The acceptability criteria are a set of standards that a model will meet to be accredited for a specific purpose. DODIG

18 Finding DLA-SM did not accredit the RAMF-SM for its intended purpose. acceptability criteria success. This assessment forms the basis for the accreditation recommendation forwarded to the accreditation authority. However, DLA SM did not accredit the RAMF SM for its intended purpose to assess supply and demand of REE to identify REE shortfalls, and prioritize and recommend REE mitigation actions. Specifically, DLA SM could not provide documentation that an accreditation assessment was performed, accreditation authority was assigned, and accreditation decision was obtained for the RAMF SM. To ensure that an accredited model is used, DLA SM needs to include accreditation requirements in the contract for the RAMF SM model. Military Standard 3022, Department of Defense Standard Practice: Documentation of Verification, Validation, and Accreditation (VV&A) for Models and Simulations, January 28, 2008, states that DoD Components may cite the standard as a contractual requirement in contracts, requiring the documentation of procedures to support the accreditation, verification, and validation of models and associated data. DLA SM contracted with IDA to develop processes for the RAMF SM to assess DoD REE supply and demand, identify REE shortfalls, and provide mitigation options. However, DLA SM did not ensure that IDA obtained accreditation for the RAMF SM, as required. DLA SM personnel stated that they were unaware of DoD requirements for the accreditation of models and associated data, but that some of the models used in the RAMF SM were included in the Defense Modeling and Simulation Coordination Office (M&SCO) catalog of models. However, according to the DoD M&SCO, inclusion in the catalog does not mean that required accreditation procedures for models were completed. DLA SM should incorporate requirements for accreditation of the RAMF SM model and associated data for its intended use into the contract with IDA. Additionally, future modeling and simulation contracts should include accreditation requirements to ensure that modeling results are useable for their intended purpose. Shortfalls Could Adversely Affect DoD Readiness DoD may not have identified all REEs with expected shortfalls, increasing the risk that those shortfalls will adversely affect critical weapons systems production in the DIB, and overall DoD readiness. Because DLA SM relied upon overstated REE supply quantities and incomplete and unverified REE 12 DODIG

19 Finding demand data in an unaccredited model to assess the supply and demand for REEs, it may have underestimated the number of individual REEs requiring mitigation action to ensure their availability for weapons systems production. Erroneous shortfall estimates may adversely affect REE mitigation actions. For example, USD(AT&L) used the results of IDA s analysis to support the 843 Report to Congress. In that report, DoD stated that only yttrium would have an estimated shortfall for In another example, based on IDA s analysis in support of the 2013 requirements report, Congress authorized the FY 2014 acquisition of quantities of dysprosium and yttrium to mitigate shortfalls in the event of an emergency. Given the deficiencies identified in this report, decisions made based on IDA s analyses, DLA SM conclusions resulting from those analyses, and USD(AT&L) s reporting; mitigation actions may be inappropriate or ineffective to meet all REE shortfalls. Ongoing Management Initiatives DLA SM has completed various studies, analyses, and other actions as part of its plan to ensure a source of supply for REEs in critical defense applications by December 31, Specifically, DLA SM is working independently or with DASD(MIBP) to: develop a National Defense Stockpile Management Plan for all stockpile reports, including the biennial requirements reports and other congressional reports; develop, for FY 2014 implementation, a web based system 9 to allow users to visually map REE supply chains and identify gaps or other issues with them; expand the RAMF SM to include steps for assessing shortfall risks, identifying and prioritizing mitigation options, and recommending REE acquisitions; and contract with various academic, governmental, and industry components to assess REE issues across entire supply chains, including potential domestic sources, new processing technologies, and geographically economical supply chains. 8 9 Although IDA s analysis also identified a shortfall in another critical REE erbium USD(AT&L) incorrectly reported a surplus in the 843 Report. Erbium is used in high performance fiber optics communications systems. The Strategic Materials Analysis and Reporting Topography system. DODIG

20 Finding While these actions may aid in ensuring a source of supply for REEs, DLA SM needs to implement procedures to ensure that reliable supply and demand data are used in the RAMF SM to determine REE shortfalls. In addition, DLA SM needs to ensure that the RAMF SM is accredited for its intended use. Recommendations, Management Comments, and Our Response We recommend that the Director, Defense Logistics Agency, Strategic Materials Division, develop and implement: 1. A verification and validation plan for Risk Assessment and Mitigation Framework for Strategic Material REE data inputs, in accordance with DoD requirements. The plan should include procedures to: a. Analyze, mitigate, and track risks that could adversely affect REE supply data inputs, through the use of probabilistic tools in the analysis and mitigation procedure steps. Defense Logistics Agency Comments The Director, Defense Logistics Agency, Acquisition Directorate agreed, stating that by September 30, 2014, DLA will complete a verification, validation, and accreditation action plan with appropriate milestones for formal verification, validation, and accreditation plans and reports. To that end, DLA will meet with the DoD M&SCO to discuss recommendations and guidance for the verification, validation, and accreditation of future DLA studies and reports, which will include the use of probabilistic tools in the analysis and mitigation steps. Our Response Although comments from the Director did not specifically address implementation of a verification and validation plan for RAMF SM REE data inputs, the Director s comments addressed the universe of future DLA studies and reports and satisfied the intent of the recommendation. No further comments are required. 14 DODIG

21 Finding b. Verify the consistency and correctness of REE consumption data inputs. Defense Logistics Agency Comments The Director, Defense Logistics Agency, Acquisition Directorate agreed, stating that DLA will work with the USGS to develop and implement procedures to verify the quality of available data. DLA has contacted USGS to verify the consistency and correctness of REE data inputs to be used in future DLA studies and reports. The Director stated that DLA will incorporate these procedures into the verification, validation, and accreditation plan described in Recommendation 1.a. Our Response Comments from the Director fully addressed the recommendation. comments are required. No further 2. Procedures to ensure that future shortfall analyses compare DoD demand and supply for REEs under the same scenarios. Defense Logistics Agency Comments (FOUO) The Director, Defense Logistics Agency, Acquisition Directorate partially agreed with our recommendation. The Director agreed that demand and supply analysis should follow consistent procedures and adhere to the same scenarios to provide the most accurate assessment of the REE supply chain. However, the Director stated that DLA will continue to compare demand and supply for REEs under the same scenarios by recognizing that the Defense Priorities and Allocation System (DPAS) can be invoked for production capacity during both peacetime and conflict conditions. He also stated that approximately 300,000 DoD contracts and purchase orders annually include DPAS ratings that can be invoked to ensure that these orders receive priority acceptance and performance, and therefore it was appropriate to use estimates of rare earth production capacity to estimate supply available to the Department during both peacetime conditions and conflict contingencies. Our Response Although the Director agreed that demand and supply analysis should follow consistent procedures, his comments did not address the market risks discussed in the report. For example, if the DPAS is invoked, the supplier must reject orders they cannot fill by the date specified. If the capacity-based production quantities DODIG

22 Finding are not realistic, as discussed previously, there is a greater likelihood that the orders could be rejected and the need not filled. Additionally, if DLA uses a scenario where supply quantities are increased because the DPAS can be invoked; for comparison purposes, DLA should also use the same scenario for the demand quantities increasing the demand to a level that would require the DPAS to be invoked, not standard peacetime demand quantities. We request that the Director, Defense Logistics Agency, Acquisition, provide additional information that shows how the current methodology considers market risks and produces realistic, achievable production levels to meet REE demand. 3. Procedures for obtaining DoD REE consumption data leveraging Service acquisition executive participation or other techniques as appropriate. Defense Logistics Agency Comments The Director, Defense Logistics Agency, Acquisition Directorate agreed with our recommendation, stating DLA will develop procedures for obtaining DoD REE consumption data and will request that any lack of responses be elevated to the USD(AT&L) to obtain Service acquisition executive participation by September 30, The Director stated that these procedures will be presented at the next Strategic Materials Protection Board. The Director noted that even a complete set of survey results for all major DoD systems would not be able to provide DLA with the comprehensive and robust set of demand estimates that the RAMF SM demand estimation methodology provides. Specifically, he stated that REE content in final systems would not include raw material lost in the manufacturing of a system or material required by the DIB for use in production facilities to manufacture defense systems. Our Response Comments from the Director fully addressed the recommendation. comments are required. No further 4. An accreditation plan for the Risk Assessment and Mitigation Framework for Strategic Materials to ensure it is appropriate for its intended use, in accordance with DoD requirements for modeling and simulation and associated data. 16 DODIG

23 Finding Defense Logistics Agency Comments The Director, Defense Logistics Agency, Acquisition Directorate agreed with our recommendation, stating that as indicated in his comments to Recommendation 1, DLA will develop plans for verification, validation, and accreditation based on guidance from the DoD M&SCO and an understanding of USD(AT&L) s expectations. DLA will complete a verification, validation, and accreditation action plan for its future studies and reports, including the RAMF SM, by September 30, Our Response Comments from the Director fully addressed the recommendation. comments are required. No further 5. Procedures to ensure that current and future contracts for models, simulations and associated data include verification, validation and accreditation procedures in the contract requirements. Defense Logistics Agency Comments The Director, Defense Logistics Agency, Acquisition Directorate agreed with our recommendation, stating that based on guidance from the DoD M&SCO and expectations of USD(AT&L), DLA will incorporate verification, validation, and accreditation procedures in future contracts involving modeling, simulation, and associated data, by September 30, Our Response Comments from the Director fully addressed the recommendation. comments are required. No further DODIG

24 Appendixes Appendix A Scope and Methodology We conducted this performance audit from May 2013 through May 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We researched internal and external web based sources to identify DoD directives, instructions, manuals, and procedures related to our audit objective, and to obtain a general understanding of REEs. We also researched the FYs 2010, 2011, 2012, and 2013 National Defense Authorization Acts, specifically sections related to REEs, as a basis for determining our audit scope, and to identify key players at the organizational level. We interviewed officials from USD(AT&L), Office of the Secretary of Defense Cost Assessment and Program Evaluation, DASD(MIPB), DLA SM, USGS, Oak Ridge National Laboratory, IDA, and Molycorp to ascertain roles and responsibilities and obtain an understanding of the DoD REE demand and supply assessment process, including the RAMF SM model. We also interviewed officials from the DoD M&SCO to understand DoD modeling and simulation verification, validation, and accreditation processes. We reviewed agreements between DLA SM and IDA, USGS, Department of Commerce, and Department of Energy to ascertain their relationships in assessing DoD REE supply and demand. During initial interviews with officials involved with REEs, and subsequent review of the RAMF SM flowchart, we determined that the RAMF SM process, with multiple models used to provide estimates for DoD REE demand and supply, was in fact, a model, or federation of models, subject to DoD guidance for models and simulations. 18 DODIG

25 Appendixes Therefore, we determined whether DLA-SM adequately complied with DoD Instruction for modeling and simulation, specifically, the verification, validation, and accreditation of the RAMF SM model and its associated data. We also obtained an IDA working paper used by DLA SM to identify REE shortfalls for the DIB, and report to Congress in the 843 Report. Our review of the 843 Report working papers included our analyses of: total U.S. REE consumption provided by USGS and used by IDA as the basis for estimating DoD REE demand; domestic REE supply forecasts (at the REO level) provided by USGS, but based primarily on Molycorp s 2010 forecasts of REO production capability; and REE demand survey data responses from the Army, Navy, Marines, and Joint Staff on REE quantities in certain DoD weapons systems. The DoD Office of Inspector General Quantitative Methods Division s assessment confirmed the deficiency we identified in the model s ability to provide a range of values for DoD REE demand and supply. We developed our own estimate of future REE supplies, using Molycorp s 2013 forecast of REO production capability to identify the potential impact of using unrealistic REE supply forecasts on REE shortfalls. We limited our analysis of the REE supply chain to domestic production of REOs. This was because of section 843 s criteria that imply an extreme supply interruption scenario where DoD would have to rely solely on domestic sources of REEs. In addition, DLA SM was still refining the shortfall identification process and developing additional steps in the RAMF SM to identify and prioritize mitigation options for REEs that DLA SM identified as having shortfalls. Use of Computer Processed Data We attempted to assess the reliability of the computer-processed data supporting estimated REE supply, demand, and shortfalls. We determined that the process and methodology that DLA SM used to obtain data inputs for the RAMF SM model was not sufficient to support the reliability of the data outputs. We discuss this process in the report finding. DODIG

26 Appendixes Use of Technical Assistance The DoD Office of Inspector General Quantitative Methods Division assisted with this audit. Specifically, Quantitative Methods Division personnel reviewed audit documents; participated in audit team meetings to interview representatives from DLA SM, DASD(MIBP), and IDA; prepared a technical assessment of the RAMF SM; and advised the audit team on DoD modeling and simulation guidance applicability to the audit objective. See Appendix D for the technical assessment of the RAMF SM model. Prior Coverage During the last 5 years, the Government Accountability Office (GAO) and the Congressional Research Service (CRS) have issued nine reports discussing REEs. Unrestricted GAO reports can be accessed at GAO GAO Report No. GAO R, Rare Earth Materials in the Defense Supply Chain, April 14, 2010 Congressional Research Service The CRS is a legislative branch agency of the Library of Congress that conducts research and analysis for Congress on a broad range of issues of national policy. The CRS issued a series of reports highlighting REE related issues and their consequences for U.S. national defense. Five CRS Reports, all numbered R41744 and titled, Rare Earth Elements in National Defense: Background, Oversight Issues, and Options for Congress, dated December 23, 2013, September 17, 2013, September 5, 2012, April 11, 2012, and March 31, 2011 Two CRS Reports, both numbered R41347 and titled, Rare Earth Elements: The Global Supply Chain, June 8, 2012, and September 30, 2010 CRS Report No. R42510, China s Rare Earth Industry and Export Regime: Economic and Trade Implications for the United States, April 30, DODIG

27 Appendixes Appendix B Rare Earth Elements REEs include the following: Rare Earth Element: Cerium Dysprosium Erbium Europium Gadolinium Holmium Lanthanum Lutetium Neodymium Praseodymium Promethium Samarium Scandium Terbium Thulium Ytterbium Yttrium Defense Uses Semiconductors and electron tubes, storage batteries, glass products, motor vehicle parts Nuclear control rods, magnets, ceramics for electronics Communications, energy wires and cables, nonferrous (non-iron) metal products, semiconductors and electron tubes Nuclear control rods, lasers, phosphors for lighting and displays Computer storage devices, semiconductors and electron tubes, electro medical apparatus, magnetic and optical recording devices Electronic components, semiconductors and electron tubes, other fabricated metal products Primary ferrous (iron) metal products, petroleum refineries, storage batteries Electrometrical apparatus, communications, energy wires and cables, semiconductors and electron tubes Magnets, lasers, capacitors Pigment, ceramics, fiber optics, medical imaging, alloying agent Compact fluorescent lamps and thickness gauges Neutron absorber for nuclear reactors, lasers, magnets, capacitors Electric lamp bulbs and parts, petroleum refineries, semiconductors and electron tubes, other aircraft parts and equipment Lasers, phosphors for lighting and displays, magnets, magnet or restrictive alloys Semiconductors and electron tubes, other electronic components, irradiation apparatus, wiring devices Communications, energy wires and cables, semiconductors and electron tubes, primary ferrous (iron) metal products, irradiation apparatus Displays and lighting DODIG

28 Appendixes Appendix C Organizations With Roles or Responsibilities Related to REEs Under Secretary of Defense for Acquisition, Technology and Logistics USD(AT&L) is the principal staff assistant and advisor to the Secretary of Defense and Deputy Secretary Defense for all matters concerning acquisition, technology, and logistics. USD(AT&L) s primary responsibilities include: supervising DoD acquisition; establishing policies for acquisition, developmental testing, and contract administration for all DoD elements; establishing policies for logistics, maintenance, and sustainment support for all DoD elements; and establishing policies for maintenance of the DIB. The Secretary of Defense tasked USD(AT&L) to submit to Congress a report on the supply and demand for rare earth materials in defense applications and the Senate Report requested discussion of national security issues related to rare earth materials in the supply chain. Deputy Assistant Secretary of Defense for Manufacturing and Industrial Base Policy DASD(MIBP) establishes policies and procedures for maintenance of the DIB and provides recommendations on supply chain management and supply chain vulnerabilities to USD(AT&L). DASD(MIBP) also carries out activities related to the Defense Production Act Committee established under the Defense Production Act of Public Law added responsibility for the DASD(MIBP) to ensure reliable sources of materials critical to national security, including REEs. Defense Logistics Agency, Strategic Materials Division USD(AT&L) designated DLA SM as the lead agency in preparing the DoD response to Congress on REEs. DLA SM is the executive agent of the Stockpile Manager, USD(AT&L). DLA SM s statutory mission is to implement the provisions of the Strategic and Critical Materials Stock Piling Act, which includes decreasing, and precluding when possible, a dangerous and costly dependency by the United States on foreign or single sources for supplies of materials in times of national emergencies. USD(AT&L) tasked DLA SM to assess the DoD supply and demand for REEs, identify shortfalls for critical DoD REEs, and recommend mitigation measures to address the shortfalls. 22 DODIG

29 Appendixes Institute for Defense Analyses IDA is a federally funded research and development center that analyzes national security issues, particularly those requiring scientific and technical expertise, and conducts related research on other national challenges. DLA SM contracted with IDA to design and implement strategic materials assessment processes for biennial National Defense Stockpile requirements reporting to Congress, and for responding to congressional concerns over REEs. Specifically, IDA developed and used the RAMF SM to assess the DoD supply and demand for REEs, identify DoD REE shortfalls, and recommend mitigation measures to address the shortfalls. (FOUO) The RAMF SM is a federation of several models. Specifically, IDA uses the following models in the RAMF SM to estimate DoD REE consumption: (FOUO) Long Term Inter Industry Forecasting Tool: Macroeconomic model of the U.S. economy that uses Council of Economic Advisors forecast data to forecast supply and demand for 97 industries. (FOUO) Inter Industry Large Scale Integrated and Dynamic Model: Model that expands Long Term Inter Industry Forecasting Tool model output data to develop a forecast of supply and demand for 360 industrial sectors, specifically, generating end user civilian, regular military, export, and import demand, in dollars. (FOUO) Input/Output Post Processing: Model used to determine dollar amounts of output from each of the 360 industry sectors. (FOUO) Forces Mobilization Model: Scenario model that develops weapon requirements needed in a conflict scenario and computes the industry output (in dollars) required to produce them. (FOUO) Material Demand Computation Model: The Material Demand Computation Model uses historical consumption values from either the Department of Commerce or USGS, and data outputs from the Inter Industry Large Scale Integrated and Dynamic and Forces Mobilization models as data inputs to determine the quantities of material needed to produce the industrial output requirements. IDA can modify the Material Demand Computation Model to provide quantities of material by application area, or other means. DODIG

30 Appendixes (FOUO) IDA uses the Stockpile Sizing Model to estimate REE supplies available and compare them with REE demands to identify shortfalls. For purposes of this report, we refer to the entire federation of models as the RAMF SM model. U. S. Geological Survey USGS collects, monitors, analyzes, and provides scientific understanding about natural resource conditions, issues, and problems. USGS carries out large scale, multi disciplinary investigations and provides impartial scientific information to resource managers, planners, and other customers. DLA SM contracted with USGS to provide REE data directly to the IDA in support of DoD s biennial reporting requirement to Congress on strategic and critical materials requirements. Molycorp, Inc. Molycorp is a U.S. based manufacturer of custom engineered rare earth and rare metal products. It develops rare earth technology and products vital to clean energy, high tech, and Defense applications and produces concentrated REOs from both light and heavy rare earth minerals. Molycorp is the largest holder of rare earth deposits outside of China. In 2008, Molycorp acquired the mining and separation facility in Mountain Pass, California, the only source of REEs in the United States, and in 2009, began limited REO production there. 24 DODIG

31 Appendixes Appendix D Quantitative Methods Division Assessment of the RAMF SM Model During the audit, the DoD OIG Quantitative Methods Division, reviewed audit documents and accompanied the audit team to briefings by DLA SM and IDA on the RAMF SM model. risks based on the estimated supplies and demands. IDA used the model to assess the REE shortfall According to Quantitative Methods Division personnel, specific quantitative measures were not needed to accomplish the audit objectives; however, in their overall analysis and interpretation, they made the following observations related to the RAMF SM model: The inputs the model used were from different sources generated by econometric models, and do not take into consideration the data variability of the input sources. The output results were generated by using a model that is deterministic and provides no measure of variability in forecast. Forecast accuracy should be measured. Forecast error, forecasted versus actual, should be tracked and used to improve the model. A stochastic modeling 10 approach could be used to generate forecast estimates with prediction intervals, or quantify the probability of the forecast estimate. Forecasts needed to be tracked and variances from actual used to measure and improve forecast accuracy. Data inputs from experts should be managed to track and assess the validity and reliability of their input. Methodologies to collect and assimilate information provided by experts should be considered. 10 A model in which the results are determined by using one or more random variables to represent uncertainty about a process or in which a given input will produce an output according to some statistical distribution. DODIG

32 Appendixes A combination of forecasting techniques, such as econometric, stochastic, and simulation-based approaches, should be considered. Collection of expert opinion should be refined and quantified to the extent possible using communication techniques, such as Delphi. 11 Surveys and Delphi, if incorporated, should be web-based. 11 The Delphi method is a defined process that uses a panel of experts in a subject area to improve understanding of an area of interest. 26 DODIG

33 Management Comments Management Comments Defense Logistics Agency DODIG

34 Management Comments Defense Logistics Agency (cont d) 28 DODIG

35 Management Comments Defense Logistics Agency (cont d) DODIG

36 Acronyms and Abbreviations Acronyms and Abbreviations DASD(MIBP) Deputy Assistant Secretary of Defense, Manufacturing and Industrial Base Policy DIB Defense Industrial Base DLA SM Defense Logistics Agency, Strategic Materials Division DPAS Defense Priorities and Allocation System IDA Institute for Defense Analyses MT Metric Ton M&SCO Modeling and Simulation Coordination Office RAMF SM Risk Assessment and Mitigation Framework for Strategic Materials REE Rare Earth Element REO Rare Earth Oxide USD(AT&L) Under Secretary of Defense, Acquisition, Technology and Logistics USGS U.S. Geological Survey 30 DODIG

37 Whistleblower Protection U.S. Department of Defense The Whistleblower Protection Enhancement Act of 2012 requires the Inspector General to designate a Whistleblower Protection Ombudsman to educate agency employees about prohibitions on retaliation, and rights and remedies against retaliation for protected disclosures. The designated ombudsman is the DoD Hotline Director. For more information on your rights and remedies against retaliation, go to the Whistleblower webpage at For more information about DoD IG reports or activities, please contact us: Congressional Liaison Congressional@dodig.mil; Media Contact Public.Affairs@dodig.mil; Monthly Update dodigconnect-request@listserve.com Reports Mailing List dodig_report@listserve.com Twitter twitter.com/dod_ig DoD Hotline dodig.mil/hotline DRAFT REPORT

38 DEPARTMENT OF DEFENSE INSPECTOR GENERAL 4800 Mark Center Drive Alexandria, VA Defense Hotline