BY ORDER OF THE COMMANDER 50TH SPACE WING 50TH SPACE WING INSTRUCTION 33-101 29 SEPTEMBER 2017 Communications and Information AFSCN COMMUNICATIONS OUTAGE RESOLUTION COMPLIANCE WITH THIS PUBLICATION IS MANDATORY ACCESSIBILITY: Publications and forms are available on the e-publishing website at www.e Publishing.af.mil for downloading or ordering. RELEASABILITY: There are no releasability restrictions on this publication. OPR: 21SOPS/ANCC Supersedes: 50SWI 33-101, 19 August 2010 Certified by: 21SOPS/ANC (Col W. Scott Angerman) Pages: 11 This publication implements Air Force Technical Manual 00-33A-1001, General Communications Activities Management Procedures and Practice Requirements. This document provides guidance for the Air Force Satellite Control Network (AFSCN) on resolving outage conditions. It addresses specific resolution procedures required to troubleshoot, restore, and properly manage the Operational Switch Replacement (OSR) that encompasses all AFSCN communications. This publication applies to all activities and organizations, including AFRC and ANG units, supporting and using AFSCN controlled resources. Contractor personnel requirements and responsibilities contained within this instruction may be modified for a specific purpose through a validated Performance Work Statement (PWS). The PWS must indicate specific paragraphs that are being modified. This publication may be supplemented at any level, but all supplements must be routed to the Office of Primary Responsibility (OPR) listed above for coordination prior to certification and approval. Refer recommended changes and questions about this publication to the OPR listed above using the AF Form 847, Recommendation for Change of Publication; route AF Forms 847 from the field through the appropriate chain of command. The authorities to waive wing/unit level requirements in this publication are identified with a Tier ("T-0, T-1, T-2, and T-3") number following the compliance statement. See AFI 33-360, Publications and Forms Management, Table 1.1 for a description of the authorities associated with the Tier numbers. Submit requests for waivers through the chain of command to the appropriate Tier waiver approval authority, or alternately, to the Publication OPR for nontiered compliance items. Ensure that all records created as a result of processes prescribed in this publication are maintained in accordance with Air Force Manual (AFMAN) 33-363, Management of Records, and disposed of in accordance with Air Force Records Information
2 50SWI 33-101 29 SEPTEMBER 2017 Management System (AFRIMS) Records Disposition Schedule (RDS). The use of the name or mark of any specific manufacturer, commercial product, commodity, or service in this publication does not imply endorsement by the Air Force. SUMMARY OF CHANGES Changes incorporated into this document include: moving the second paragraph of the summary to the overview; removing any references to specific carrier agencies, equipment models, and communications types; adding the use of the Schriever Conference Bridge and direction thereof; adding references to 50SWI10-224, AFSCN Outage Reporting System; updating work centers and contact numbers; and correcting minor spelling errors and some language usage. 1. Overview.... 2 2. Organizations.... 3 Table 1. Organizations and Call Signs.... 3 3. Coordination Procedures.... 3 4. OSR Tech Net.... 4 5. Responsibilities.... 4 6. Restoration Checkouts.... 5 Attachment 1 GLOSSARY OF REFERENCES AND SUPPORTING INFORMATION 6 Attachment 2 COORDINATION PROCESURES 7 1. Overview. 1.1. The 22nd Space Operations Squadron (22 SOPS) is responsible for overall AFSCN operations. The 50th Space Communications Squadron (50 SCS) is responsible for the overall operations and maintenance of the AFSCN communication segment. The 50 SCS will be responsible for the day-to-day operations and maintenance of the SAFB Operational Control Node (OCN) as well as fault detection/fault isolation (FD/FI) of those resources. The AFSCN Network Control Center (ANCC) is responsible for operations and maintenance of VAFB OCN and FD/FI of all AFSCN communication outages that affects both OCNs. 1.2. OSR is a dynamic communication environment that uses inter-dependent and redundant communications systems and links. Due to this network implementation, user coordination for FD/FI and fault recovery activities between the OCNs and Remote Tracking Stations (RTS) is of paramount importance. The primary source for identifying anomalies is the ANCC. The OSR Distributed Communications Controller (DCC) and the Enterprise Management (EM) workstations will identify problems and indicate communication system problems, notably failure to establish connections and loss of network connectivity.
50SWI 33-101 29 SEPTEMBER 2017 3 1.3. What may have been an appropriate action in the past can be problematic in OSR. OSR network implementation has built-in redundancy and depending on the problem detected, it may still be providing full operational capabilities, despite fault indications. Immediate action may not always be warranted. 2. Organizations. 2.1. The various agencies, facilities and work centers involved in outage resolution are described in Table 1. 2.2. Many agencies and work centers coordinate efforts on shared voice nets or voice conferences. Call-signs are used to help distinguish between organizations on a net. The organizational descriptions in Table 1 include call signs used by those agencies. Table 1. Organizations and Call Signs. Organization Work Center Call-sign 50 SCS Comm Focal Point Comm Focal Point Tech Control Facility Schriever Tech 21 SOPS ANCC Comm Control 21 SOPS Crypto Operations Dice Comm. 21 SOPS AFSCN Site Administrator (also part of IMOC) Dice Admin. 21 SOPS Vandenberg Tracking Station Cook 21 SOPS, Det 1 Diego Garcia Tracking Station Reef 21 SOPS, Det 2 Guam Tracking Station Guam 21 SOPS, Det 3 Hawaii Tracking Station Hula 22 SOPS AFSCN Network Crew Commander Crew Commander TBD Colorado Testing Station Pike 23 SOPS New Hampshire Tracking Station Boss 23 SOPS, Det 1 Thule Tracking Station Pogo 23 SOPS, OL-B Eastern Vehicle Checkout Facility Beach 23 SOPS, OL-A Oak hanger Telemetry and Command Station Lion 3. Coordination Procedures. To ensure all actions that could affect AFSCN communications are properly coordinated, there are specific coordination procedures that must be followed. These coordination procedures are included in Attachment 2. NOTE: Many coordination procedures are used during the resolution of communication outages, but some coordination procedures will occur on a regular basis as a part of preventative maintenance, scheduled downtimes, and crypto operations. These coordination procedures must be followed to prevent any communication outages. 3.1. All coordination procedures follow these basic precepts: 3.1.1. Any action that will affect the normal operation of OSR communication equipment requires prior coordination with 22 SOPS, 50 SCS, and 21 SOPS. 3.1.2. All those involved in an OSR communication effort will meet on the OSR Tech Net before taking any action. 3.1.2.1. If the OSR Tech Net becomes unavailable, all parties will make all efforts to coordinate with the applicable parties through whatever means are available.
4 50SWI 33-101 29 SEPTEMBER 2017 3.1.3. If any anomaly or outage condition occurs during a routine crypto operation, administrative change, or maintenance effort, all further actions will be suspended while 50 SCS and ANCC investigate the cause of problem. 4. OSR Tech Net. 4.1. The OSR Tech Net is a conference that facilitates coordination between agencies while troubleshooting is in progress. It is established on the VAFB Conference Bridge. 4.1.1. 30 SW tenant performs routine maintenance on the VAFB Conference Bridge located in building 12006. 4.2. ANCC will continuously monitor the Tech Net when connected with the VAFB Conference Bridge and will oversee, assist and coordinate in any technical effort. 4.3. AFSCN communication work centers have direct access to the OSR Tech Net via administrative phones. When needed, ANCC will contact the work center to have the area access the OSR Tech Net. 4.4. 50 SCS and 21 SOPS will request AFSCN work centers and users to join the OSR Tech Net as needed to aid troubleshooting efforts. 4.5. All agencies and work centers can access the respective OSR Tech Nets by dialing directly into the SAFB Conference Bridge at CMCL (719) 567-1117 or DSN 560-1117, and entering the conference with 2666# (C-0-M-M-#), or by dialing into the VAFB Conference Bridge at CMCL (805) 606-9960 or DSN 276-9960, and entering the conference with 2666# (C-0-M-M-#) 4.5.1. Should ANCC not be readily available on the OSR Tech Net, immediately contact ANCC directly at CMCL (805) 605-3840 or DSN 275-3840. 4.5.2. ANCC will ensure net discipline is maintained on the OSR Tech Net by all parties during the troubleshooting effort. 5. Responsibilities. 5.1. AFSCN Users: 5.1.1. Contact 22 SOPS NCMDR immediately upon identification of an outage condition. Failures are to be reported as soon as possible, but no later than 15 minutes after occurrence or outage declaration. 5.1.2. Note anomalies during sorties and report them to 22 SOPS NCMDR within 15 minutes for tracking and analysis. 5.2. 50th Space Communications Squadron Technical Control Facility (TCF): 5.2.1. Monitors AFSCN communications with SAFB. 5.2.2. Reports communication outage conditions to the ANCC. 5.2.3. Coordinates with ANCC on the OSR Tech Net for any OSR troubleshooting efforts. 5.2.4. Responsible for submitting downtime requests for 501h SCS crypto material changes
50SWI 33-101 29 SEPTEMBER 2017 5 5.3. 21st Space Operations Squadron Network Directors (ANCC): 5.3.1. Monitors AFSCN communications. 5.3.2. Coordinates with TCF on the OSR Tech Net for any SAFB OSR troubleshooting efforts. 5.3.3. Reports AFSCN Outage Conditions IAW 50SWI10-224. 5.3.4. Coordinates with 22 SOPS to obtain any needed windows or downtimes required to perform restoration actions or troubleshooting efforts. 5.3.5. Responsible for submitting downtime requests for VAFB AFSCN Defense Red Switch Network (DRSN) crypto updates. 5.4. 22nd Space Operations Squadron: 5.4.1. Track AFSCN resource status for real-time scheduling. 5.4.2. Ensures up-to-date AFSCN resource status is available to all AFSCN elements. 5.5. Remote Tracking Station Communication Areas: 5.5.1. Contact ANCC immediately upon identification of any Comm. outage condition. Failures are reported as soon as possible, but no later than 15 minutes after occurrence or outage declaration. 5.5.2. Note anomalies during sorties and report them within 15 minutes for tracking and analysis. 5.5.3. Join ANCC and TCF on the OSR Tech Net for any OSR troubleshooting efforts involving that site. 5.5.4. Contact ANCC and TCF prior to performing any maintenance on OSR equipment or crypto updates. 6. Restoration Checkouts. 6.1. After restoration of any AFSCN communications, ANCC and TCF will ensure that all services have been restored to nominal levels prior to closing network outages IAW 50SWI I 0-224. 6.2. Prior to an outage being closed any service affected or potentially affected by a communication outage should be tested to ensure nominal capability is restored. L. GRANT, Colonel, USAF Commander
6 50SWI 33-101 29 SEPTEMBER 2017 References Attachment 1 GLOSSARY OF REFERENCES AND SUPPORTING INFORMATION AFTMAN 00-33A-1001, General Communications Activities Management Procedures and Practice Requirements, 14 July 16 AFMAN 33-363, Management of Records, 1 March 2008 50SWI10-224 AFSCN Outage Reporting System, 24 June 2013 incorporating Change 1, 11 Oct 2016. Abbreviations and Acronyms ANCC AFSCN Network Control Center AFSCN Air Force Satellite Control Network DCC Distributed Communications Controller DET Detachment FD/Fl Fault Detection/Fault Isolation IAW In Accordance With EXU External Users N CMDR - Netwo rk Crew Commander OCN Operational Control Node OSR Operational Switch Replacement SAFB Schriever Air Force Base SCS Space Communications Squadron SOPS Space Operations Squadron TCF Technical Control Facility RTS Remote Tracking Stations VAFB Vandenberg Air Force Base
50SWI 33-101 29 SEPTEMBER 2017 7 Attachment 2 COORDINATION PROCESURES A2.1. Crypto Operations. All crypto operations coordination procedures involve the following four agencies: 21 SOPS Network Director (ANCC), 50SC Sech Control Facility (Schriever Tech), 21 SOPS Crypto Operations (Dice Comm), V A F B D R S N and a Remote Tracking Station communications area (RTS Comm). A2.1.1. Additional Path Aggregate Crypto Material Changes. RTS Comm, Dice Comm and SAFB TCF will initiate this coordination procedure based on the given schedule for crypto material changes. Coordination is required on a situational basis as normal Crypto Material Changes for Firefly Vector Set Keying Material could occur at different dates for different locations. A2.1.1.1. ANCC, Schriever Tech, RTS Comm, will meet on the OSR Tech Net. NOTE: Should any anomalies or outage conditions occur during the crypto material change, RTS Comm will inform ANCC and Schriever Tech. Crypto material changes will be suspended while ANCC and Schriever Tech investigate. A2.1.1.2. RTS Comm will coordinate with Schriever Tech and Dice Comm the required crypto material change as per local procedures. A2.1.1.3. RTS Comm will notify Schriever Tech and/or ANCC when the crypto material change has been accomplished successfully. A2.1.1.4. ANCC and / or Schriever Tech will coordinate the restoration of any required connections that were dropped earlier. A2.1.2. Secure Voice Crypto Material Changes. A RTS w i 11 initiate this coordination procedure based on the given schedule for crypto material changes. A2.1.2.1. Schriever Tech, RTS Comm, and V AFB D R SN will coordinate with RTS Comm to update crypto. NOTE: Should any anomalies or outage conditions occur during the crypto material change, RTS Comm will inform ANCC. Crypto material changes will be suspended while ANCC investigates. A2.1.2.2. RTS Comm will request ANCC coordinate the restoration of the DRSN connections if there is a loss of connectivity. Once connections have been restored, Schriever Tech and/or ANCC will inform RTS Comm that they may proceed. A2.1.2.3. RTS Comm will coordinate with Schriever Tech and Dice Comm the required crypto material change as per local procedures. A2.1.2.4. RTS Comm will notify Schriever Tech and/or ANCC when the crypto material change has been accomplished successfully. A2.1.2.5. ANCC will coordinate restoral of secure voice connections that were dropped for the crypto material change, if required. A2.1.3. Crypto Restarts. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition. Should a routine maintenance action require a crypto restart to take place, ANCC and Schriever Tech will be contacted to coordinate. NOTE: Any
8 50SWI 33-101 29 SEPTEMBER 2017 work center discovering a crypto down will immediately notify ANCC and Schriever Tech. ANCC and Schriever Tech will investigate and direct further action. A2.1.3.1. ANCC, Schriever Tech, RTS Comm and Dice Comm will meet on the OSR Tech Net. A2.1.3.2. ANCC and Schriever Tech will direct the restart of the crypto. A2.1.3.3. ANCC and Schriever Tech will ensure full capability is restored. A2.1.4. Crypto Material Reload. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition. If a routine maintenance action requires a crypto material reload to take place, ANCC and Schriever Tech will be contacted to coordinate the re-load. A2.1.4.1. ANCC, Schriever Tech, RTS Comm and Dice Comm will meet on the OSR Tech Net. A2.1.4.2. ANCC and Schriever Tech will direct the reload and restart of the crypto. A2.1.4.3. ANCC and Schriever Tech will ensure full capability is restored. A2.1.5. Crypto Replacement. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition. Should a routine maintenance action require a crypto replacement to take place, ANCC and Schriever Tech will be contacted to coordinate. A2.1.5.1. ANCC and Schriever Tech will ensure that an AFSCN outage or scheduled A2.1.5.2. ANCC, Schriever Tech, RTS Comm and Dice Comm will meet on the OSR Tech Net. A2.1.5.3. ANCC and Schriever Tech will direct the replacement, reload and restart of the crypto. A2.1.5.4. ANCC and Schriever Tech will ensure full capability is restored. A2.2. Communication Equipment. All communication equipment coordination procedures involve the following agencies: 21 SOPS Network Director (ANCC), 21 SOPS AFSCN Maintenance (Dice Maintenance), 50 SCS Tech Control Facility (Schriever Tech), 50 SCS AFSCN Maintenance (Schriever Maintenance) and a Remote Tracking Station communications area (RTS Comm). A2.3. WANIU Reboot, Power Down and Power Up, or Configuration Reload/Restoration. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition. Should a routine maintenance action require a WANIU power down or configuration change, ANCC and Schriever Tech will be contacted to coordinate. A2.3.1. ANCC, affected RTS Comm(s), and Schriever Tech will meet on the OSR Tech Net. If a VAFB WANIU is affected, Dice Maintenance will join the OSR Tech Net. If a SAFB WANIU is affected, Schriever Maintenance will join the OSR Tech Net.
50SWI 33-101 29 SEPTEMBER 2017 9 A2.3.2. ANCC and Schriever Tech will ensure that an AFSCN outage or scheduled A2.3.3. ANCC and Schriever Tech will direct the reboot, power down, any appropriate maintenance and/or power up of designated WANIU equipment. A2.3.4. ANCC and Schriever Tech will ensure full capability is restored. A2.4. AFSCN Router/Switch Power Down and Power Up or Configuration Reload/ Restoration. ANCC and SAFB Tech will initiate this coordination procedure based on a given outage condition. If a routine maintenance action requires an AFSCN Router/Switch power down or configuration change, ANCC and SAFB Tech will be contacted to coordinate or schedule a downtime with 22 SOPS Range/Plans. A2.4.1. ANCC, affected RTS Comm(s), and Schriever Tech will meet on the OSR Tech Net. If the VAFB AFSCN Router/Switch is affected, Dice Maintenance will join the OSR Tech Net. If a SAFB AFSCN Router/Switch is affected, Schriever Tech may have their circuit actions join the OSR Tech Net. A2.4.2. ANCC and Schriever Tech will ensure that an AFSCN outage or scheduled A2.4.3. ANCC and Schriever Tech will direct the power down, appropriate maintenance actions, and/or power up of affected equipment. A2.4.4. ANCC and Schriever Tech will ensure full capability is restored. A2.5. AFSCN Router/Switch/Server Module Hot Swap. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition. If a routine maintenance action requires an AFSCN Router/Switch module swap, ANCC and Schriever Tech will be contacted to coordinate. A2.5.1. ANCC, affected RTS Comm(s), and Schriever Tech will meet on the OSR Tech Net. If a VAFB module is being swapped, Dice Maintenance will join the OSR Tech Net. If a SAFB module is being swapped, Schriever Tech may have their circuit actions join the OSR Tech Net. A2.5.2. ANCC and Schriever Tech will ensure that an AFSCN outage or scheduled A2.5.3. ANCC and Schriever Tech will ensure the module is free of any traffic. A2.5.4. ANCC and Schriever Tech will direct the module swap. A2.5.5. ANCC and Schriever Tech will ensure full capability is restored. A2.6. Miscellaneous communications equipment Power Down and Power Up or Configuration Reload/Restoration. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition. Should a routine maintenance action require a power down or configuration change, ANCC and Schriever Tech will be contacted to coordinate.
10 50SWI 33-101 29 SEPTEMBER 2017 A2.6.1. ANCC, affected RTS Comm(s), and Schriever Tech will meet on the OSR Tech Net. If VAFB equipment is affected, Dice Maintenance will join the OSR Tech Net. A2.6.2. ANCC and Schriever Tech will ensure that an AFSCN outage or scheduled A2.6.3. ANCC and Schriever Tech will direct the power down, appropriate maintenance actions, and/or power up of affected equipment. A2.6.4. ANCC and Schriever Tech will ensure full capability is restored. A2.7. Miscellaneous communications equipment Card Reset. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition. Should a routine maintenance action require a card reset, ANCC and Schriever Tech will be contacted to coordinate. A2.7.1. ANCC, affected RTS Comm(s), and Schriever Tech will meet on the OSR Tech Net. If an OAFS card is affected, Dice Maintenance will join the OSR Tech Net. A2.7.2. ANCC and Schriever Tech will ensure that an AFSCN outage or scheduled A2.7.3. ANCC and Schriever Tech will direct the card reset. A2.7.4. ANCC and Schriever Tech will ensure full capability is restored. A2.8. Path Procedures. All path procedures involve the following AFSCN agencies: 21 SOPS Network Director (ANCC), 50 SCS Tech Control Facility (Schriever Tech) and any appropriate carrier agency for the communications path. A2.9. Virtual Communications Testing or Downtime. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition or scheduled carrier agency downtime. ANCC and Schriever Tech do this to prevent AFSCN traffic from routing over the virtual communications given over to the carrier agency for testing during an outage or for scheduled maintenance. A2.9.1. ANCC, affected RTS Comm(s), and Schriever Tech will meet on the OSR Tech Net. If VAFB virtual communications is affected, Dice Tech will join the OSR Tech Net. A2.9.2. ANCC and Schriever Tech will ensure the virtual communications is free of traffic. A2.9.3. ANCC and Schriever Tech will direct the shutdown of the virtual interface to prevent any new traffic. A2.9.4. ANCC and Schriever Tech will contact the carrier agency to proceed with testing or downtime. A2.9.5. ANCC and Schriever Tech will direct the restoration of the virtual interface upon return of service from the carrier agency. A2.9.6. ANCC and Schriever Tech will ensure full capability is restored.
50SWI 33-101 29 SEPTEMBER 2017 11 A2.10. Serial Communications Testing or Downtime. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition or scheduled downtime. ANCC and Schriever Tech do this to prevent AFSCN traffic from routing over the serial communications given over to the carrier agency for testing during an outage or for scheduled maintenance. A2.10.1. ANCC, affected RTS Comm(s) and SATCOM(s), and Schriever Tech will meet on the OSR Tech Net. A2.10.2. ANCC and Schriever Tech will ensure the serial communication s is free of traffic. A2.10.3. ANCC and Schriever Tech will contact the carrier agency to proceed with testing or downtime. A2.10.4. ANCC and Schriever Tech will ensure full capability is restored upon return of service from the carrier agency. A2.11. Inter-nodal Communications Testing or Downtime. ANCC and Schriever Tech will initiate this coordination procedure based on a given outage condition or scheduled carrier agency downtime. ANCC and Schriever Tech takes this action to prevent AFSCN traffic from routing over inter-nodal communications given over to the carrier agency for testing during an outage or for scheduled maintenance. A2.11.1. ANCC, Dice Tech and Schriever Tech will meet on the OSR Tech Net. A2.11.2. ANCC and Schriever Tech will ensure inter-nodal communication is free of traffic. A2.11.3. ANCC and Schriever Tech will direct the shutdown of both inter-nodal communication interfaces to prevent any new traffic. A2.11.4. ANCC and Schriever Tech will contact the carrier agency to proceed with testing or their downtime. A2.11.5. ANCC and Schriever Tech will direct the restoration of the interface upon return of service from the carrier agency. A2.11.6. ANCC and Schriever Tech will ensure full capability is restored.