The International Space Station: Operating an Outpost in the New Frontier Page 179 (213 of 438)

179 DAY IN THE LIFE: PREPARING FOR THE UNEXPECTED CHAPTER 10 italicized information within the square brackets helps explain or clarify what is going on or what is meant by a particular acronym, word, or statement, but it is not part of the actual transcript. CRONUS: FLIGHT, CRONUS, for the IAC. [This is CRONUS calling the flight director about the previously failed IAC. Since the FLIGHT loop is the key loop, its name generally does not need to be used.] FLIGHT: CRONUS, FLIGHT. [The flight director is acknowledging the call and indicating that she is ready to talk to the CRONUS operator.] CRONUS: Yes, FLIGHT, so when we saw the IAC failure, we were initially on IAC 2. Audio FDIR [Failure Detection Isolation Recovery, an automated recovery algorithm, see Chapter 5] swapped us over to IAC 1. That swap was not successful, and then it brought us back up on IAC 2. So in order to clean up and recover voice, I ran Ground Avionics procedure 2.311, just blocks 1 and 2 which basically covers inhibiting Audio FDIR and reconfiguring [voice loop configuration] calls. I’d like to continue to press through that procedure. Normally you’d run this procedure to power up the failed IAC and check it out but since we started up on IAC 2 and it looks healthy, since we are back on IAC 2 now, I’d like to actually power up IAC 1, which is the one we weren’t able to recover on, and see if I can see any issues with that. FLIGHT: Ok, I concur, you’re go. CRONUS: Copy. [Shorthand for “I hear and understand you.”] SPARTAN: FLIGHT, SPARTAN, for status. FLIGHT: SPARTAN, FLIGHT. [i.e., “Go ahead, I’m listening.”] SPARTAN: FLIGHT, the pump is back up and running. At this point I am ready to re-integrate the interface heat exchangers [the automated response from an external pump failure is to isolate the external ammonia loop from the heat exchangers that transport heat from the internal water cooling loops in the various modules to the external ammonia loops], beginning with Node 2, Node 3, IPs [referring to the Japanese and European ISS modules], then Lab. FLIGHT: I’m sorry, say again, pump is running, then what? SPARTAN: The loop Bravo pump is up and running, at this point I am ready to re-integrate the interface heat exchangers, beginning with Node 2, going to Node 3, then the IPs, then the Lab. ETHOS: And FLIGHT, ETHOS, I copy, and whenever the heat exchangers are re-integrated a lot of times we get a little bit of overshoot, and there’s a potential for some undertemp [i.e., too cool] messages on the board but that should level off pretty quickly. There’ll be no action [by crew or ground] for those once we are integrated. FLIGHT: Copy. [Shortly after SPARTAN completed his commanding, a caution-and-warning message was displayed to the ground and the crew about an undertemp in the TCS] ETHOS: And FLIGHT, ETHOS that enabled caution was what I was talking about, no action for the crew. CAPCOM [on the S/G-1 loop]: Station, Houston, on 1, no action for the TCS caution. That was expected. ISS CREW [on the S/G-1 loop]: Copy Houston, no action for the TCS caution, thanks. The People Behind The Curtain The simulator—basically a series of computers that can emulate the behavior of ISS systems and the space environment—is a powerful tool. When a student is in the training control room, the data on his or her computer screen or console will look exactly like it would if it were the actual space station in orbit. SPARTAN, for example, can watch the solar arrays rotate as sunlight is converted to electricity and routed around the ISS until the station orbits into the Earth’s shadow and the batteries begin supplying all the power needed. With the flick of a wrist, the simulation team can fail a bus or converter. However, the key to the simulation is the training team that operates it. This team is led by the Chief Training Officer (CTO), who is essentially the flight director of the training world. Scenarios developed by the training team depend on the type of simulation being conducted. As the name implies, generic simulations focus on general skills of the team—i.e., communication, coordination, and problem resolution. In these types of simulations, the training team will induce a failure that impacts multiple systems. For example, a power bus may fail, which can affect every other group. Due to the robust redundancy of most ISS systems, these types of failures usually result in the flight control team learning how to reconfigure the operating systems (e.g., activating the redundant unit), troubleshoot the failed one, and recover the failed system, if possible (e.g., reboot the computer if that is the issue). At all times, the team must try to keep the planned events

Previous Page Next Page

Purchased by unknown, nofirst nolast
From: Scampersandbox (scampersandbox.tizrapublisher.com)

Extracted Text (may have errors)

179 DAY IN THE LIFE: PREPARING FOR THE UNEXPECTED CHAPTER 10 italicized information within the square brackets helps explain or clarify what is going on or what is meant by a particular acronym, word, or statement, but it is not part of the actual transcript. CRONUS: FLIGHT, CRONUS, for the IAC. [This is CRONUS calling the flight director about the previously failed IAC. Since the FLIGHT loop is the key loop, its name generally does not need to be used.] FLIGHT: CRONUS, FLIGHT. [The flight director is acknowledging the call and indicating that she is ready to talk to the CRONUS operator.] CRONUS: Yes, FLIGHT, so when we saw the IAC failure, we were initially on IAC 2. Audio FDIR [Failure Detection Isolation Recovery, an automated recovery algorithm, see Chapter 5] swapped us over to IAC 1. That swap was not successful, and then it brought us back up on IAC 2. So in order to clean up and recover voice, I ran Ground Avionics procedure 2.311, just blocks 1 and 2 which basically covers inhibiting Audio FDIR and reconfiguring [voice loop configuration] calls. I’d like to continue to press through that procedure. Normally you’d run this procedure to power up the failed IAC and check it out but since we started up on IAC 2 and it looks healthy, since we are back on IAC 2 now, I’d like to actually power up IAC 1, which is the one we weren’t able to recover on, and see if I can see any issues with that. FLIGHT: Ok, I concur, you’re go. CRONUS: Copy. [Shorthand for “I hear and understand you.”] SPARTAN: FLIGHT, SPARTAN, for status. FLIGHT: SPARTAN, FLIGHT. [i.e., “Go ahead, I’m listening.”] SPARTAN: FLIGHT, the pump is back up and running. At this point I am ready to re-integrate the interface heat exchangers [the automated response from an external pump failure is to isolate the external ammonia loop from the heat exchangers that transport heat from the internal water cooling loops in the various modules to the external ammonia loops], beginning with Node 2, Node 3, IPs [referring to the Japanese and European ISS modules], then Lab. FLIGHT: I’m sorry, say again, pump is running, then what? SPARTAN: The loop Bravo pump is up and running, at this point I am ready to re-integrate the interface heat exchangers, beginning with Node 2, going to Node 3, then the IPs, then the Lab. ETHOS: And FLIGHT, ETHOS, I copy, and whenever the heat exchangers are re-integrated a lot of times we get a little bit of overshoot, and there’s a potential for some undertemp [i.e., too cool] messages on the board but that should level off pretty quickly. There’ll be no action [by crew or ground] for those once we are integrated. FLIGHT: Copy. [Shortly after SPARTAN completed his commanding, a caution-and-warning message was displayed to the ground and the crew about an undertemp in the TCS] ETHOS: And FLIGHT, ETHOS that enabled caution was what I was talking about, no action for the crew. CAPCOM [on the S/G-1 loop]: Station, Houston, on 1, no action for the TCS caution. That was expected. ISS CREW [on the S/G-1 loop]: Copy Houston, no action for the TCS caution, thanks. The People Behind The Curtain The simulator—basically a series of computers that can emulate the behavior of ISS systems and the space environment—is a powerful tool. When a student is in the training control room, the data on his or her computer screen or console will look exactly like it would if it were the actual space station in orbit. SPARTAN, for example, can watch the solar arrays rotate as sunlight is converted to electricity and routed around the ISS until the station orbits into the Earth’s shadow and the batteries begin supplying all the power needed. With the flick of a wrist, the simulation team can fail a bus or converter. However, the key to the simulation is the training team that operates it. This team is led by the Chief Training Officer (CTO), who is essentially the flight director of the training world. Scenarios developed by the training team depend on the type of simulation being conducted. As the name implies, generic simulations focus on general skills of the team—i.e., communication, coordination, and problem resolution. In these types of simulations, the training team will induce a failure that impacts multiple systems. For example, a power bus may fail, which can affect every other group. Due to the robust redundancy of most ISS systems, these types of failures usually result in the flight control team learning how to reconfigure the operating systems (e.g., activating the redundant unit), troubleshoot the failed one, and recover the failed system, if possible (e.g., reboot the computer if that is the issue). At all times, the team must try to keep the planned events

Help

CHAPTER 10 DAY IN THE LIFE: PREPARING FOR THE UNEXPECTED 178 and one pump—either the Moderate Temperature (MT) or the Low Temperature (LT)—pushes fluid around the entire system, which is called Single MT or Single LT, respectively. This is significant because, at the time of the failure, the US Lab internal active Thermal Control System (TCS) was in Single LT mode, meaning one water pump on the LTL was performing all of the heat transport of cooling water to US Lab systems. The LTL pump remains running in the event of an LA-1 MDM failure as several key pump performance parameters are being reported to and controlled by a different MDM. However, loop pressure, flow, temperatures and, most importantly, pump accumulator quantities, are lost. The pump accumulator quantity is especially important because a sudden decrease in accumulator quantity indicates that cooling water is leaking out of the system. If the leak is not stopped, enough water will be lost to cause the cooling loop to fail and critical equipment to overheat. A sudden increase in accumulator quantity indicates that another fluid is being injected into the system. Given the way in which the internal cooling loop is plumbed, the only possible fluid that could be injected into the system is the 100% anhydrous ammonia that is used as a coolant for ISS external systems. The ammonia and water come into thermal contact to exchange heat, but are kept separate from each other at the IFHX (see Chapter 11). This grade of ammonia is fatal to humans, even in small quantities. Also, one of two redundant Internal Audio Controllers (IACs), which route all on-board voices to different destinations, failed prior to the simulation start in what is known as an initial condition of the simulation. Another initial condition was that the External TCS (ETCS) Loop B experienced a transient failure, which shut the pump down. Communications One of the most critical skills that a flight control team needs to exercise is communication. Owing to the complicated, often time-critical nature of spaceflight, a specific shorthand and cadence was developed to facilitate communication between the various controllers, the flight director, and the crew. Flight controllers in Mission Control communicate over loops, which is nothing but a dedicated phone line, so to speak, between parties. Typically, each person must listen to more than a dozen of these voice loops at the same time. Using a headset, a flight controller plugs into the system to hear the calls that are defined on a computer panel. There are literally thousands of loops from which to choose, but each person usually monitors only a small subset. Conversations can occur simultaneously on all the loops during busy times. Sometimes the flight controller needs to follow along with only a few, but at other times he or she may need to directly participate in the discussion. Learning to process all these simultaneous conversations is a key skill the flight controllers have to master. Calls between controllers begin with the name of the person being called, followed by the position that is making the call. For example, “SPARTAN, CRONUS on SYS COORD” indicates that the SPARTAN controller is calling Communications Rf Onboard Network Utilization Specialist (CRONUS) on the voice loop called SYS COORD, which stands for Systems Coordination. When CRONUS hears this, he or she will talk directly to SPARTAN. The most critical loop is the FLIGHT loop, which is owned by the flight director. In a real sense, this is the “king” of loops and everyone in every control center around the globe always has to monitor this critical loop. The only other special loop is the Space- to-Ground (S/G) loop. Only the CAPCOM talks to the astronauts on this loop. Since an astronaut’s time is so critical and communication can be limited, everyone is required to stop talking during a call from the CAPCOM to the astronauts, or vice versa. This prevents the ground from talking over and therefore missing an important item, or making it necessary for the crew members to repeat themselves. A key part of this communication training is to learn how to talk concisely. Unlike an office meeting where employees can take all the time necessary, discussions in Mission Control are usually time critical. If the discussions are not time critical, they are still kept to a minimum since everyone is monitoring multiple loops. Another way to keep communications crisp is to add a brief phrase to explain why the person is calling. This is indicated by stating “for” followed by what the call is about, thus allowing the receiver to prioritize and prepare for the discussion. Following is an example of a simple exchange during that simulation. NOTE: The transcript has been edited for readability. The

Previous Page Next Page

Purchased by unknown, nofirst nolast
From: Scampersandbox (scampersandbox.tizrapublisher.com)

CHAPTER 10 DAY IN THE LIFE: PREPARING FOR THE UNEXPECTED 180 (e.g., the spacewalk) continuing as scheduled. Flight-specific simulations are focused on a particular upcoming event wherein the actual team members who will execute the event train together. Generic simulations are populated with people at various levels of training. Personal photo courtesy of Robert Dempsey Figure 3. The training team, led by CTO William Frank (green shirt) in their own Mission Control-like facility. A computer generates the fake ISS data that the flight controllers see in Mission Control. The training team is also looking at these fake data. However, unlike the flight controllers, the instructors can inject failures into the simulation at the stroke a mouse click. For example, they can make a pump suddenly overheat and fail, or make a computer start generating erroneous messages. Failure scenarios can become fairly complicated. One or two system failures may not be much of an impact to the operations however, they can combine to cause significant constraints or vulnerabilities to an additional failure. For example, consider a module that has two internal thermal control loops and two power strings: A and B. Thermal pump A is on power string A thermal pump B is on power string B. If power string A fails, string B is used. Now, thermal pump B becomes more critical. Should anything happen with the thermal loop (e.g., the pump fails) or if power string B is interrupted, all cooling to that module will be lost. In this type of scenario, the team will try to anticipate the next-worse failure after the first failure. Thus, when string A goes down, the team will try to anticipate what to do to best protect the vehicle for the next failure. In this example, the team might get a power jumper ready to reactivate pump A by plugging it into the B system. The training team spends a significant amount of time learning the systems, the timeline, and the objectives in preparation for a simulation. This research may involve going into the simulator, testing failures, and seeing how the software responds. Not only does this help lead to a realistic simulation, on many occasion real software bugs have been caught before being loaded onto the actual vehicle. Several weeks in advance of a simulation, the CTO will lead development of a script for the run. The script defines what failures will occur, and when and what the training team anticipates the flight control team will do in response. Sometimes, however, the flight control team will make a decision, either in error or on purpose, and choose a different response than anticipated. In this case, the training team will conduct

Previous Page Next Page

Purchased by unknown, nofirst nolast
From: Scampersandbox (scampersandbox.tizrapublisher.com)

Page iFront Matter click to expand contents

Page 379Appendix click to expand contents

Extracted Text (may have errors)

Help

loading