According to measures I have used for many years and the fact that my oldest son is still younger than I was at the time, I consider myself as being just a kid when I drove down to Atlanta to interview for an obscure job I found in the back of the Atlanta Sunday paper. Sure, I was just over a year removed from 4 years in the US Marine Corps where I had chased fighter jets around the globe and into the exotic far reaches of the US military during the height of the cold war, but I was still just a kid. I found the company’s office in a small strip shopping center in an affluent suburb on the north side of the city. At first sight, I considered the office and location more suited for a lawnmower repair shop and remember thinking that this business was not going to support the career objectives I was searching for. I briefly considered backing out of the parking space and passing on the interview.
There was one big room, then occupied by 3 or 4 women doing administrative or accounting-like things and one private office where the interview occurred. The young man that interviewed me was about my age, seemed nice enough and his card said “Office Manager” but, he kept referring to someone else whom I would eventually learn was the owner of this brand new and still very small start-up. The company name was not very innovative, just an acronym for the company’s only product. MOVATS, also known as Motor-Operated Valve Analysis and Test System followed by “Incorporated”, seemingly an afterthought needed to make it sound like an official business. All I really gleaned from the interview was that they did something in nuclear power plants and the growth potential for the business was enormous. A couple of days later, I was called back for a second interview with another guy but still not the owner.
I was promoted to an in-house position a couple of months later as the materials manager where I purchased and staged tools, equipment and construction materials for the many jobs we were working around the North Georgia area. Depending on the workload, we consistently employed 40-50 electricians and an equal number of apprentices or “helpers”. There seemed to always be 25 or more jobs going at the same time and I quickly learned to not expect the job site leaders to plan their projects, ensure they were prepared or tell anyone about what they might need until the exact point in time that they really needed it.
As a consequence, and usually around mid-morning, I would get the daily calls from irate electricians complaining that they had some number of people sitting on their hands because they did not have the tools or equipment necessary to work. It was a very inefficient and needy culture and I had to develop systems and processes in order to get ahead of it. These electricians were all well-seasoned (meaning grumpy old men) and I was the kid, so my solution turned into managing their jobs for them and only having to worry about whether my delivery drivers got there before they finished the morning coffee and pre-job socializing.
I had to decide Rotork or Limitorque and figure out the sizing process in order to get the right actuators in place. Up until mid-way through the second interview, I had not yet made the connection between what is referred to as a motor-operated valve in the nuclear power industry and the electric actuators at the waste treatment plant. As the interviewer was doing his best to describe in simpleton terms what an MOV was, one of those aha moments occurred and I casually responded, “like a Limitorque or Rotork, right?” That being the first time anyone interviewing for the advertised position or any previous position for that matter actually knew what an MOV was beforehand, he sat back with a puzzled look and asked me to explain what I knew about Limitorque and Rotork actuators.
I really did not know much other than they opened and closed valves, came in different sizes and the process required to buy the right one was fairly complex, especially when you have to teach yourself. I also pointed out that immediately after the interview I planned to visit a job site about 3 or 4 miles away where we had just installed a large population of electric actuators. That coincidental tidbit out of the way, he continued to explain that MOVATS is the equipment that the company’s owner came up with to test these actuators in nuclear power plants.
However, I accepted the job offer and reported to work a couple of weeks later. When I arrived on that first day, I seemed to have caught the women in the office and the office manager from the initial interview by surprise. The second interviewer which was now my new boss, the owner and everyone else were out-of-town at far away nuclear power plants, they did not know when anyone would be back and they did not know what I should do.
The office manager told me that there was some MOVATS equipment and an MOV on a test stand out at the calibration lab and suggested I go there. This “calibration lab” turned out to be a small room built in the back of an aircraft hangar at the local municipal airport where the owner kept his plane. There was a guy at that location that maintained the MOVATS equipment but he was quick to point out that he had no knowledge of how it was used.
But, I had still not met the owner. As I learned later, the owner was always on the road, selling, politicking, identifying the problems and getting potential customers to confess their need. At first, he did not really want to sell his product. “Just let us come out and test a few MOVs, maybe just the bad actors and see if we can help” was the approach. Just like a dope pusher trying to bait a youngster into a life of dependence, he was very good at it. But, in this case, the target customers were nuclear power plant management teams at the highest levels. And, you might call it strong-arming but there was only one decision they could make. That was why I was off to Virginia…to test some “bad actors”.
He could have been speaking any language but English. I looked over at my new partner and noticed he was looking at me for help. Realizing help was not coming he simply turned back to the engineer and sounding very confident replied something along the lines of, “well, where is the valve pit?” “And”, the systems engineer continued, “this was a hot topic in the morning meeting so the plant manager, some of the other managers, the resident NRC inspector and possibly several others want to come out and watch the test. Let me know when you are ready.” As we were led toward the valve pit, my new partner asked, “Any idea what he said or what this thing is?”
There were several of these main feedwater isolation valves and the problem in this case was the steel stems that mated the valve to the actuator sheared when the valve was opened and in some cases the 18-inch diameter valve discs fell back into the pipe and could not be removed until the valve was disassembled. This is not the type of problem that an operating nuclear plant needs. I found a folding metal chair lost long ago in the crawl space and I sat it beside the valve near the top of the actuator to keep the test system off the floor and near eye level since I was on my knees.
The operating speed for the feedwater isolation valves at Sequoyah was just over 2.5 inches per second. When the large, high torque electric motor is energized and turning at 3600 RPM it does not simply stop when the power is removed. There is a significant inertial coast-down that is directly connected through the gearing to the valve. The SB designation indicates that there is a large internal spring called a compensator that absorbs some of the inertial energy to keep things from breaking.
The SB designation also indicates the compensator only works in the valve’s closing direction. The valve is not expected to contact a hard mechanical obstruction in the open direction so the compensator is not needed…. unless of course, an error is made during the set up.
First generation MOVATS testing required attachment of plates and load cells to the top of the actuator in order to calibrate the internal spring mechanism that is used to limit the torque output. It was more or less a blocking device that caused the actuator to stall against a calibrated load cell. As the load increased, the torque limiting mechanism responded and the electric motor would be de-energized. So, I was sent into the crawl space to attach a cage-like structure in order to block or restrain something powerful enough to lift a jumbo jet off of the ground.
The steel connecting rods holding everything together stretched, the plates bowed and I was so startled that I simply dove away when things got hairy and I missed capturing the data. The plant engineer yelled through the door, “did you get it?” Though nothing but unrepeatable yet eloquently put together expletives perfected by my former Marine drill instructors would come from my mouth, the answer was “NO”. So, we repeated that step in the procedure and found that the total measured force output was just over 250,000 pounds. I was questioning whether the rods, plates and other attachments could really withstand that much force but, good engineering wins again.
The next step was to remove the blocking plates and load cell and monitor the full cycle of the valve. Like most of the valves that had been tested by MOVATS engineers to that point, the problem was a simple switch position issue. Keep in mind the extreme inertia that continued to carry the valve after the motor was de-energized. Well, a recent procedure had been implemented that changed the position of these switches. Now, during normal operation the motor would not get de-energized until the valve was very near its open travel limit and even though de-energized, it would continue the inertial movement into the mechanical limits of travel.
As a consequence, components would come together at greater than a quarter million pounds of force that were not designed to do so and, things break. It was an easy mistake to make, an easy mistake to find and easy to fix provided you monitored the MOV and its response during operation. TVA had just become a long time customer and we sent several engineers up to test the remaining feedwater valves and many others.
Art had been at INPO prior to starting MOVATS where he had a key role in development of the 1983 INPO report titled, “Assessment of Motor-Operated Valve Failures”. MOVs were already known to be a big problem area for nuclear power plants but there was not a concerted effort to figure out how bad the problem was or what should be done. At some point during or after the development of the INPO report, Art came up with the idea of attaching sensors and other devices to the actuators and recording the electronic signatures during operation as a diagnostic test much like an EKG.
Things did not start fast and at several points Art had to take drastic financial measures to keep the business going. In the end, I think even Art was shocked by what would eventually be discovered, just how bad the real condition of MOVs in nuclear power plants really was and how successful his start-up was about to become.
There were several MOVs that failed during the event, the most notable being the auxiliary feedwater pump discharge MOVs. As expected and just like the majority of MOVs we had tested at plants around the country, these MOVs also had incorrectly set control switches. In this case, it was the torque bypass switch. The bypass circuit provides a parallel electrical path around the torque control switch during the early part of operation such that the MOV does not get de-energized during the high loads that occur when the valve is unseating.
If the bypass opens too early, the high loads will open the torque limiting switch and the MOV will simply stop. That is what happened at Davis Besse and the Auxiliary Feedwater pumps could not make up the cooling water loss in the secondary side of the steam generators and they boiled dry. At that point, and much like a car running with a dry radiator, there was no way to remove heat from the reactor primary system.
Davis Besse was a watershed event for the nuclear power industry and it would open a number of prying eyes. It was not unusual to get accompanied by entire teams of NRC people during the MOVATS testing that followed. There was also a contingent of congressmen and other local and state dignitaries that came out to understand the problems and how we were correcting them. We must have tested each of the failed auxiliary feedwater isolation valves 10 or more times mostly so dignitaries could watch.
The official conclusion published by the plant owner and by the NRC in various reports was that the torque bypass switches had been improperly set and did not provide the necessary bypass function during the high loading that occurs at valve unseating. There was also discussion around whether the bypass setting at the 5% stroke distance coverage required by the existing plant procedure was adequate for the high loads caused by differential pressure.
I recalled that the high load was present for much more than the target 5%. And, since I was somewhat familiar with the equations that are used to determine how much force might be required when the valve is operated under dynamic conditions, such as when the auxillary feedpump is pushing its entire output against the side of the closed valve, I was quite sure the forces would be very different compared to what we were seeing during our simple, after-the-fact static testing. The solution for the extensive debate that followed was to simply create a pressure on one side of each valve and open them while they were being monitored. Though I would later learn of other full flow test programs performed on nuclear valves in various test facilities, to my knowledge this was the first instrumented differential pressure test of an MOV in a nuclear power plant.
The results of that differential pressure testing indicated that it took much more thrust or force to open the valves than was expected when the MOVs were originally sized and the high loads due to differential pressure extended well beyond the 5% of stroke length assumed in the plant’s set-up procedure. As a result, we began testing a broader population of the plant’s MOVs by applying a differential pressure across each valve during the test. My curiosity got the best of me and I started researching all of the equations used around the industry to determine what the appropriate values should be.
I remained at Davis Besse for most of the summer but as the volume of testing declined we reduced our staff and I returned to Atlanta. I continued to analyze the differential pressure test results and finally shared some of the graphs with Art. This created a new wave of excitement and I think his term was “smoking gun”. This was the reason we (the collective industry) were having such issues with MOVs. Where we thought that simply extending the bypass coverage was the answer, we were accidently masking the real issue. And, since we had always assumed the original equations were adequate and therefore only tested under static conditions, we were not seeing what was happening when the valves opened and closed under dynamic system operating conditions when high flow rates and high differential pressures were affecting performance. Art gave me some suggestions on cleaning up the charts and graphs and walked away with my copies.
This was a brand new carousel with brand new material. We really had not discussed or rehearsed how any part of the meeting or presentation was to go but early in the presentation Art asked if I had my calculator and of course I did. He then asked me to enter the necessary design specifics to calculate the thrust requirements for a specific valve size. I did but asked which results he wanted; the actuator OEM, the valve OEM, any of the various equations used by different engineering companies, the textbook example, I had several. He asked me for each one, but one at a time and he wrote the results on a board.
There was a lengthy discussion on the various equations and what caused the differences in the results. I think this part of the discussion was all about establishing credibility and maybe a little about bringing them up to speed on the process before revealing the new reality. Art had cleverly demonstrated to the attendees that we fully understood what he was about to present and prepared them for maximum impact.
Art advanced the carousel to the next slide and there appeared a re-creation of my tables that he had walked away with a couple of weeks earlier. I remember thinking that it was a good thing I had not made any changes to my program and all of the numbers were exactly the same as those written on the board. However, in this case there was an extra column representing the actual values measured during the dynamic tests. We only had about 30 dynamic test data points at that time but, in most cases the actual measured values were much higher than the calculated values. Art asked me to describe how the test results were obtained and I pointed out that the differential pressures created for these tests were artificially produced by using a hydraulic pressure pump and all were in the open direction meaning the valve was closed and the space upstream of the valve was pressurized before the valve was opened.
There were no closing direction results and there were no operational results where we could judge what might happen when the normal or accident operating conditions continued to act on the valve during the cycle since the pressure bled off to zero very quickly when the valves started to open. Art had knowledge of MOV full flow testing that had occurred at the Marshall Steam Station where the results indicated that the closing force was much higher than calculated for the tested valves and surmised that we should expect something similar for most, if not all other valves.
I was sitting by the pool late in the evening enjoying a warm, early November sunset since it was already cold back in Atlanta. My boss, the same guy from the second interview walked up with some papers rolled up in his hand and tossed them into my lap. He said, in an almost irritated way, “I hope you guys are happy, our lives have just changed in a very big way”. Those pages were a faxed copy of NRC Inspection and Enforcement Bulletin, IEB 85-03, “Motor-Operated Valve Common Mode Failures During Plant Transients Due to Improper Switch Settings” issued by the NRC earlier that morning. The bulletin required a number of actions but the one that jumped out at me was the requirement for plants to test the MOVs covered by the bulletin under dynamic, differential pressure conditions which was something I was clamoring for since my differential pressure data had stalled at the 30 or so tests that I had before the NRC presentation.
The bulletin also described in detail the events at Sequoyah and Davis Besse. As I was reading, he went on the say, “I don’t really know whether this is a good or bad thing and I don’t know where you are going to find the people to do all of the testing that is about to come”. Several minutes passed before it occurred to me that I had just been informed that I was going to be responsible for developing and managing the organization that would make it happen.
The bulletin gave each plant 2 years to comply with the requested actions. As the testing projects that evolved as a consequence of 85-03 progressed, the results continued to reveal just how serious the problems really were. Some began to suggest that Art’s open line to the NRC was out of bounds but, he never tried to hide or cover up those discussions. I traveled with Art quite a bit over the following few years while 85-03 was being implemented, including 3-4 additional trips to D.C. to present updates to the NRC.
When we met with the plant management organizations (at almost every plant), his carousel would always include slides presented to the NRC and he would identify them as such and talk about those discussions. Some thought he was all about the money and he was using the NRC to make his market but that was far from the truth. Just like he was still at INPO, Art saw MOVATS and our many engineers as an extension of plant organizations and he wanted each of us to be just as concerned about our customer’s businesses as they were. He brought in a very long list of well-respected nuclear industry engineers at a very high cost including engineers deep in statistical analysis processes in order to make the most of the growing differential pressure test data.
He maintained a fleet of aircraft such that we could respond to any customer within just a few hours, he kept a research and development group focused on improving the test equipment and understanding the issues and how to solve them, he built a functional flow loop so we could perform live differential pressure testing in our own facility, he created and sponsored an industry users group to make sure everyone was getting the right information as soon as it was available and countless other things that took all of the profits right off the bottom line. It may have become all about profits later as the ownership changed hands but during the first few years and 1985 in particular, it was not.
Some plants conducted massive and costly MOV refurbishment projects only to reestablish the same flawed switch settings that were the real root cause of their problems. After analyzing the results of the 85-03 data from each plant and other research results set in motion during that period, the NRC would eventually extend the scope of the bulletin to cover all safety-related MOVs through Generic Letter 89-10, “Safety-Related Motor-Operated Valve Testing and Surveillance” issued in June of 1989.
After countless offers, Art finally gave in and sold the business as part of a plan to create a larger, one-of-a-kind nuclear valve services company, which did happen and we were extremely successful. However, the new owner ran into a number of significant financial problems and on May 10, 1990, Art was more or less fired while those of us that remained watched the Cinderella story of the nuclear industry get sold through what closely resembled an auction process. During that time, we had MOVATS engineers on long term assignments at many plants and I was getting regular updates from the field as the potential buyers visited our customers as part of their due-diligence process before constructing their offers to buy the business.
The most rewarding update came from a field engineer that by chance was in an elevator with a potential buyer and a manager at a nuclear plant where they were checking our references and the strength of our relationship. The manager asked the potential buyer’s rep why they did not simply design and market their own system. His response was, “We could do that easily; in fact, several companies have done just that. However, nobody can compete with the MOVATS field services organization”. Looking back, that was very true. Customers did not even ask for resumes or work histories of the engineers being assigned to their projects, they just told us how many and trusted they would be qualified. We had achieved Art’s vision of being a real extension of our customer’s organizations.