Reflections on MOV testing

According to measures I have used for many years and the fact that my oldest son is still younger than I was at the time, I consider myself as being just a kid when I drove down to Atlanta to interview for an obscure job I found in the back of the Atlanta Sunday paper. Sure, I was just over a year removed from 4 years in the US Marine Corps where I had chased fighter jets around the globe and into the exotic far reaches of the US military during the height of the cold war, but I was still just a kid. I found the company’s office in a small strip shopping center in an affluent suburb on the north side of the city. At first sight, I considered the office and location more suited for a lawnmower repair shop and remember thinking that this business was not going to support the career objectives I was searching for. I briefly considered backing out of the parking space and passing on the interview.

There was one big room, then occupied by 3 or 4 women doing administrative or accounting-like things and one private office where the interview occurred. The young man that interviewed me was about my age, seemed nice enough and his card said “Office Manager” but, he kept referring to someone else whom I would eventually learn was the owner of this brand new and still very small start-up. The company name was not very innovative, just an acronym for the company’s only product. MOVATS, also known as Motor-Operated Valve Analysis and Test System followed by “Incorporated”, seemingly an afterthought needed to make it sound like an official business. All I really gleaned from the interview was that they did something in nuclear power plants and the growth potential for the business was enormous. A couple of days later, I was called back for a second interview with another guy but still not the owner.

During the year and few months since getting out of the Marine Corps, I had worked for a small electrical contracting company. I started as an electrician’s helper though I had far more formal training than anyone I worked with. That training was a big advantage and within a few short months I was given my own jobs which were mostly robotics and automation projects in various types of manufacturing plants including the local cotton mills.

I was promoted to an in-house position a couple of months later as the materials manager where I purchased and staged tools, equipment and construction materials for the many jobs we were working around the North Georgia area. Depending on the workload, we consistently employed 40-50 electricians and an equal number of apprentices or “helpers”. There seemed to always be 25 or more jobs going at the same time and I quickly learned to not expect the job site leaders to plan their projects, ensure they were prepared or tell anyone about what they might need until the exact point in time that they really needed it.

As a consequence, and usually around mid-morning, I would get the daily calls from irate electricians complaining that they had some number of people sitting on their hands because they did not have the tools or equipment necessary to work. It was a very inefficient and needy culture and I had to develop systems and processes in order to get ahead of it. These electricians were all well-seasoned (meaning grumpy old men) and I was the kid, so my solution turned into managing their jobs for them and only having to worry about whether my delivery drivers got there before they finished the morning coffee and pre-job socializing.

First off, as a naïve country boy from a small North Georgia cotton mill community I had a hard time coming to grips with the fact that the same products used to control sewage are also used to control critical safety-related processes in nuclear power plants. Secondly, the fact that they expected to build this huge global business around the process of attaching sensors to these actuators and recording the output with an oscilloscope seemed a bit far-fetched. The broad range of test equipment that I had used to troubleshoot the many different avionics and electrical systems on military aircraft and the sensors and systems I had installed on the robotics and automation projects seemed to make me grossly overqualified for what they needed.

However, I accepted the job offer and reported to work a couple of weeks later. When I arrived on that first day, I seemed to have caught the women in the office and the office manager from the initial interview by surprise. The second interviewer which was now my new boss, the owner and everyone else were out-of-town at far away nuclear power plants, they did not know when anyone would be back and they did not know what I should do.

The office manager told me that there was some MOVATS equipment and an MOV on a test stand out at the calibration lab and suggested I go there. This “calibration lab” turned out to be a small room built in the back of an aircraft hangar at the local municipal airport where the owner kept his plane. There was a guy at that location that maintained the MOVATS equipment but he was quick to point out that he had no knowledge of how it was used.

It was 2 weeks later that my new boss was finally back in town so I reported to the main office to determine what was next. He wanted to know what I had been doing for the past 2 weeks and I explained that I went over to the hangar, found a test system and actuator and started running tests. I would connect the equipment, run a test, take the actuator apart, make changes and run another test, then another, on and on for 2 weeks. I had written down some questions on why certain things occurred and why they performed certain steps in the test procedure. His next question was whether I wanted to go to a nuclear plant in Virginia…. tomorrow morning. So, in March 1985, less than a month after the coincidental connection that landed me the job and 2 weeks of self-teaching on MOVATS, I was off to the Surry Nuclear Power Plant as a MOVATS engineer and MOV diagnostic expert.

But, I had still not met the owner. As I learned later, the owner was always on the road, selling, politicking, identifying the problems and getting potential customers to confess their need. At first, he did not really want to sell his product. “Just let us come out and test a few MOVs, maybe just the bad actors and see if we can help” was the approach. Just like a dope pusher trying to bait a youngster into a life of dependence, he was very good at it. But, in this case, the target customers were nuclear power plant management teams at the highest levels. And, you might call it strong-arming but there was only one decision they could make. That was why I was off to Virginia…to test some “bad actors”.

I was met at the Norfolk, Virginia airport by an even younger MOVATS engineer and keep in mind, I was still just a kid. He had started a few months earlier, had already been to a handful of nuclear plants and was by far the most experienced person I had met to that point. By all accounts, he was a pretty good college baseball pitcher prior to this job but, the lengthy scar around his shoulder was all the evidence needed to understand why he was now a nuclear valve tester.

For those not familiar with a Limitorque SB-4, it is a big and powerful machine. It has a large 460 volt 3-phase electric motor about the size of a 55-gallon drum. It can produce a lot of torque, 200 ft-lbs in this case. The mechanical gearbox gives the actuator the significant boost in torque required to open and close very large valves operating under very high pressure conditions. It could also lift a large and fully fueled commercial jet-liner off the ground with ease.

The operating speed for the feedwater isolation valves at Sequoyah was just over 2.5 inches per second. When the large, high torque electric motor is energized and turning at 3600 RPM it does not simply stop when the power is removed. There is a significant inertial coast-down that is directly connected through the gearing to the valve. The SB designation indicates that there is a large internal spring called a compensator that absorbs some of the inertial energy to keep things from breaking.

The SB designation also indicates the compensator only works in the valve’s closing direction. The valve is not expected to contact a hard mechanical obstruction in the open direction so the compensator is not needed…. unless of course, an error is made during the set up.

First generation MOVATS testing required attachment of plates and load cells to the top of the actuator in order to calibrate the internal spring mechanism that is used to limit the torque output. It was more or less a blocking device that caused the actuator to stall against a calibrated load cell. As the load increased, the torque limiting mechanism responded and the electric motor would be de-energized. So, I was sent into the crawl space to attach a cage-like structure in order to block or restrain something powerful enough to lift a jumbo jet off of the ground.

The first cycle of the test itself was quite alarming. This particular actuator was on its side and the stem was horizontal. Since I had placed the folding chair and test system right at the top of the actuator where the load cell and blocking plates were installed, I had an up close and personal view of the 3 or maybe 4-inch diameter steel valve stem as it raced upward at just over 2.5 inches per second. When it contacted the load cell, the actuator simply moaned and pushed harder.

The steel connecting rods holding everything together stretched, the plates bowed and I was so startled that I simply dove away when things got hairy and I missed capturing the data. The plant engineer yelled through the door, “did you get it?” Though nothing but unrepeatable yet eloquently put together expletives perfected by my former Marine drill instructors would come from my mouth, the answer was “NO”. So, we repeated that step in the procedure and found that the total measured force output was just over 250,000 pounds. I was questioning whether the rods, plates and other attachments could really withstand that much force but, good engineering wins again.

The next step was to remove the blocking plates and load cell and monitor the full cycle of the valve. Like most of the valves that had been tested by MOVATS engineers to that point, the problem was a simple switch position issue. Keep in mind the extreme inertia that continued to carry the valve after the motor was de-energized. Well, a recent procedure had been implemented that changed the position of these switches. Now, during normal operation the motor would not get de-energized until the valve was very near its open travel limit and even though de-energized, it would continue the inertial movement into the mechanical limits of travel.

As a consequence, components would come together at greater than a quarter million pounds of force that were not designed to do so and, things break. It was an easy mistake to make, an easy mistake to find and easy to fix provided you monitored the MOV and its response during operation. TVA had just become a long time customer and we sent several engineers up to test the remaining feedwater valves and many others.

Things were happening quickly at this point and I have often compared what we were experiencing as loosely similar to John Wayne’s portrayal of Red Adair, the famous oil well fire fighter in the late 60s movie “Hell Fighters”. Hanging out around the shop or the pool table, waiting for the call to board the company jet and rushing out to save the day but, instead of an oil well it was a nuclear power plant and thank goodness, no fires. At some time around the Sequoyah trip, I finally met Arthur G. Charbonneau (Art), the owner and single person responsible for what we were doing.

Art had been at INPO prior to starting MOVATS where he had a key role in development of the 1983 INPO report titled, “Assessment of Motor-Operated Valve Failures”. MOVs were already known to be a big problem area for nuclear power plants but there was not a concerted effort to figure out how bad the problem was or what should be done. At some point during or after the development of the INPO report, Art came up with the idea of attaching sensors and other devices to the actuators and recording the electronic signatures during operation as a diagnostic test much like an EKG.

Things did not start fast and at several points Art had to take drastic financial measures to keep the business going. In the end, I think even Art was shocked by what would eventually be discovered, just how bad the real condition of MOVs in nuclear power plants really was and how successful his start-up was about to become.

On June 9, 1985, the Davis Besse nuclear power plant near Toledo, Ohio experienced nearly the same sequence of events that led to the accident at Three Mile Island. Again, the failure of critical MOVs played a key role in the event and Art was on the phone most of the day with various people at the site. Either late that day or early the next day my boss and another MOVATS engineer left for the site. I was sent up a couple of days later.

There were several MOVs that failed during the event, the most notable being the auxiliary feedwater pump discharge MOVs. As expected and just like the majority of MOVs we had tested at plants around the country, these MOVs also had incorrectly set control switches. In this case, it was the torque bypass switch. The bypass circuit provides a parallel electrical path around the torque control switch during the early part of operation such that the MOV does not get de-energized during the high loads that occur when the valve is unseating.

If the bypass opens too early, the high loads will open the torque limiting switch and the MOV will simply stop. That is what happened at Davis Besse and the Auxiliary Feedwater pumps could not make up the cooling water loss in the secondary side of the steam generators and they boiled dry. At that point, and much like a car running with a dry radiator, there was no way to remove heat from the reactor primary system.

Davis Besse was a watershed event for the nuclear power industry and it would open a number of prying eyes. It was not unusual to get accompanied by entire teams of NRC people during the MOVATS testing that followed. There was also a contingent of congressmen and other local and state dignitaries that came out to understand the problems and how we were correcting them. We must have tested each of the failed auxiliary feedwater isolation valves 10 or more times mostly so dignitaries could watch.

The official conclusion published by the plant owner and by the NRC in various reports was that the torque bypass switches had been improperly set and did not provide the necessary bypass function during the high loading that occurs at valve unseating. There was also discussion around whether the bypass setting at the 5% stroke distance coverage required by the existing plant procedure was adequate for the high loads caused by differential pressure.

I had performed an MOV test at a different plant where the open signature showed higher than expected unseating loads during the first cycle but the load characteristics changed on the following cycles. We had deduced that there must have been some differential pressure across the valve during the initial opening cycle but we had relieved that pressure during that first opening and it was not present on the following cycles.

I recalled that the high load was present for much more than the target 5%. And, since I was somewhat familiar with the equations that are used to determine how much force might be required when the valve is operated under dynamic conditions, such as when the auxillary feedpump is pushing its entire output against the side of the closed valve, I was quite sure the forces would be very different compared to what we were seeing during our simple, after-the-fact static testing. The solution for the extensive debate that followed was to simply create a pressure on one side of each valve and open them while they were being monitored. Though I would later learn of other full flow test programs performed on nuclear valves in various test facilities, to my knowledge this was the first instrumented differential pressure test of an MOV in a nuclear power plant.

The results of that differential pressure testing indicated that it took much more thrust or force to open the valves than was expected when the MOVs were originally sized and the high loads due to differential pressure extended well beyond the 5% of stroke length assumed in the plant’s set-up procedure. As a result, we began testing a broader population of the plant’s MOVs by applying a differential pressure across each valve during the test. My curiosity got the best of me and I started researching all of the equations used around the industry to determine what the appropriate values should be.

In 1985, there were few computers and those that were around were generally on the desks of secretaries or other administrative types where they were used mostly for typing memos using early word processing software. We did not have Microsoft Office with Word, Excel and all of the other tools we have today. In fact, I did not get my first desktop PC with Lotus 1,2,3 until late 1986. However, I did have a Sharp pocket calculator that could be programmed using Basic to run scripts. I programmed it to calculate MOV design thrust requirements using 5 or 6 of the various equations used in the nuclear industry at that time. I would input the necessary parameters such as valve size, stem size and differential pressure and it would provide the results for each method. I used a lot of graph paper to manually plot the results.

I remained at Davis Besse for most of the summer but as the volume of testing declined we reduced our staff and I returned to Atlanta. I continued to analyze the differential pressure test results and finally shared some of the graphs with Art. This created a new wave of excitement and I think his term was “smoking gun”. This was the reason we (the collective industry) were having such issues with MOVs. Where we thought that simply extending the bypass coverage was the answer, we were accidently masking the real issue. And, since we had always assumed the original equations were adequate and therefore only tested under static conditions, we were not seeing what was happening when the valves opened and closed under dynamic system operating conditions when high flow rates and high differential pressures were affecting performance. Art gave me some suggestions on cleaning up the charts and graphs and walked away with my copies.

Several days passed, maybe 2 weeks at the most when Art asked me to join him on a trip to Washington, D.C. to meet with the NRC. Art was the master of slide presentations and keep in mind we did not have PCs, PowerPoint or computer projectors. However, almost everyone had a 3M slide projector and Art always carried 3 or 4 slide carousels in his big brown flight case and, depending on the audience make-up would select the most appropriate.

This was a brand new carousel with brand new material. We really had not discussed or rehearsed how any part of the meeting or presentation was to go but early in the presentation Art asked if I had my calculator and of course I did. He then asked me to enter the necessary design specifics to calculate the thrust requirements for a specific valve size. I did but asked which results he wanted; the actuator OEM, the valve OEM, any of the various equations used by different engineering companies, the textbook example, I had several. He asked me for each one, but one at a time and he wrote the results on a board.

There was a lengthy discussion on the various equations and what caused the differences in the results. I think this part of the discussion was all about establishing credibility and maybe a little about bringing them up to speed on the process before revealing the new reality. Art had cleverly demonstrated to the attendees that we fully understood what he was about to present and prepared them for maximum impact.

Art advanced the carousel to the next slide and there appeared a re-creation of my tables that he had walked away with a couple of weeks earlier. I remember thinking that it was a good thing I had not made any changes to my program and all of the numbers were exactly the same as those written on the board. However, in this case there was an extra column representing the actual values measured during the dynamic tests. We only had about 30 dynamic test data points at that time but, in most cases the actual measured values were much higher than the calculated values. Art asked me to describe how the test results were obtained and I pointed out that the differential pressures created for these tests were artificially produced by using a hydraulic pressure pump and all were in the open direction meaning the valve was closed and the space upstream of the valve was pressurized before the valve was opened.

There were no closing direction results and there were no operational results where we could judge what might happen when the normal or accident operating conditions continued to act on the valve during the cycle since the pressure bled off to zero very quickly when the valves started to open. Art had knowledge of MOV full flow testing that had occurred at the Marshall Steam Station where the results indicated that the closing force was much higher than calculated for the tested valves and surmised that we should expect something similar for most, if not all other valves.

It is always hard to judge the impact of a meeting and presentation like the one I had just sat through. They took few notes but the discussion was candid and everyone seemed well informed. A couple of the NRC participants even joined us for lunch at a nearby sandwich shop. I was watching a major nuclear safety issue unfold right before my eyes and it was not exactly clear what everyone else was thinking. A couple of weeks later I was at a hotel near the Crystal River nuclear plant where we were testing another handful of “bad actors”.

I was sitting by the pool late in the evening enjoying a warm, early November sunset since it was already cold back in Atlanta. My boss, the same guy from the second interview walked up with some papers rolled up in his hand and tossed them into my lap. He said, in an almost irritated way, “I hope you guys are happy, our lives have just changed in a very big way”. Those pages were a faxed copy of NRC Inspection and Enforcement Bulletin, IEB 85-03, “Motor-Operated Valve Common Mode Failures During Plant Transients Due to Improper Switch Settings” issued by the NRC earlier that morning. The bulletin required a number of actions but the one that jumped out at me was the requirement for plants to test the MOVs covered by the bulletin under dynamic, differential pressure conditions which was something I was clamoring for since my differential pressure data had stalled at the 30 or so tests that I had before the NRC presentation.

The bulletin also described in detail the events at Sequoyah and Davis Besse. As I was reading, he went on the say, “I don’t really know whether this is a good or bad thing and I don’t know where you are going to find the people to do all of the testing that is about to come”. Several minutes passed before it occurred to me that I had just been informed that I was going to be responsible for developing and managing the organization that would make it happen.

The bulletin gave each plant 2 years to comply with the requested actions. As the testing projects that evolved as a consequence of 85-03 progressed, the results continued to reveal just how serious the problems really were. Some began to suggest that Art’s open line to the NRC was out of bounds but, he never tried to hide or cover up those discussions. I traveled with Art quite a bit over the following few years while 85-03 was being implemented, including 3-4 additional trips to D.C. to present updates to the NRC.

When we met with the plant management organizations (at almost every plant), his carousel would always include slides presented to the NRC and he would identify them as such and talk about those discussions. Some thought he was all about the money and he was using the NRC to make his market but that was far from the truth. Just like he was still at INPO, Art saw MOVATS and our many engineers as an extension of plant organizations and he wanted each of us to be just as concerned about our customer’s businesses as they were. He brought in a very long list of well-respected nuclear industry engineers at a very high cost including engineers deep in statistical analysis processes in order to make the most of the growing differential pressure test data.

He maintained a fleet of aircraft such that we could respond to any customer within just a few hours, he kept a research and development group focused on improving the test equipment and understanding the issues and how to solve them, he built a functional flow loop so we could perform live differential pressure testing in our own facility, he created and sponsored an industry users group to make sure everyone was getting the right information as soon as it was available and countless other things that took all of the profits right off the bottom line. It may have become all about profits later as the ownership changed hands but during the first few years and 1985 in particular, it was not.

Over the next 2 years, MOVATS revealed a broad range of problems with MOVs and the solutions required much more than simple testing and adjustment. The general condition of both the actuators and the valves across the industry had reached a fairly sad state. Prior to MOVATS, nuclear plants did little more than the periodic recording of the time required for the MOV to complete an open or close cycle which proved to be completely ineffective. I came across a couple of plants that used a chart recorder to capture motor current signatures during operation but that process was not capable of identifying the basic problems.

Some plants conducted massive and costly MOV refurbishment projects only to reestablish the same flawed switch settings that were the real root cause of their problems. After analyzing the results of the 85-03 data from each plant and other research results set in motion during that period, the NRC would eventually extend the scope of the bulletin to cover all safety-related MOVs through Generic Letter 89-10, “Safety-Related Motor-Operated Valve Testing and Surveillance” issued in June of 1989.

After countless offers, Art finally gave in and sold the business as part of a plan to create a larger, one-of-a-kind nuclear valve services company, which did happen and we were extremely successful. However, the new owner ran into a number of significant financial problems and on May 10, 1990, Art was more or less fired while those of us that remained watched the Cinderella story of the nuclear industry get sold through what closely resembled an auction process. During that time, we had MOVATS engineers on long term assignments at many plants and I was getting regular updates from the field as the potential buyers visited our customers as part of their due-diligence process before constructing their offers to buy the business.

The most rewarding update came from a field engineer that by chance was in an elevator with a potential buyer and a manager at a nuclear plant where they were checking our references and the strength of our relationship. The manager asked the potential buyer’s rep why they did not simply design and market their own system. His response was, “We could do that easily; in fact, several companies have done just that. However, nobody can compete with the MOVATS field services organization”. Looking back, that was very true. Customers did not even ask for resumes or work histories of the engineers being assigned to their projects, they just told us how many and trusted they would be qualified. We had achieved Art’s vision of being a real extension of our customer’s organizations.