Train, inspect, test, maintain: Data Center Managers have a lot to do to ensure a generator responds when needed
The stakes are quite high when it comes to optional or life-safety emergency standby power systems. Failure to start or failure to run can have enormous consequences. Regular testing, maintenance, exercising, and inspection can help keep standby generators ready to perform when needed.
For optional standby generators (not required by life-safety code), critical loads supported by the generator system typically include data center and call center equipment such as UPS systems, cooling, phone systems, and desktop equipment (computers, etc.).
For emergency standby generators (required by life-safety code), critical loads supported by the generator system typically include emergency lighting, fire alarm systems, fire pumps, and elevators. Life-safety generators are also sometimes used to additionally support optional loads, such as data centers; however, in this case the life-safety loads take precedence over the optional loads.
Good design, quality equipment, trained operating personnel, commissioning, regular inspections and exercising, preventative maintenance, trained service support, and performance testing are all key to reliable performance.
Types Of Power
It is important to understand the difference between continuous onsite power generation (prime power), long-term power backup systems (standby generators, almost always diesel-fired), and short-term power backup systems (uninterruptible power systems, or UPS).
Prime power generation includes natural gas-fired turbines and micro-turbine generators, fuel cells, solar, and wind-generated power. Prime power generation is designed to run continuously or at least for many hundreds of hours per year as required by demand or energy economics. To improve prime power economics, engines can operate from alternative fuels (methane, bio-diesel) and heat byproducts can be re-used for other purposes (combined heat/power or co-generation). It is almost never economical to use diesel-fired generators for prime power generation, and in many areas this is prohibited by air quality regulations. When operating critical equipment from onsite-generated prime power, utility power can normally be used as a long-term backup power source. Prime power sources do not normally make good standby backup power sources; prime power sources can take many minutes to start up and often require steady base loads for stable operation.
Backup power system main components can include the engine, fuel storage, fuel delivery, speed governor, voltage regulator, engine exhaust and filter, engine cooling, alternator, circuit breaker(s), engine cranking batteries, battery charger, block heater, paralleling switchgear, load prioritizing and shedding controls, distribution switchboard, automatic and manual transfer switches (ATS, MTS), and local and remote monitoring. Improper operation, lax maintenance, or random failures in any of these components can result in failure of the backup power system to perform when it is needed.
Long-term power backup systems are normally off and in a standby state, waiting for a command to start and provide power for minutes, hours, or days (depending on available fuel and re-fueling capability) during a failure of prime power (typically the power utility). Backup generators are almost always diesel-fired, for quick starting, acceptance of large load blocks, and for effective speed/frequency/voltage regulation with load changes. Backup generators cannot replace UPS systems, however, because they simply cannot start fast enough to maintain adequate voltage continuity for IT equipment.
UPS systems, even though they may run continuously, are intended only to provide short-term power backup, in the matter of seconds or minutes, most often using batteries for energy storage. UPS systems take care of the majority of prime power bumps and failures, which are typically less than a minute in duration. However, UPS systems are not normally suitable for heavy block loads common with HVAC equipment, fire pumps, etc. UPS systems cannot be used for long-term power backup because supported loads are normally limited to IT equipment, and the scale of required stored energy in the form of batteries for UPS systems for long operation escalates exponentially compared to that of diesel fuel for generators. One barrel of diesel fuel is equivalent in power storage energy to a large room full of batteries.
Keys To Backup Power System Success: Features, Flexibility, And Redundancy
Backup power system success begins with features, flexibility, and redundancy to fit the need. Name brand, quality equipment with adequate local sales and service support should be sourced.
Once installed, backup power systems should be thoroughly commissioned. This includes operation of all components and key features in all applicable modes (normal, bypass, failover, etc.), preferably under loads ranging from very light to full design load. Integrated system testing with actual loads connected and loaded (HVAC, UPS, etc.) including remote monitoring and BMS tie-in should be performed, culminating in multiple utility-power-fail “pull-the-plug” tests. Site-specific standard and emergency operating procedures should be developed in advance of commissioning. The commissioning phase of the project is the best opportunity to validate procedures and train operators.
Ideally, operators become comfortable actually performing manual operation of generators and ATS. However, the equipment is highly automated. The goal for operator training on backup and support equipment is for operators to be able to observe normal automatic operation, to know when not to intervene (when to keep one’s hands in one’s pockets), and to recognize when something does not look or sound right.
A classic example of an operator fumble with a backup generator system supporting a UPS system plays out like this: The generator is running, the open-transition ATS is in “emergency” position, with the generator supporting the UPS, and the UPS is then transferred to bypass (no battery backup available). The previous steps are often manually performed during various maintenance tasks, although the system can find itself in this state automatically or through a combination of auto and manual actions. At this point it is mandatory to put the ATS into manual or test mode or to open the “normal” source to prevent an ATS re-transfer. If the ATS is allowed to remain in auto mode, and if normal utility then becomes available (or was never off) the ATS will time out, then automatically transfer back to the utility. The problem occurs if the UPS is still in bypass and is not ready for the transfer. The UPS cannot provide battery backup for the short-term ATS re-transfer power bump when it is in bypass. Then the critical equipment supported by the UPS sees a power failure and crashes.
Regular inspections are needed to ensure readiness. Unless the following items are remotely monitored with alerts programmed, direct observation is required on at least a weekly basis, sometimes as often as daily:
- Generator controls in automatic
- ATS controls in automatic
- No alerts, alarms
- Battery charger on with normal charge voltage and current (also check the power source which is often a single-pole breaker in a panel remote from the generator)
- Engine block heater functioning — block should always be warm to the touch
- Alternator main breaker closed
- All input and output breakers closed in generator switchgear, switchboards (as applicable)
- Underground fuel storage tank leak detection
Regardless of how often the above items are checked, any time maintenance of any sort is performed on any of this equipment, or on any equipment in the same general area, always re-check these items after all work is complete and service personnel have left the area.
Exercise Standby Generators To Keep Them In Fighting Shape
Unless your local utility regularly serves up power failures (of more than a few seconds), an exercising program is required for standby generators. It is standard practice to set ATS or switchgear controls for automatic engine starting and running at no load (no ATS transfer) for about 30 minutes; this is done as often as weekly but no less frequently than monthly. Exercising can be manually initiated as well. If automatically initiated, it’s a good idea for operators to be present onsite observing normal performance.
However, it is important to operate the generator with significant load periodically to prevent no-load wet-stacking, where unburned diesel fuel loads up the exhaust system and can cause problems. At least 40 percent load should be applied at least quarterly, up to as often as monthly. Exercising the generator under load can be accomplished by performing an ATS transfer after starting the generator, if adequate building load is available, and if the risk of equipment failure under test is acceptable. With an open-transition ATS, two brief power bumps will occur during transfer and re-transfer testing, causing lights to blink, UPS and batteries to also be exercised, and non-UPS-protected critical loads to be affected. With closed-transition ATS or switchgear, risk of failure may actually be higher during brief synchronization and paralleling but there will be no power bumps during transfer or re-transfer. Load banks can be used if adequate building load is not available or the risk of live transfer and unnecessarily operating the facility on generator is a concern. Permanent load banks rated at least 40 percent of engine capacity can be installed or purchased built into the generator enclosure.
When the system is running, whether under a test or during a utility power failure, several observations should be made:
- Normal engine sounds and vibrations
- Stable voltage and frequency
- No excessive exhaust smoke
- Normal engine temperature
- Proper fuel transfer, in particular between main and days tanks as applicable
- Normal oil pressure
- No engine oil leaks, fuel leaks, or coolant leaks
- No alarms or alerts
- Log actual load and conditions (time-of-day, day-of-week, critical building loads that are on or off) — tracking actual building load is key to overall load management
- Critical load acceptance of generator source (HVAC, drives, UPS, etc.)
Preventive Maintenance Is Critical To Dependable Standby Generators
Standby generators should be regularly serviced by, and under a 24/7 service agreement with, a qualified local service organization. Preventive maintenance (PM) service must be performed on schedule. Even OEM and qualified service organizations can become lax about PMs; it’s up to the owner to ensure that these contractual requirements are performed or to hold the service organization accountable.
PMs are generally focused on engine mechanic details including fuel and oil pumps and filters, fuel/ water separators, air filters, coolant, hoses, and belts. Don’t forget the required PMs for ATS and generator switchgear. However, particular attention should be given to battery condition and fuel quality, as these are notorious causes of failure to start and failure to run issues. You cannot run to failure with batteries and fuel. Engine-cranking batteries need attention beyond regular starts. Individual battery condition and connections must be checked during the PM. Batteries should be pro-actively replaced before end-of-life, further adjusted based on local climate and usage. Fuel quality must be inspected and tested during PMs and regardless of inspection and test results, and even when much of the fuel is used, fuel must be “polished” (cleaned of water, bacterial growth, contamination) annually.
Owner-operators should weigh in on optional set points associated with engine controls, typically located in the ATS or switchgear. These include engine start-time delay when sensing a power bump, how long to remain on generator after utility power returns, and engine cool down after unloading.
Engine start-time delay settings range from 0.5 seconds to 30 seconds. The only reason most facility managers would want the start signal sent to the generator in less than 3 seconds would be if it is supporting a UPS that is using something other than conventional batteries for short-term backup (typically 5 to 15 minutes), such as flywheels (typically only 7 to 20 seconds). One concern, as most people are well aware, is that the majority of utility power bumps last less than 3 seconds. Therefore you can have quite a few unnecessary engine starts with start programming set for less than 3 seconds.
Most operators set the generator start time delay in the 3 to 5 second range. If utility power is down for more than several seconds, it will probably be down for several minutes or hours, so you might as well start your engines. If the facility is dependent on lower cost UPS batteries (such as those supporting the vast majority of data centers), the start time should be at about 3 seconds. It’s best not to challenge UPS batteries any longer than necessary and risk the chance of a UPS failure. However, starting the generator for every light flicker takes its toll on equipment, with impacts on reliability, maintenance, and environmental emissions.
For emergency backup systems, safety codes typically require 10 seconds maximum for life-safety loads. Other loads can wait until additional engines are online.
Performance testing of the entire system at design load should be performed annually or at least triennially. If a permanent load bank is not connected or of sufficient capacity, then a mobile load bank must be brought in and connected to allow for sufficient load through the ATS and generator switchgear to allow for effective infrared thermal scanning. The engine should be operated at or near full load for several hours to accomplish a full heating and cool down cycle. The ATS should be operated in all manual and automatic modes and in all bypass isolation modes. Ideally, after all service and testing is performed and full confidence in the backup power system has been achieved, an actual power failure “pull-the-plug” test should be performed, observing not only proper generator and transfer switchgear operation but also proper notification of conditions and performance of related equipment such as HVAC and UPS systems.
