July 2006, Vol. 18, No.7

Problem Solvers

Switching to the Right Switch

Problem: Workstation network too slow.
Solution: Long-distance Ethernet switches.

The fiber-optic communications network used at the 45,420-m3/d (12-mgd) Pleasant Grove Wastewater Treatment Plant in Roseville, Calif., originally was designed in 1999. At the time, 10–megabit-per-second (Mbps) half-duplex data rates and multimode fiber-optic cable were considered adequate to handle network traffic, as well as data and video transmission distances.

However, when the system was implemented in late 2004, it became apparent that it needed some upgrading. According to Charles Aycock, electronic maintenance coordinator in the City of Roseville’s Environmental Utilities Department, operators began complaining about the slow speed of the remote workstations.

“Workstations at the remote locations [were] bogged down noticeably ... even when the surveillance cameras were not online,” Aycock explained. “Operators would click on a graphic interface ... and wait 3 to 5 seconds for each screen to refresh.”

Because of the slow screen-refresh times, Aycock said it often took operators several minutes to make routine adjustments, such as controlling dewatering pumps or adjusting chemical dosing levels.

“Operators reverted to making changes manually or utilizing the control room workstations in the administration building, rather than using the remote workstations,” Aycock said. The administration building control room workstations, which connected directly to the supervisory control and data acquisition (SCADA) servers via an Ethernet switch, did not exhibit graphical update problems. The problem was confined to the optical communications modules (OCMs) that fed the remote workstations.

Further investigation revealed the source of the problem: The graphics software for the treatment process required considerably more bandwidth than originally anticipated during design. The process screens that were developed contained more dynamic data and symbols as the operations staff determined their process control needs.

“The SCADA system passed all performance criteria for static and dynamic graphic update times; however, the testing criteria [were] defined for less than 40% of the information that was eventually depicted on each process screen,” Aycock explained. He also noted that the manufacturer of the OCMs acknowledged its product was never intended to pass the volume of network traffic required by the SCADA system.

Initially, 10-Mbps half-duplex Ethernet OCMs on a redundant fiber-optic ring were used to provide the communications transport for the workstations and cameras. The OCMs were “fault-tolerant” (often called “redundant”) and would automatically switch over to a second pair of fibers in case of a fiber-optic cable failure.

Research by Dan Mazorra of the engineering consulting firm Carollo Engineers (Phoenix) determined that TC Communications (Irvine, Calif.) had a product that would meet the treatment facility’s needs. A new redundant fiber network would use gigabit Ethernet switches to support the workstations and surveillance cameras.

This solution was theoretically sound, but the multimode fiber presented an obstacle in terms of available bandwidth at gigabit speeds (1000 Mbps). It turned out that the 335 m (1100 ft) between the Ethernet switch in the administration building and the switch in the influent pump station exceeded the 275-m (900-ft) maximum range for standard gigabit Ethernet networks. The OCMs did not encounter this problem because they fell well within the 2 km specification for 10/100M Ethernet.

“We encountered the distance problem when we initially tested standard versions of the Ethernet switches,” Aycock explained. “They were right on the edge. Sometimes they worked, sometimes they didn’t.”

Aycock again consulted the communications company, which suggested a special long-distance version of the switch that supports distances up to 2 km (1.2 mi) over 50/125 or 62.5/125 multimode fiber. List price for these long-distance versions is $4795 per unit.

“The long-distance multimode fiber versions tested perfectly, and we didn’t look back,” Aycock said.

Pleasant Grove’s fiber-optic communications network is a ring topology that consists of 24 strands of 62.5/125 multimode fiber. It connects seven workstations and eight surveillance cameras in various buildings and locations on plant grounds to the central computer room in the administration building.

“The operators were ecstatic with the new system,” said Bob Lawrence, the plant’s chief operator. “They were able to make changes on the fly at any of the workstations, and we could finally get useful images from the cameras.” Aycock noted that the cameras image-capture rates were previously slowed down to seconds per frame instead of normal video-capture rates in frames per second to minimize impact to the SCADA workstation network.

The new switches also maximized network reliability by supporting a fault-tolerant ring network topology. Each switch has dual fiber-optic ports. If a fiber cable or device failure occurs, the data path automatically switches over to the secondary path of fiber-optic cables to maintain ring network integrity.

Pleasant Grove also plans to connect the alarm dry contacts on the rear of the switches to various programmable logic controllers throughout the system. This will tie the switches into the SCADA system and alert plant operators if there is an alarm condition. It also will pinpoint the network location of the fiber-optic cable or equipment failure. The redundant feature of the switches provides for automatic switchover to the secondary data path in less than 50 milliseconds.