In this edition of OPC Talk with Win and Marc, we’re going to address some common questions we get about redundancy involving OPC software. If you’re new to the industrial automation space and want to learn more about redundancy in automation in general, visit our Demystifying Redundancy in Automation blog post.
How do OPC Servers handle redundant PLCs or controllers?
Win: Well that depends on what OPC server you are using. Some OPC Servers, such as TOP Server for Wonderware, have configuration settings that let specify a master PLC and backup PLC plus criteria for when to failover and fail back to the primary PLC.
Marc: I recently worked with a system integrator that was doing a project with 3 redundant controllers, and we were able to handle that using these same features in TOP Server for Wonderware.
Are there ways for me to view from my HMI or SCADA software what the status is of the various points of redundancy in my setup?
Marc: The TOP Server also provides built-in tags that let an HMI, SCADA or MES system provide operator visibility into which network path or controller is currently being communicated with.
When using software to manage your OPC redundancy you need to check with the vendor to see if they support a way to view which OPC Server connection is active. For example, when using the Cogent DataHub for OPC Redundancy you can create 4 tags; Current Source, Source 1 State, Source 2 State and Preferred Source. With these 4 tags you can see which sources are active as well as write to the preferred source to force a failover.
What if I have redundant network cards in my computer for separate paths to my PLCs?
Marc: Well like Win said, that depends on the OPC server that you are using. With TOP Server you can specify a secondary communications path the same way you specify a redundant PLC or controller, along with failover criteria. Built in tags provide operational status visibility to your HMI, SCADA, or MES.
I need to have my HMI or SCADA system automatically switch between a redundant pair of OPC servers. How do I do that?
Marc: That depends on what HMI or SCADA system you are using. Some systems such as Wonderware System Platform, have built-in redundancy objects that handle the switching. In them you configure the primary and backup OPC server and they handle the switching and operational state visibility to the operator automatically.
Win: I’ve worked with users many times that don’t have the luxury of having redundant OPC connection management built into their HMI or SCADA system. For those users we helped them implement a tool like the Cogent DataHub or the Kepware Redundancy Master. Both Cogent DataHub and Redundancy Master establishes a connection to primary and secondary OPC servers and then allows for various levels of failover speed that are described in our Maximizing OPC availability blog post.
I have had some users whose requirements included very specific needs that a point-and-click solution wouldn’t address. With those users, we used the Cogent DataHub which includes a scripting engine in each license that handles more complex failover scenarios as well as an optional email/SMS notification plug in. As Marc mentioned earlier the Cogent DataHub allows you to configure 4 tags to give visibility its which sources are active and allow you to force a failover by writing to a tag.
I need to have redundant MES systems, how to I get my production data to these redundant systems?
Win: This is just another case of the situation where you have an HMI or SCADA system that needs redundant data. The answer will depend on the methods your MES supports to interface with your SCADA system. If the MES supports OPC DA or OPC UA then you can use something like Cogent DataHub to manage redundancy. The image below is a great example of dual channel redundancy that one of our customers implemented using Cogent DataHub and their MES.
Each SCADA has a local connection to a DataHub. Each MES also has a local connection to a DataHub. From there both of the DataHub’s on the SCADA side have redundant connections to the DataHub’s on the MES side. This way if any one of the connections between the MES and SCADA layers fails there are still multiple active paths. If you’re interested in doing something like this, contact me and I’ll be glad to help you adapt this architecture to your needs.
If the MES does not support OPC then you should talk to the MES vendor and find out what interfaces they do offer. Some may support ODBC database connections or have a custom API. If either of these are the case we have ways to interface with both ODBC databases and custom applications using .NET, Java and C++.
I have a custom application that I’ve written that connects to my OPC data sources and I need to setup redundancy in that client application, what do I need to do?
Marc: In this case you have a couple of choices. You could in your custom application write logic to detect a failure in communications and switch over to the backup OPC server. You would basically be doing the same thing an HMI/SCADA system does. Most OPC Servers will sit idle until the client connects to request data, handling the switch in your client insures that your devices are not overwhelmed by requests from multiple servers.
Keep in mind that if you don’t activate the backup OPC server to start scanning the devices until you switch to the backup OPC server, your time to see initial data from the new server will increase the length of the perceived “bump”, a topic we discussed in our post Demystifying Redundancy in Automation. The only way around that is to have both OPC servers polling devices at the same time. We have seen some users have the backup server poll the devices at a lower scan rate, and then just increase the scan rate when you failover to the backup. This can all be accomplished from your OPC client application’s custom code. If you are using our OPC Data Client toolkit in your custom application, we can show you how to do this, just contact support for help.
Your other alternative is to put the Cogent DataHub Redundancy Add-On on the same machine as your custom OPC client application. Configure the Cogent DataHub to connect to your primary and backup OPC servers and then connect your OPC client to the Cogent Datahub instead of directly to the OPC servers. Adding the Datahub in the middle will only add milliseconds of latency as it’s capable of moving 100,000 points per second, so you most likely don’t need to worry about it slowing things down.
How do I decide how much redundancy I need?
Win: This should be a decision driven off of business factors, not just trying to achieve technical perfection at any cost. There needs to be a balance between the need for high availability and making the system too complex to maintain. In our post Demystifying Redundancy in Automation, we discuss about points of failure, bumps and consequences and how they help you decide how much redundancy is enough. For example, if restarting a process after failure is a long process that results in costly scrap and lost production time, then there may be cost justification for more complex redundancy.
I need to go all the way and have few or no points of failure in my OPC software redundancy setup, what are my options?
Win: So to go to the point of no single point of failure can get complex and expensive, but it is definitely possible. The scope of insuring no single point of failure clearly goes beyond just your OPC server, client, and redundancy software. It will mean redundant everything all the way down to your servers, power supplies in those servers, battery backups, power feeds for those backups, etc. We have clients that have gone all the way or close to it, and are used to having those conversations if your business justifications says it must be that way.
We’re happy to discuss alternatives with your team, you and your system integrator team, and dive deep on technical details while also being able to insure the business side is considered in weighing various options from the software part of the solution – in other words we want to provide a reliable solution while keeping in mind that your funds are not be unlimited.
Can there be such a thing as too much control system redundancy?
Marc: The engineer in me wants everything to be perfectly redundant with no single points of failure. But my business mind says yes you can go too far. What does ‘going too far’ look like? ‘Too far’ means the benefits of having a redundant system are outweighed by the complexity to the point that it is hard for your to team understand how the system works and keep it running properly. You could also go too far if you spend more on a redundancy configuration than the costs or revenue loss that multiple downtime periods from lack of redundancy could cause.
Like Win said, we encourage our clients to make informed business decisions based on operational priorities and needs. We’re happy to dig deep in technical details, brainstorm ideas for specific needs, help them understand what different options might be possible, what they cost, but ultimately advocate making balanced decisions.
If you’d like to learn more, here are a couple of related blog posts you might want to read:
