DataHub V10: Integrating Data Sources with AVEVA Insight

Cogent DataHub® (from our developer partner/publisher Skkynet) provides a wide selection of connectivity options to OPC DA, OPC UA, OPC A&E, Databases, Excel, ODBC, DDE, Linux, Modbus.  As you may have already learned recently, DataHub Version 10 now includes support for integration with a wide range of common external historians. This expands your options for which data sources are compatible for historization in your chosen historian solution beyond the historian's native interfaces.

In this blog post, we'll detail how to get started integrating your data sources accessible by DataHub with AVEVA™ Insight, by writing data to your AVEVA Insight instance while also making that same historical data available for trending, visualization and reporting for both DataHub and external historical clients supporting OPC UA Historical Access (HA). 

Part of the AVEVA™ Connect platform, AVEVA™ Insight, while acting as a cloud-based historian, goes beyond that with functionality for monitoring and analyzing asset performance, Operational Equipment Effectiveness (OEE), big data applications and more. And with publishers to most other common AVEVA (formerly Wonderware) solutions, users can bring all facets of their enterprise together for comprehensive decision making. And DataHub V10's ability to integrate data sources beyond those already accessible by Insight expands the data source options for Insight.

And even further beyond just expanding data sources, DataHub V10 provides AVEVA Insight users with the following additional capabilities:

• Store and Forward support to eliminate loss of data when there is a disconnect with the historian
• Secure tunneling of historical data for enterprise-wide integration
• Access to historized data from external OPC UA HA capable client applications

Separately, those features all contribute to a robust and wide ranging data source for AVEVA Insight. When those features are combined, however, DataHub V10 takes your AVEVA Insight projects even further.

This post specifically covers a basic configuration for getting DataHub connected to your AVEVA Insight instance for basic historization and writing/reading data to AVEVA Insight from your process sources configured in DataHub.

DataHub Data Sources for Historization

The first step for integrating data with any of the external historians supported by DataHub, including AVEVA Insight, is having the data from your source or sources available in DataHub. Whether that is data from an OPC UA or DA client or server, a DDE client or server, a database, an MQTT client or broker, a remote DataHub, a Modbus server, or one of the other many data sources DataHub supports, once it's in DataHub, it can be historized.

For example, towards the purpose of compiling the steps below, my DataHub V10 is sourcing data from OPC UA servers and storing that data in a DataHub data domain named "ToAVEVAInsight". Regardless of the specific source, if the data is in a data domain in DataHub, the remaining steps in this post will help you get that data into AVEVA Insight.

Details on configuring specific data sources is beyond the scope of this post but our DataHub how-to video resources available here can help you get your data sources setup and ready for historization.  You can also run DataSim or DataPid (installed with DataHub) to generate simulation data to test with.

How to write DataHub data sources to AVEVA Insight

1. Confirm Your Log-in Credentials for Accessing AVEVA Insight

Cogent DataHub V10 makes it easy to get your data into Insight as a data source. All you need is your log-in credentials for accessing your AVEVA Insight account to get started.

You don't actually need to log-in to Insight yet (though you certainly can). We just need those credentials for configuring the Insight historian connection in DataHub.

2. Configuring AVEVA Insight connection settings in DataHub

Assuming you have the data you'd like to historize in DataHub, the next step is configuring DataHub for the connection to your AVEVA Insight instance where you'd like to historize your data using the information you just gathered. For the purposes of this post, we are using an AVEVA Insight free trial but the same steps apply to fully licensed Insight accounts.

To start configuring a new AVEVA Insight connection, we go to the External Historian configuration section in DataHub and click the "Add" button:

Then select "AVEVA Insight" from the Historian Type dropdown menu:

The key settings of interest here, for a basic connection, are in the Connection Settings and Storage Settings sections.

The Connection Settings are:

• Label - this is an internal name for this specific connection to AVEVA Insight - if you'll be connecting to multiple instances of AVEVA Insight, it is recommended to use a logical description for this label.
  
  
• Connection Key - this section is the most important part of connecting to AVEVA Insight. Clicking the Elipses (...) button to the right of this field will launch the AVEVA Insight Publisher for DataHub and allow you to enter your region, log-in credentials and other details for your Insight account and the organization of the data you will be historizing.
  
  
  
  • The AVEVA Insight Publisher, after connecting to the AVEVA Connect service, will prompt you to select your specific region. This is due to Insight being hosted in multiple regions globally. Options include North America, Europe and Australia. Select the region based on which option is closest to you. (For this example, we selected North America.)
    
    

• • You're then prompted to confirm if you'll be logging in with an Existing account or a New account (if you don't currently have an account, but want to sign up for a 45-day free trial for AVEVA Insight, you can select New and proceed through that process). We'll select Existing here, since we already have a trial account.
    
    

• • You're then prompted to enter the username and password credentials for your AVEVA Insight account.
    
    

• • You'll then name the Data Source (this corresponds to Assets in AVEVA Insight) where your DataHub data will be logged in AVEVA Insight. If the Data Source doesn't already exist, it will be created for you. If it does already exist, you'll be given options to decide if you want to replace the existing source, modify the name you used or share the Data Source with other applications.
    
    

• • For our purposes, we're defining a new Data Source specific to DataHub and will enter a representative name "DataHub2AVEVAInsight".
  • Then you'll be prompted to specify a Location for your Data Source. You can select an existing location, if one already exists from previous configuration, or enter a new location.
    
    

• • Then you'll be prompted to specify optional Geolocation details about the location of the Data Source, including Latitude, Longitude and Elevation. (If you don't have or don't care about geolocation variables being associated with your asset in AVEVA Insight, you can just skip this and click Publish).
    
    

• • Then you'll click Publish to apply the previously specified details for the connection and your DataHub configuration will populate the required "Connection key" for connecting to Insight.

• API Bearer key (for queries) - a unique API bearer key created in the AVEVA Insight portal used with the API calls that DataHub makes for reading data out of AVEVA Insight. This is only necessary if you plan to read data from Insight.
  
  
  • To generate an API Bearer key, while logged into your AVEVA Insight account, click the Grid button at the top-right and select Administration.
    
    

• • Then click on the REST API button for Integration Settings.
    
    

• • Then click the + (Add) button to start adding a new API authentication key.
    
    

• • Enter a Name for the new API key and click Add (the Name should be representative of how the key will be used in your external application - for this example, we're using "DataHub").
    
    

• • You're then presented with the Token string - you can click the Copy button here, which will allow you to paste the string into the API Bearer key field back in DataHub. Click Close to finish.
    
    

• AVEVA Insight data source name - this is the AVEVA Insight data source that you wish DataHub to access for reads/queries. This is only necessary if you plan to read data from Insight. This should match the Data Source name you specified for logging/writes. To confirm data sources, you can log into your AVEVA Insight account, under Administration and access the Data Sources section and find the Data Source you want to read from and enter that asset here.
  
  
• Accumulation time (ms) - the number of milliseconds that DataHub will buffer data in memory before sending it to AVEVA Insight. Leave this field blank to send data as soon as it is available (this adds network overhead but minimizes latency).
  
  Specify a time here to accumulate data before transmission to improve network performance (adds latency to data availability in AVEVA Insight)
  
  
• Maximum # of buffered values - the maximum number of values that DataHub will buffer in memory before transmitting them to AVEVA Insight. Setting this value to zero will allow any number of values to be buffered within the accumulation time. If this limit is reached before the accumulation time has been reached, the currently buffered data will be transmitted.
  
  
• Read-only - do not write to database - Enable this setting to prevent DataHub from writing to this configured historian instance.

And the key Storage Settings to be aware of are:

• Storage Type - The type of storage on AVEVA Insight. The default option is Delta.
  
  • Not Stored - No values are stored
  • Cyclic - Only values that occur at a specified time interval are stored.
  • Delta - Only changed values are stored (Recommended)
  • Forced - All collected values are stored (useful if you need all values historized even if the value for a tag hasn't changed)

• Min store and forward duration (minutes) - the minimum number of minutes that data will be saved in the store-and-forward queue while waiting to be transmitted. Data that cannot be transmitted within this time may be discarded so it is recommended to specify a time period that provides ample time for the connection to be recovered if maximum data retention is desired. The default is 15 minutes. (NOTE: the longer the time frame, the larger the data queue will be which will impact hard drive space on the local machine)
  
  
• Store and forward free disk space (MB) - the minimum amount of free space that must be available on disk for store-and-forward to retain data. When/if free disk space drops below this value then stored data will be discarded without transmission.
  
  
• Store and forward path - the directory on the DataHub machine where data will be buffered prior to transmission according to the previously specified duration and amount of disk space.

Other settings can be left at the defaults. Suffice it to say, there are additional settings that allow you to fine tune what and how much data to transmit to AVEVA Insight, such as the Data Sampling settings which allow you to send samples or "snapshots" of the data for specified time intervals to the supported historian, rather than the full data set. This is useful for situations where you want to conserve space in Insight. However, if capturing all value changes is desired, sampling settings should be left at the defaults.

In addition, DataHub also supports Forwarding of the same dataset to a separate instance of AVEVA Insight as a backup, which is beyond the scope of this particular blog post.

Last, but definitely not least, at that bottom of the Edit Historian Connection dialog, is the Available Points section, which provides the ability to browse the available tags in DataHub and select the ones you want to historize in AVEVA Insight.

Simply expand the tree view where your desired tags are located and tick the box next to each one and ensure the desired points are added to the "Selected Points" section on the right. For this example, we're historizing a several points from two different OPC UA servers (TOP Server simulated data and OmniServer connecting to a TPI vibration monitor).

You can also click Add Pattern to specify a regular expression that matches one or many tag names.

Once you've entered the correct information, you can click "OK" and then make sure to click "Apply" back in the main Cogent DataHub interface to ensure that all of your settings take effect.

3. Confirming DataHub is Successfully Connected to AVEVA Insight

And, to confirm your DataHub is connected to AVEVA Insight, you can check the following:

• View Connections in Cogent DataHub - you can confirm the running status for the AVEVA Insight connection including statistics on sends and receives
  
  
  The resulting window includes all DataHub connections and allows us to confirm that DataHub is connected and transmitting data.
  
  
• Data Acquisition in the AVEVA Insight Asset Status Board - you can confirm that AVEVA Insight is successfully receiving updates from DataHub including how many values and for how many tags.
  
  
  
  From the Home page in AVEVA Insight, you can you the Search field to find your Asset or click the top-left hamburger menu and select Assets, then drilldown into the asset.
  
  
  Your point or points should be listed with current values in the Status Board in the center.
  
  

4. Verifying Historized Data from Cogent DataHub to AVEVA Insight

From the same Asset Status Board, you can also view a trend for any of the points that DataHub is now historizing. Simply click anywhere in the Status Board to be taken to the Numeric Data section.

At the bottom, you can specify the time period. Available trends will be shown to the right of each point - clicking one and then clicking the Trend button brings up the desired trend for that point. 

How to Read Historized DataHub Data from AVEVA Insight

Now that your DataHub data is being successfully historized in AVEVA Insight, it's a simple incremental step to make it possible to expose that same historized data (this is only possible for data that is being historized to AVEVA Insight by DataHub, i.e. read operations are not possible for historized data in AVEVA Insight from other sources).

NOTE: If you have more than one external historian configured in DataHub, it is only possible to select one as the read source for historical data.

First, you must have generated and entered an AVEVA Insight API Bearer Key (as detailed above) in the DataHub Connection Settings for Insight, as well as, specifying the AVEVA Insight Data Source Name (should match the data source that was specified for logging/writing to Insight).

Simply select your AVEVA Insight historian that we just configured for writes from the "Read values from this connection" dropdown under the Read from Historian section. Alternately, you can just check the box "Src" next to your desired AVEVA Insight instance under the Write data to Historians section.

Now the tags you're historizing can be used by historical clients including the internal DataHub clients QuickTrend and WebView and by external OPC UA HA capable client applications.

To quickly see this in action, we can use the DataHub QuickTrend to trend our historical data from AVEVA Insight. Just go to the QuickTrend section of DataHub and click "Open QuickTrend".

From the Data Point section at the bottom of the QuickTrend, simply select the desired Data Point or Points at the bottom that are part of the data being historized and, on the left under "X Axis" specify the desired "Time Span" and "Update (ms)" rate. You can also adjust the scale Minimum and Maximum for the "Y Axis"

And it's just that easy to historize all of your DataHub accessible data sources to AVEVA Insight. In future posts, we will cover the more advanced features that we discussed earlier such as secure tunneling and store-and-forward and how they help you ensure all of your vital process data is there for decision making. To get started integrating your own data sources with AVEVA Insight with the Cogent DataHub, the free trial version is available here.

And don't forget to subscribe to our blog to not only be notified of future posts on DataHub V10 and also to find out about more relevant technical topics applying to industrial process automation and solving related challenges.