Features, discussions, tips, tricks, questions, problems and feedback

The importance of time synchronization (Worked example)

Description of the problem:

A customer reported a problem where a count that was being logged by a DBLog agent seemed to be logging out of order in no discernable pattern. And at very random times.

Notes:

  • The datascope was analysed, and was found the value updated sequentially in order , matching figure 2.
  • This driver (SIEMENS) does not send historical data (it does however execute time stamp on arrival).
  • It must be noted a value cannot stand alone in a scada-world it always does have a time component , whether it is time at source(PLC) if the protocol supports this, or time of arrival at the server.
  • Logging was configured for change only!

Customers normally extract from SQL in time order. Therefore if you examine figure 1 below, you will note that the 3rd column values 206 -209 are out of sequence.

image
Figure 1 : Log sorted by time. {IDX,Time,Value}

Now look what happens when the data is extracted in order of idx (Column 1) as per figure 2 below,
image
Figure 2 : Log sorted by idx {IDX,Time,Value}

From Figure 2 : we see that the values did get logged in order, but the time column seemed to go back in time between values 205 and 206.

Figure 3 below shows a timestmaped datascope

Using “/T” in the datascope, adds a {day hour:minute:Second} time-stamp prefix, encircled, for every message entering the datascope log , this is not part of the drivers protocol or any particular timer, it is derived from the machine system time.

Once can see in the datascope that the time jumped back 86 secs … this is indicative of time sync’ing having taken place on the agent server PC. Or the time may have been manually corrected.


Figure 3 : Time-jump encircled in yellow

Historical value updates:

Due to the possibility of any historical value having an important implication, and there is no way to accurately determine states like value or time dead banding the decision is made to allow it into the log regardless of current or previous state. This action results in the insertion of the “historical” value and timestamp pairs into the log, this only occurs for the period of the time adjustment, in this case 86 seconds , until the system catches up.

Therefore it is important to make sure computers in the SCADA system, are properly time synchronized, preferably to the same time source if possible.

This is especially important between cluster pairs, and any machine that requires a remote client connection.

Typically, telemetry protocols such as OPC (UA&DA), DNP3, Spectrum and any protocol that uses event or time at source need to be well synchronized, many protocols allow for the protocol driver itself to time synchronize the outstation hardware via protocol. This still requires that the machine be synchronized usually via NTP (Network time protocol), GPS (Global positioning system : Time services) etc.

VM’s can at times rely on their ESX host for time synchronization.

Resolution:

This worked example had an underlying VMware ESX host that had improperly configured Time Server.

2 Likes