Automating the processing workflow

Author: EIVA CEO Jeppe Nielsen

One result of the technological development seen in the maritime industry, along with the business model innovation among offshore and subsea specialists, is that we are seeing a continual increase in the amount of data used during survey operations. 

Especially when it comes to the addition of high resolution laser and photogrammetry data to survey operations, an extensive amount of data comes into play. Moreover, with operational setups running several autonomous vehicles in parallel, combined with clients expecting fast and cost-effective delivery of end-products, the task of processing the data in a timely manner without compromising on quality has never been more challenging.

Increasing amount of data processed in less time with the Workflow Manager

Starting in 2017 and continuing in 2018, EIVA is part of several leading-edge projects where the processing speed of massive amounts of data with minimum human involvement is in focus. This has resulted in a series of product enhancements, optimisations, and last but not least a new tool – the NaviSuite Workflow Manager.

The Workflow Manager is designed to enable a previously unseen degree of automatic data processing of massive amounts of subsea data. With this follow a number of key benefits:

• Faster data processing throughout your project
• Fewer data processing staff members required
• Better utilisation of your data processing staff’s time and competences

Consequently, the tool will improve your data processing efficiency as:

• Repetitive tasks are automated
• Data processing staff can focus on errors, interpretations and quality control
• Tasks are executed in parallel, that is, the full power of the computer is used
• Many features normally carried out in NaviEdit and NaviModel are not only automated but have also been performance optimised for use with the Workflow Manager

The result is a powerful cocktail of high-speed data processing features.

How does it work?

The Workflow Manager is a configurable workflow process automation tool. In other words, it allows you to automate the small steps you normally go through when processing subsea data. 

It can be applied to functions in NaviEdit and NaviModel, the NaviSuite products dedicated to navigation/positioning and data modelling/visualisation respectively (special versions of NaviEdit and NaviModel that run without a user interface are made available together with the Workflow Manager). The tool is also able to activate third-party scripts and tools. 

You define the sequence of processing steps and also which steps to run in parallel with other steps. With the setups we have seen so far, the task of defining your workflow is minimal – although it often gives you food for thought as to how you can optimise the way you work.

There are many types of steps available. A few examples are:

• Loading files, including waiting for files to become available
• Applying tide, SVP, etc
• Cleaning data in NaviModel
• Checking a number of quality control parameters such as density, gaps, noise levels, TVU, etc
• Correcting seabed height level across multiple surveys, etc
• Exporting data in different formats

• Current steps (blue): The currently running step(s) that are executed in parallel. You can define how many parallel tasks the Workflow Manager should use to ensure that it matches the capacity of your computer.
• Completed (green): The step has been completed successfully.
• Error (red): When an error is encountered, the processing of the specific step stops, and the operator can decide what to do by investigating the error in NaviEdit/NaviModel, manually change the data as necessary, and restart the automatic processing from a given step.
• User action (yellow): The tool can be set up to stop at a given point when carrying out a specific task. This is useful for processing steps that require manual quality control or steps that require operator-provided data or interpretation before the task can continue.

So how fast is it?

We are now seeing the first performance measurements on real-life operations data and workflow setups by our customers – and we are impressed (to put it mildly).

The performance measured is:

50 hours of data =  ~1.8 hours of processing

The customer behind this particular performance measurement is running six AUVs continuously, in parallel – processing all the data brought in by these with a single crew, as opposed to increasing the crew size accordingly.

With the Workflow Manager, a single data processor can handle data from 3-4 dives during a 12-hour shift.

EIVA was selected from among our competitors for this project because of our processing speed even before we made the Workflow Manager – the introduction of which only made the performance go up a notch or two.

The Workflow Manager is nothing short of a game changer in how we work with data. It raises the bar for how large amounts of data can be efficiently processed during collection. 

What about performance?

The Workflow Manager has a built-in performance measurement tool that measures the time spent on each processing step, so it is very easy to see how much time each part of the workflow takes, and consequently where it is worth optimising. 

As part of implementing the Workflow Manager, we noticed a number of optimisation possibilities in NaviEdit and NaviModel, which have of course been implemented as well:

• Faster loading in parallel into NaviEdit
• Efficient handling of AUV sound velocity profiles, which can be very large (for example 50-hour long) due to continuous recording during the entire mission
• Faster INS processing
• Faster DTM generation
• Faster TIFF exporting

The Workflow Manager has also given a lot of hints to an optimal system configuration and thus a high-speed processing machine, since the tool will maximise the use of CPU, GPU, disk and memory. 

These hints include:

• CPU: Many tasks are CPU heavy. Consequently, using several CPU cores will allow for more parallel processing. For a single machine, we recommend AMD Ryzen Threadripper 16/32 cores as the best value option right now. The latest Intel i9X series is also a good (but more expensive) choice. Server configuration can be made with much more cores available, however at a much higher price.
• Disk: Having the fastest possible disks is essential. The fastest currently available is the M2-Ultra (3500 MB/s compared to 550 MB/s of a normal SSD). We recommend adding M2-Ultra disks internally to your processing server to perform local processing on those disks. High-end SAN drives should give you a similar performance as an SSD (550 MB/s). Some NAS drives have very poor performance on random access read/write, so avoid using NAS for data during processing.
• GPU: Some NaviModel features such as EC-3D are GPU optimised, so add a graphics card with as many CUDA cores as you can afford.
• Memory: This is rarely critical (we have seen a max load between 32-37 GB) so 64 GB is enough.

How to get started with Workflow Manager

The Workflow Manager comes in two versions:

Workflow Manager – Desktop

• Can run one NaviEdit and one NaviModel task in parallel
• Available for free when purchasing NaviEdit Pro and NaviModel Producer licences and for existing users of the same software products who subscribe to the EIVA software maintenance and support programme (SMS)

Workflow Manager – Server

• Can run an unlimited number of NaviEdit and NaviModel workflow steps in parallel (typically 30-100), resulting in a significant parallelisation of processing for large data sets, and significant savings in terms of man-hours spent on data processing 
• Annual subscription price EUR 9,995 per licence

When to use the Workflow Manager

The Workflow Manager is applicable in many types of operations, for example:

• AUV/USV operations, where data is recovered at the end of each mission. This is the scenario where the Workflow Manager server solution and its ability to process hundreds of steps in parallel shows its real potential.
• Shallow water surveys (in combination with NaviSuite Kuda software), where sonar and terrestrial LiDAR data is processed as you sail. The Workflow Manager desktop variant will automatically process each sonar and laser file as it is recorded, thus completing most of the processing during acquisition, leaving quality control as the only task for the data processer to carry out onshore. For those customers streaming data to the shore, the Workflow Manager can run either onboard before streaming, or onshore receiving streamed data.
• Pipeline/cable route inspections (in combination with NaviSuite Nardoa software), where sonar, subsea laser, images and video data are acquired during ROV operations. The Workflow Manager Desktop edition can automate much of the processing tasks, leaving more time for the data processer to perform manual eventing and quality control. 

Want to know more?

The Workflow Manager is in operation with and being tested by selected customers before official release in Q1/Q2 2018. If you would like to learn more about the tool, please feel free to reach out to us.