Automating the processing workflow
Author: EIVA CEO Jeppe Nielsen
One result of the technological development seen in the maritime industry, along with the business model innovation among offshore and subsea specialists, is that we are seeing a continual increase in the amount of data used during survey operations.
Development in the typical data sets used during survey projects
Increasing amount of data processed in less time with the Workflow Manager
- Faster data processing throughout your project
- Fewer data processing staff members required
- Better utilisation of your data processing staff’s time and competences
- Repetitive tasks are automated
- Data processing staff can focus on errors, interpretations and quality control
- Tasks are executed in parallel, that is, the full power of the computer is used
- Many features normally carried out in NaviEdit and NaviModel are not only automated but have also been performance optimised for use with the Workflow Manager
How does it work?
- Loading files, including waiting for files to become available
- Applying tide, SVP, etc
- Cleaning data in NaviModel
- Checking a number of quality control parameters such as density, gaps, noise levels, TVU, etc
- Correcting seabed height level across multiple surveys, etc
- Exporting data in different formats
- Current steps (blue): The currently running step(s) that are executed in parallel. You can define how many parallel tasks the Workflow Manager should use to ensure that it matches the capacity of your computer.
- Completed (green): The step has been completed successfully.
- Error (red): When an error is encountered, the processing of the specific step stops, and the operator can decide what to do by investigating the error in NaviEdit/NaviModel, manually change the data as necessary, and restart the automatic processing from a given step.
- User action (yellow): The tool can be set up to stop at a given point when carrying out a specific task. This is useful for processing steps that require manual quality control or steps that require operator-provided data or interpretation before the task can continue.
So how fast is it?
What about performance?
- Faster loading in parallel into NaviEdit
- Efficient handling of AUV sound velocity profiles, which can be very large (for example 50-hour long) due to continuous recording during the entire mission
- Faster INS processing
- Faster DTM generation
- Faster TIFF exporting
- CPU: Many tasks are CPU heavy. Consequently, using several CPU cores will allow for more parallel processing. For a single machine, we recommend AMD Ryzen Threadripper 16/32 cores as the best value option right now. The latest Intel i9X series is also a good (but more expensive) choice. Server configuration can be made with much more cores available, however at a much higher price.
- Disk: Having the fastest possible disks is essential. The fastest currently available is the M2-Ultra (3500 MB/s compared to 550 MB/s of a normal SSD). We recommend adding M2-Ultra disks internally to your processing server to perform local processing on those disks. High-end SAN drives should give you a similar performance as an SSD (550 MB/s). Some NAS drives have very poor performance on random access read/write, so avoid using NAS for data during processing.
- GPU: Some NaviModel features such as EC-3D are GPU optimised, so add a graphics card with as many CUDA cores as you can afford.
- Memory: This is rarely critical (we have seen a max load between 32-37 GB) so 64 GB is enough.
How to get started with Workflow Manager
Workflow Manager – Desktop
- Can run one NaviEdit and one NaviModel task in parallel
- Available for free when purchasing NaviEdit Pro and NaviModel Producer licences and for existing users of the same software products who subscribe to the EIVA software maintenance and support programme (SMS)
Workflow Manager – Server
- Can run an unlimited number of NaviEdit and NaviModel workflow steps in parallel (typically 30-100), resulting in a significant parallelisation of processing for large data sets, and significant savings in terms of man-hours spent on data processing
- Annual subscription price EUR 9,995 per licence
When to use the Workflow Manager
- AUV/USV operations, where data is recovered at the end of each mission. This is the scenario where the Workflow Manager server solution and its ability to process hundreds of steps in parallel shows its real potential.
- Shallow water surveys (in combination with NaviSuite Kuda software), where sonar and terrestrial LiDAR data is processed as you sail. The Workflow Manager desktop variant will automatically process each sonar and laser file as it is recorded, thus completing most of the processing during acquisition, leaving quality control as the only task for the data processer to carry out onshore. For those customers streaming data to the shore, the Workflow Manager can run either onboard before streaming, or onshore receiving streamed data.
- Pipeline/cable route inspections (in combination with NaviSuite Nardoa software), where sonar, subsea laser, images and video data are acquired during ROV operations. The Workflow Manager Desktop edition can automate much of the processing tasks, leaving more time for the data processer to perform manual eventing and quality control.
Want to know more?
Summer sale of VideoRay ROV
As a supplier of VideoRay ROVs, one already offering highly competitive prices, we’re happy to highlight thisJuly 13 - 2018
EIVA to support new hydrography course established at the University of Gothenburg
EIVA is always happy to hear about new academic courses involving ocean sciences, but we’re even happier whenJuly 5 - 2018
EIVA ramp up ScanFish production to meet demand
With record levels of demand for our flexible ROTV product range, there’s not a lot of down-time on the producJune 25 - 2018