Leonardo: the retired tamping machine that sees everything

Strukton is responsible for seven performance-driven maintenance (PDM) areas and is therefore maintaining many hundreds of kilometres of track. Not only the tracks, but the overhead lines, the signalling systems, the verges and the signage; everything on it and attached to it. This is why we like to know the condition of the tracks and everything surrounding the tracks. Of course you can go there and take a look for yourself, but you cannot just drive from the office to Leeuwarden or Maastricht, for example, for every little thing. This is why, in the past, we preferred to make use of the images produced by Eurailscout. But because of the heavy railway traffic during the day, Eurailscout had to start operating during the night. And of course on darker images it is impossible to see the condition of fencing and whether there are any overhanging branches.

Google Street View for railway tracks

This is how I got to think: wouldn’t it be nice if we had our own version of Google Street View, but then for the railway tracks. Strukton Rail View in other words. I proposed it to our managing director Tjark de Vries, who not only responded enthusiastically, but also put me into contact with someone who had a 360-degree camera. We put it on a locomotive and went touring for a day. It turned out to work well, although the resolution was too low for zooming in. We therefore put an ordinary camera beside it. This way we imaged many hundreds of kilometres of track over a period of eighteen months. Later on we mounted the cameras on an old tamping machine and this became Leonardo. The images proved to be ideal for colleagues involved in greenspace maintenance. The images enabled them to perfectly inspect the verges, ditches and trees and perform maintenance where necessary.

Geoconda

One thing for which we did not have a solution right away, was how to provide effective access to all of these images. Half an hour of filming produces approximately 30 gigabytes of data. How do you provide users with easy access to all of this data? You would have to load the video and then move a slide to try to get to the right point within the video. Not exactly user-friendly! This is when two very smart Strukton Systems colleagues developed a program for this purpose: Geoconda. This is a GIS map to which the video images are linked. This makes it possible to very easily retrieve images for a specific location or for a section of track. In addition, you can easily switch between cameras.

Point Cloud

A number of cameras has since been added to this at the request of various colleagues. For example, Signalling asked for cameras low to the ground, Tracks wanted details about the track structure and Overhead Lines wanted cameras pointed upwards. In addition, Trackscan photos that provide top views of the tracks will also be added to Geoconda. These photos are connected together, thus creating a video. In addition, we added a laser to a 360-degree camera. The laser creates a point cloud that can be used to perform measurements. By superimposing this point cloud over images of an ordinary camera you can see what you are measuring, for example the distance of a branch from the overhead lines. These images have not yet been added to Geoconda. This requires a separate software program and providing all users with access to this program represents a significant investment. Preferably we would like to make all the data we collect accessible via GIS. That would probably mean that we need to develop this software ourselves, so that other data, such as radar data and Trackscan data, would also be available in GIS.

Banging

Sound is something else that should not be overlooked. Because when you walk along the tracks you not only see all kinds of worthwhile things, what you hear can also be of importance. For example, is there any banging when a train drives over a switch or a crossing? We conducted various trials with microphones on Leonardo, but this did not work. A Doppler effect is created because we are driving at very high speeds. As a result what you hear on the recording is not what you would hear if you were standing next to the tracks. The images themselves also do not display any bumps, because the cameras are equipped with image stabilisers. Fortunately, the Trackscan indicates whether the tracks are level.

Today we are driving Leonardo three times each year through all of our PDM areas: in spring, summer and autumn. I expect we will be building a second machine given the demand for increasingly more images. For example, the Strukton Rail Projects team would like images of certain areas outside our PDM contracts, so at one point it may very well be decided that we will produce those images once a year. External parties have also expressed interest. We will then use the second machine for production and the first machine for experimentation.

Machine Learning

There are enough wishes and ideas. For example, we are thinking about ground radar that can image the ground below the tracks using sound waves. Then you would not only know what type of soil it is, but also whether there are any cables, pipelines and explosives buried in it. We are going to run a trial with this in the near future. Thermal cameras could also be very interesting. When a cable gives off too much heat, it could be a forewarning of a potential malfunction. Another interesting possibility is a plant-recognition tool. Trainees had an app that you can use to identify plants and used it to scan Leonardo’s images. This turned out to work very well.

More than enough left to innovate

The ultimate goal? Image recognition using machine learning. We then no longer review the images ourselves, but give Geoconda a command: for example, show me all locations with damage to fencing or all sleepers with cracks greater than five millimetres. There is still more than enough left to innovate.