DRONE DATA PROCESSING

Doing some drone data processing tonight and thought I would share a little snippet of what a densely reconstructed point cloud looks like from a flight that contained just over 300 images.

The concept of working on drone data has recently made waves in the geospatial sector, professionals have realized the reliability and cost-effectiveness of data collected using an unmanned aerial vehicle in a safe manner. Land surveyors, in particular, are getting exposed to more options that an unmanned drone is providing them.

Having said all that, a major factor of effectively using data collected by a drone is processing it efficiently. Gathering data using drone might just take a flight but processing it and making it actionable is yet another story.

Drone data is processed using the technique called Digital Photogrammetry. The word photogrammetry is derived from: photo — meaning “picture”, and grammetry — meaning “measurement”. It is a science of obtaining measurements such as length, area, volume etc from an image.

Processing information collected by an unmanned aerial vehicle is broadly a three-step process:

1. Structure From Motion (SFM)

2. Multi-View Stereo (MVS)

3. Images rectification

These processes can be further understood into following sub-processes.

Image Feature Extraction

Each drone image has a collection of unique features which differentiate it from other images. These are known as key points. Key points from each image are extracted using automatic computer vision algorithms (SIFT, BRISK, etc.) These features consist of the building’s corner, roads, edges, etc.

Generally, images with good texture variation have 40,000+ features. It can be easily understood why photogrammetry performs poor in areas of low texture variation like water bodies, dense forest, sand, sky etc. Keypoints extraction becomes difficult in textureless surfaces.

The density of features is very high in the leftmost and bridge region since it has a lot of edges, color changes, etc. Very few features are detected in sand and water regions due to textureless surfaces.

Feature Matching

Extracted features are then searched (in the nearby images) and matching is performed. Using GPS data to search for relevant images makes the matching process much faster and accurate. From matched features, fundamental matrix is derived and the relative position between two cameras is estimated. Techniques like Flann is often used to conduct search and match.

Bundle Adjustment (BA)

Relative position estimated from the fundamental matrix is generally prone to errors. BA is used to simultaneously refine the 3D coordinates (Lat, Long, Elevation), orientation parameters (Yaw, Pitch, Roll), and the optical characteristics (distortion parameters) of the camera(s) employed to acquire the images. BA is a nonlinear iterative optimization process where the objective function is Mean Reprojection Error (MRE) and parameters are the position, orientation and camera distortion coefficients. BA can be of two types – incremental or global.

Geotag data stored in images are used to georeference and scale the model.

Depth Map estimation and Point Cloud generation

Depth value is estimated for every pixel in the image using Multi-View Stereo algorithm. MVS algorithms are able to construct highly detailed 3D models from structured images. So, the output of SFM will act as an input to the MVS algorithm. It will output the depth map corresponding to every input image.

Individual Depth Map is fused together with the depth map of the neighboring

image to obtain a 3D point. These points are often called as the dense point cloud. It may even consist of greater than 1 crore points for a relatively smaller area.

Digital Elevation Model (DEM)

3D Points are triangulated and gridded in 2.5 Dimension space to create 2.5D Digital Elevation Model (Raster). Every pixel in raster has latitude, longitude and elevation information. Interpolation techniques like IDW are often used to do 3D point cloud to 2.5D grid/raster conversion.

Orthomosaic (HD Maps)

Orthorectification of each photo is done using DEM. Orthorectification step involves creating a visibility or occlusion map with respect to each image. These maps tell us which pixels are visible or occluded (not visible) from a particular image. Only visible pixels are then selected and color values are extracted. These orthorectified and occlusion-free photos are mosaiced together to create a large HD Map.

Once the processing is over; Post-processing techniques such as Image Blending, Color/Contrast adjustment are conducted to remove seam lines present on the boundary of images. Image blending uniforms the color and removes artifacts.