Rig Camera Alignment & Exposure Balancing

The Overview and Try It sections provide a basic understanding of the process. Subsequent sections have more in–depth information and can be used as a reference. Step by step tutorial lessons are included.

Overview

The Rig Camera Alignment & Exposure Balancing process provides automatic band alignment and exposure balancing for imagery taken with a multi-camera rig system.

A multi-camera rig can acquire hundreds and thousands of image frames, each made up of multiple bands. Since each spectral band is taken by a separate camera, the resulting image bands are not correctly co–registered with each other. In addition, digital cameras are designed to automatically adjust shutter speed and ISO for each frame (based on the overall brightness of the scene), which means the resulting images have different exposures.

This process is designed to correct these problems in three operations: it performs multi-band alignment for each frame, exposure balancing from one frame to the next, and creates a contrast table for the set of images. The following analysis tools are available to help you set up the process and check the input and results at each step: View Track, View Images, View Histogram, and Image Band Correlation.

Processing of frames is extremely fast, typically requiring less than one second per frame after the initial alignment. A complete drone survey can be processed in the field on a laptop computer, typically in less time than the original flight. The process results in a corresponding output image for every input image. Note that resulting images are not georeferenced or mosaicked.

output

The Rig Camera Alignment & Exposure Balancing process showing the flight path.
The selected frame is represented by the red dot in the Track View window.

multi-camera rig — multiple, separate camera sensors on a UAS (drone) or piloted aircraft. Each camera is set up to capture a 'photo' for a specific spectral band (i.e red, green, or NIR wavelength range). The cameras are mounted together on a rigid frame and synchronized to take a picture at the same time. This set of photos taken at the same time is a single frame in the set of images acquired on the flight.

image — this process works with multi–band images, but generally speaking, an image for a given scene may be composed of one or more rasters for the same area on the ground.

frame — a single image in a series of images that were taken in sequence. Imagery is collected along a flight path and each image has a frame number that indicates the order in which it was acquired.

In the context of acquiring aerial imagery, a 'frame' may also refer to the physical structure the cameras are mounted on.

co–registered — raster bands for the same scene are co-aligned and features in them are correctly positioned relative to the other bands.

Try It

Prerequisite Skills: Displaying Geospatial Data and TNT Product Concepts.

Sample Data: Download the micasense_red_edge.zip file for use with the exercises included here. This data contain 15 frames from a MicaSense RedEdge camera rig.

Tutorial Lessons:

The following five lessons walk you through the process from start to finish.

  1. Load input
  2. Look at misaligned bands using Image View
  3. Create a new base alignment model using the selected frame
  4. Assess the accuracy of the base alignment model
  5. Run the process and assess the results

rig camera

Image frames loaded in the Rig Camera Alignment & Exposure Balancing process.

The basic steps to run the process are to: select the folder with input files, create or select the base alignment model, check if the model correctly aligns all the frames, and if so run the process.

Use the micasense_red_edge.zip sample data for the following exercises. See the Input section for information about acquiring imagery.

Exercise 1 – Load input

  1. extract the .tif files in the micasense_red_edge.zip file into a folder
  2. open the Rig Camera Alignment & Exposure Balancing process
  3. Tips: always select Model before input:

  4. select the Model: MicaSense RedEdge
  5. click the Open Folder button and select the folder with the .tif files
  6. look at the input frame list and notice frame number 0001 is selected
  7. click Track View and notice the frame number 0001 is indicated by a red dot
  8. in the main dialog, click on another row in the input list and look for the corresponding red dot in the Track View window
  9. close the Track View window
  10. select frame number 0001 again
  11. Frame number 0001 will be used to check alignment in the rest of the exercises as well. It has a building with straight edges that makes it easy to see how bands align.

    Optionally exclude frame(s) from being processed:

  12. in the input list, right–click on the Include in Output box (for any frame) and choose Include by Altitude Range
  13. look at the values in the Altitude column of the input list
  14. set the range to 213.00 to 214.00
  15. click the Include in Output box for frame 0001 to include it again

rig camera

Showing misaligned bands in input image. Zoom in and move mouse along feature edge.
Look at the different cross–hair positions in other bands (left). Displaying the same frame
zoomed out on three–band composite layers also shows misalignment (right).

(continued from previous exercise)

Exercise 2 – Look at misaligned input bands using Image View

  1. click View Images and notice the Image View window is in Preview mode and showing bands in the selected frame
  2. set it to show 4 views (Split View)
  3. optionally adjust which bands are showing in each view (via drop down list)
  4. zoom in on a feature edge (via mouse scroll wheel)
  5. adjust portion of the image you can see (via mouse click and drag)
  6. move cursor over the view with the NIR image and compare to the tracking cross–hairs in other views, notice they do not match up

(continued from previous exercise)

Exercise 3 – Create a new base alignment model using the selected frame

  1. in the main dialog with frame number 0001 selected, click Auto Align and name the base alignment model when prompted (it will take a short time to compute)
  2. look at Image View window and notice the Preview has been updated
  3. in the main dialog window, select another frame in the input list and examine it (then select frame number 0001 again)

(continued from previous exercise)

Exercise 4 – Assess the accuracy of the base alignment model

  1. in Image View, zoom in and scan around the image to verify that the co-registration is good for all bands
  2. use the Add option in the band list and add a composite layer to see how bands overlay each other in a single layer
  3. select a different frame in the input list and examine it (when done, select frame number 0001 again)

If bands are still misaligned, you have options to: exclude the frame from processing (via the Include in Output in the input frame list), recompute the base alignment model using a different input frame, or adjust the base alignment for each frame (via Refine alignment for each frame option). Then use Auto Align again to replace or create a new base alignment model.

rig camera

The Preview mode to check the base alignment model (left) and Last Processed Result mode shown
after running the process (right) have the same alignment correction. The difference you see is
that exposure balancing and the contrast table have been applied to the final result (right).

(continued from previous exercise)

Exercise 5 – Run the process and assess the results

  1. in the main dialog with frame number 0001 selected, click the Run button
  2. when prompted select where to put output folders: CRTIFF (.tif) and CRRVC (.rvc)
  3. The time it takes to run depends on the number of frames to process. When the process completes, the View Images window automatically changes to Last Processed Result mode. Also note, if you reload input after processing, the Last Processed Result mode is automatically available (as long as the output folder is still there).

  4. when it is finished running, compare Preview and Last Processed Result modes
  5. zoom in at 1X and scan around the image to verify that the co-registration is good for all bands
  6. click View Histogram to open the Histograms window (leave it open)
  7. click Image Band Correlation to open the Image Band Correlation window (leave it open)
  8. select another frame in the input list in the main dialog and notice the analysis windows are updated for the selected frame

Operation Details

Multi-band Alignment Operation

This operation resamples bands in an image in order to correctly overlay features and co-align the rasters. Specifically, one band is used as the master and the other bands are adjusted to match it.

Imagery acquired from a multi-camera rig will have bands (in a single frame) that are slightly offset or misaligned with each other. First, a base alignment model is figured using a reference frame. The resulting model contains the adjustments needed to properly align the bands from all of the cameras. When the process is run the bands are resampled using the base model.

The band alignment operation is designed to work on imagery collected over a flat area at the same height above the ground. Before running the process on an entire set of image frames, you need to create a base alignment model that is specific to the camera rig and the height above ground (i.e. within a reasonable range of height values).

base alignment model — a rig–specific and height–specific model created that is used to automatically align bands in a set of images.

master camera — the process sets one camera as the master and figures out what adjustments are needed to align bands from the other cameras to it. For example, on a MicaSense rig the master is the NIR camera, which has the widest band width and is the sharpest image.

The image you use to create the base alignment model is an important consideration. In fact, a planned flight to take reference 'photos' is ideal. Choose an area that is flat and that has a lot of sharp spatial features over the entire scene (such as the lines on a parking lot). Acquire a separate reference image for any height you plan to acquire imagery at. Models created from a reference image can then be used for all imagery taken at the same height using the same camera rig. After you have the base alignment models you can correctly align any imagery you acquire even if it has no distinctive features.

Tip: when saving the model include the camera height (above ground level) in the name.

Exposure Balancing Operation

The exposure balancing step corrects for variations in exposure times and/or ISO settings from one frame to the next. It also corrects for vignetting.

When acquiring an image band, the camera records the brightness values of a patch of ground for a specific wavelength range. However, the cameras automatically adjust the shutter speed and ISO from one frame to the next based on the overall scene (i.e. more bright or dark features). This results in different recorded brightness values for the same feature, which is not desirable when using raster values as measurements. (For example, imagine the same tree in two sequential frames, where one frame is much brighter overall than the other. The raster cell values for the same tree will not be the same in the two frames.) Thus exposure balancing corrects for this by modifying the cell values accordingly. This operation assumes the amount of light from the sun is the same for all frames.

Contrast Table Operation

Contrast tables are also created for the resulting images and are automatically used when displaying them. The contrast table is identical for all frames and is computed based on overall range of values from all frames.

Analysis — Examine Input, Base Alignment, and Results

Analysis tools are available for the user to check input, preview base alignment model results, and study the final results. Study the selected row (frame) in the input list via these analysis windows: , View Images, View Histogram, and Image Band Correlation. In addition the View Track option lets you see the flight path for all of the frames.

After creating or selecting a base alignment model, check that it correctly aligns the other frames in the input set. Use Image View with the Preview option and set it up to show multiple views (via Split View). Move cursor over the master band at feature edges and look at the corresponding crosshair positions over the other bands. Zoom in to verify the co-registrations are accurate.

Tip: display the master camera band in one view and compare it one–by–one to each of the other bands. Each band should be checked to make sure the alignment model correctly moves it relative to the master.

If the model results in a frame with misaligned band(s), you have options to: exclude the frame from processing (via the Include in Output in the input frame list), recompute the base alignment model using a different input frame, or try to adjust the base alignment for each frame (via Refine alignment for each frame option).

Tip: look at the altitude of the misaligned frame and see if it is within 20% of the intended height. If the height is not close enough to the others it will not correctly align the frame. Exclude frames at the start and end of your flight path if they have different altitudes.

Input

The process input is raw imagery taken from a multi-camera rig system, where each band is acquired with its own sensor and lens (camera). Any number of frames taken in a single flight can be processed at the same time.

Acquire imagery

Imagery should be acquired within a narrow height range. Individual frames outside of this range, such as at the beginning and end of the flight path, can be excluded from processing. An accurate height is based on the camera's height above the ground. Note, the altitude shown in the input frame list is the GPS altitude, which is not the same as the camera height above ground.

If you don't already have a base alignment model (computed for the same rig and height) be sure to include an image with good feature edges that can be used to create one.

Select input

Input files must be stored in the folder structure specified by the camera model chosen. Each rig-camera system has their own specification on how files are organized and named. This structure determines what file corresponds to each band and the image track order (frame number). Supported multi-camera rigs include models from MicaSense, Parrot, MAPIR, SlantRange and SAL Engineering.

If your rig camera system is not listed, contact MicroImages about the possibility of adding it. In general, only commercially available (i.e., not custom–built) rig–camera systems will be included.

Choose the camera Model, then Open Folder to load the imagery. Folders with imagery that are not from the selected rig Model will not be selectable.

Adjust input

After selecting the input folder, each input frame is listed in order by frame number, which is computed based on the order it is added. If needed, exclude a frame from being processed by turning off the Include in Output box near the frame number. The frame list includes columns showing EXIF information about each band in that frame including: longitude, latitude, altitude, exposure index (EI), ISO speed, exposure time (Exp), and black level (Blk).

The Settings icon lets you choose what columns to show (i.e. ISO speed and exposure time).

Results & Output

Controls are available to set the output band order and specify the object/file structure. Choose to output images as multi-layer or multi-page TIFF files, or with each band in a separate file. The output folder and file naming conventions are set via the Settings icon.

Two results folders:

CRTIFF – contains camera-corrected and co-registered set of .tif files and a new GPSdata.csv file.

CRRVC – contains camera-corrected and co-registered .rvc files with the raster objects (linked to the corresponding .tif file in the CRTIFF folder).

'CR' is the default naming convention, which is an abbreviation for camera-corrected and co-registered images.

Two output files:

RigAlignSettings.ini – retains settings for the last run, which is saved in the same folder as the input images. This allows loading previous settings when using the input again and may also be used to find corresponding output. This file is specific to each datasets. This step usually takes 1-2 minutes depending on the number of cameras.

GPSdata.csv – contains GPS data in a format for use within photogrammetry software (i.e. Pix4Dmapper Pro or SimActive Correlator 3D)

Interface

Rig Camera Alignment & Exposure Balancing window

Top Toolbar

output

The top toolbar of the main process window.

Open Folder – select folder with input files. Set the Model prior to selecting input.

Run – Use Run icon to generate output. After running, View Histogram and Image Band Correlation icons are available.

The following options (in top toolbar) open analysis windows for evaluating input and result imagery: View Track, View Images, View Histogram, and Image Band Correlation.

Model – select the camera model from the list, which includes:

  • Parrot Sequoia
  • Parrot Chlorophyll
  • MicaSense RedEdge-MX
  • MicaSense RedEdge-M
  • MicaSense RedEdge
  • MicaSense Altum
  • MAPIR Kernel (pre-rotated)
  • MAPIR Kernel

Auto-Align – used to create a base camera alignment model using the currently selected image. Use Image View to Preview the base alignment model on the selected image. Click Auto Align to replace or create a new base alignment model.



Settings

Result file prefix – add text to beginning of file name, default is 'IMG_'

Result file suffix – add text to end of file name, default is '_CR'.

TIFF subfolder name – containing results. Default is 'CRTIFF'.

RVC subfolder name – containing results. Default is 'CRRVC'.

Raster (link) object naming – name objects using Band Name, Band Abbreviation, or Band Code.

Band Subfolder Naming – name subfolder using Band Name, Band Abbreviation, Band Code, or Wavelength.

Single-Band File Suffix – append text to file name using Band Name, Band Abbreviation, Band Code, Wavelength, or Sequential options.

Automatically determine output folder from input – when off, the user is prompted to choose the output folder after selecting Run.

Transfer relevant metadata from source to output – Metadata includes: Dimensions in pixels, Bit depth, resolution (DPI), Camera maker and model.

Treat zero-value cells as null – if on, 0 cells will not be processed.

List ISO speed and exposure time – show columns in input list.

List Black Level if applicable – show column in input list.

Group duplicate columns by camera – useful for single-sensor (Bayer-pattern) cameras. It lets you avoid listing duplicate columns in the main window since the exposure settings are always identical across all bands. (For true multi-sensor cameras like MicaSense this setting will have no effect.)

Input Frame List

Each row represents a frame and shows columns with EXIF information for all of the bands (i.e. .tif files). Column backgrounds are color coded for quick band identification (i.e. red, green, blue, etc).

The listed metadata includes:

Include in Output box – it includes right–mouse button options to: Include All, Exclude All, and Include by Altitude Range.

Frame number – the sequence number for each input frame. It is set by the process in the order frames are listed.

Longitude / Latitude – longitude and latitude of the aircraft at the moment the image was captured.

Altitude – GPS altitude of the camera above mean sea level in meters.

EI – Exposure index, which is a combination of exposure time and ISO setting. EI values change as the drone flies over different features (i.e. bright sand and dark water).

ISO speed – ISO speed.

Exp – Exposure time.

Blk – black (Blk) values (dark current).

Parameters

Output Bands – by default there is an output for every input band. Select Other to specify the band number for each output band and/or choose to exclude band(s) from the output. output to file or folderMulti-Band in Single File, Multi-Page in Single File, Single-Band All In Same Folder, and Single-Band In SubFolders. Exposure Balance – automatically compensate for exposure differences between frames. Such differences can be due to automatic exposure adjustment on the camera when flying over bright versus dark surfaces. This improves matching of images while allowing the entire range of the sensor to be used.

Vignetting correction – automatic vignetting / flat field correction when appropriate coefficients are available.

Override Coefficient – enter a value between 0 and 1 to override (or supply if not available) the vignetting coefficient.

Adjust for variable lighting using reference band – compensate for variable lighting conditions using the selected band to adjust the others.

Refine alignment for each frame – used to adjust the base alignment for each frame. This fine–tunes each frame individually when the process is Run. Try this options if frames are taken at significantly different heights above ground. If on, process speed will slow down significantly.

Track View window

Show GPS point locations over a suitable reference image (or Bing layer) to show the track flown and aid in excluding the beginning and ending images if needed.

rig camera

Track View window with Bing reference image.
The red point indicates the selected input frame.

Image View window

View automatically contrast enhanced combinations corresponding to the selected input image. Assign a spectral band to each of the display color components (red, green, and blue). Up to four views can be shown at once.

Show Preview — preview the alignment correction using the selected base alignment model on the selected frame.

Show Last Processed Results — after running the process, shows result for the selected input image. Alignment adjustment will remain the same for the frame used to create the base alignment model, however, it will look different since the contrast is applied.

Compare preview (pre-processed) and processed (aligned) frames in any 3-band combination or grayscale, with up to 4 simultaneous displays. Zoom in to see if co-registration is accurate.

rig camera

Select a band or composite layer from the drop down list
or use the Add option to set up a new composite layer.

View Histogram window

Multi-band histogram with options to view a single selected input frame and/or the total histogram for all frames. When total and single histograms are displayed together, the Percent option for the graph's vertical scale gives you a way to compare them in the same graph.

rig camera

Histograms for each band of the selected image (bold line)
overlaid on the total for all images (fainter line).

Image Band Correlation window

Image Band Correlation for the bands in the selected input frame, with automatic tracking of cell values based on cursor location in the image view window.

rig camera

Image Band Correlation window.