Article Reproduce: Depth Synthesis and Local Warps for Plausible Image-based Navigation

This paper aims to solve the problem that: Given a dataset with multiple RGB images for a scene, how to generate images from other view points?

There are some techniques enable 3D reconstruction from images, but often fail on some parts where weren’t sampled enough and result in hollows. To address this problem, this article proposed a way by interpolation from similar neighbors. Specifically,

  1. Utilize existing 3D reconstruction methods to get a point cloud.
  2. Oversegment the input images and creat superpixels of homogeneous color content which often tends to preserve depth discontinuities. Therefore, we have reasonable estimation for poorly reconstructed regions.
  3. Wrap and interpolate pixels from images taken by nearby view points.

An example of superpixels and the similar neighbors(in green) of a superpixel(in blue):

Superpixels and neighbors

Some reconstructed images:

Optimize 3D reconstruction using Octrees

Visual Hull

  1. Using the contours and camera parameters to generate point cloud for a Visual Hull. Octree is used here to speed up the algorithm.

  2. For each points, find the neighbor point clouds, then calculate the normal vectors and decide the sign using minimum spanning tree.

  3. Generate the surface mesh

A image from the input dataset

A output model

Fingerprint Enhancement

  1. Segmentate the image to 8x8 blocks and run DFT to decide if they are parts of the fingerprint.
  2. Estimate the ridge direction and frequency.
  3. Smooth the ridge direction map and frequency map.
  4. Construct the enhanced fingerprint using the ridge maps.
    • The lines are connected again after enhancement
  • Works well even with noises

Estimate the speed of the football from a video

Object detection and Kalman Filtering

A example Input:

Football dectection:

Tracking result:

Max. Speed: 19.26m/s, Avg. Speed: 17.11m/s