Google’s Lens Blur Feature

Earlier today Google announced that they added a new feature to their android camera application called “Lens Blur”. From their blog post we can see that they take a series of images and solve for a multiview solution for the depth image of the first frame in the sequence. That depth image is then used to refocus the image. It is probably better described that they selectively blur an already fully focused image. This is similar to how they create bokeh in video games, however their blog states that they actually invoke the thin lens equations to get a realistic blur.

I thought this was really cool that they can solve for a depth image on a cellphone in only a few seconds. I also wanted access to that depth image so I can play with it and make 3D renderings! However that will come at another time.

Accessing the Data

Everything the GoogleCamera app needs to refocus an image is contained right inside the JPEG file recorded to /sdcard/DCIM/Camera after you perform a Lens Blur capture. Go ahead and download that IMG of the picture you took. How is this possible? It seems they use Adobe’s XMP format to store a depth image as PNG inside the header of the original JPEG. Unfortunately, I couldn’t figure out how to make usually tools, GDAL, read that.

So instead let’s do a manual method. If you open the JPEG file up in Emacs, right away you’ll see the XMP header which is human readable. Scroll down till you find the GDepth:Data section and copy everything between the quotes to a new file. Now things get weird. For whatever reason, the XMP format embeds binary containing strings of extension definition and a hash periodically through this binary PNG data you just copied. This doesn’t belong in the PNG and libpng will be very upset with you! Using Emacs again you can search this binary data for the http extension definition string and then delete everything between the \377 or OxFF upfront to the 8 bytes after the hash string. Depending on the file length you’ll have to perform this multiple times.

At this point you now have the raw PNG string! Unfortunately it is still in base64, so we need to decode it.

> base64 –D < header > header.png

Viola! You now have a depth image that you can open with any viewer.

Going back to the original XMP in the header of the source JPEG, you can find some interesting details on what these pixels mean like the following:

GFocus:BlurAtInfinity="0.036530018"
GFocus:FocalDistance="20.225327"
GFocus:FocalPointX="0.59722227"
GFocus:FocalPointY="0.5756945"
GImage:Mime="image/jpeg"
GDepth:Format="RangeInverse"
GDepth:Near="12.516363143920898"
GDepth:Far="47.468021392822266"
GDepth:Mime="image/png"

How did they do this?

There is no way for me to determine this. However looking at GoogleCamera we can see that it refers a librefocus.so which seems to contain the first half of the Lens Blur feature’s native code. Doing a symbol dump gives hints about what stereo algorithm they use.

librefocus.so:00221b68 D vtable for vision::optimization::belief_propagation::BinaryCost
librefocus.so:00221b88 D vtable for vision::optimization::belief_propagation::GridProblem
librefocus.so:00221bc0 D vtable for vision::optimization::belief_propagation::LinearTruncatedCost
librefocus.so:00221ca0 V vtable for vision::sfm::RansacSolver<vision::sfm::EssentialMatrixProblem>
librefocus.so:00221cb8 V vtable for vision::sfm::RansacSolver<vision::sfm::FundamentalMatrixProblem>
librefocus.so:00221c78 D vtable for vision::sfm::StdlibRandom
librefocus.so:00221b58 D vtable for vision::image::FixedPointPyramid
librefocus.so:00221c00 D vtable for vision::shared::EGLOffscreenContext
librefocus.so:00221800 V vtable for vision::shared::Progress
librefocus.so:00221be0 V vtable for vision::shared::GLContext
librefocus.so:00221cd0 V vtable for vision::stereo::PlaneSweep
librefocus.so:00221ce8 D vtable for vision::stereo::GPUPlaneSweep
librefocus.so:00221d20 D vtable for vision::stereo::PhotoConsistencySAD
librefocus.so:00221d00 V vtable for vision::stereo::PhotoConsistencyBase
librefocus.so:00221d60 D vtable for vision::stereo::MultithreadPlaneSweep
librefocus.so:00221ae8 V vtable for vision::features::FeatureDetectorInterface
librefocus.so:00221b00 D vtable for vision::features::fast::FastDetector
librefocus.so:00221d78 V vtable for vision::tracking::KLTPyramid
librefocus.so:00221a90 V vtable for vision::tracking::KLTPyramidFactory
librefocus.so:00221dd0 D vtable for vision::tracking::KLTFeatureDetector
librefocus.so:00221da0 D vtable for vision::tracking::GaussianPyramidFactory
librefocus.so:00221db8 D vtable for vision::tracking::LaplacianPyramidFactory

So it appears they use a KLT system that is fed to an SfM model to solve for camera extrinsics and intrinsics. Possibly they solve for the fundamental between the first sequential images, decompose out the intrinsics and then solve for the essential matrix on the remainder of the sequence. After that point, they use a GPU accelerated plane sweep stereo algorithm that apparently has some belief propagation smoothing. That’s interesting stereo choice given how old plane sweeping is and the lack of popularity in the Middlebury tests. However you can’t doubt the speed. Very cool!

Interesting GDAL Usage Examples

Despite first appearances, GDAL is way more than an image I/O library. I routinely use it resize images, transcode images, and reproject images. But did you know that it can also draw contour lines, write kml, and orthorectify RPC images? Even if you did, I think it’s good to document here because I keep forgetting.

Render Contour Lines

Publishing DEM’s created by ASP is a tricky because it is easiest visualize height change through a color gradient. That loses its usefulness when publishing to a conference where they print in gray scale. A snazzier option would be a hill-shaded render of a DEM with contour lines plotted on top. You can see an example on right and the commands below:

gdaldem hillshade <DEM> output.tif -z 0.000012
gdal_contour -i 50 <DEM> contour
gdal_rasterize -burn 0 -l contour contour output.tif

The “-z” value on gdaldem hillshade requires some playing around to figure out what is best for your image. The “-i” value on gdal_contour is the interval between lines in units of height defined by the DEM (for ASP products, it’s meters).

KML Outline of a Cube File’s footprint

Rendering of KML outputEver wanted to provide context for your ISIS cube file image on Google Earth? These commands allow you visualize the image on right. The first two lines are ISIS commands.

footprintinit from=<ISIS Cube File>
isis2gml from=<ISIS Cube File> to=output.gml label=footprint
ogr2ogr -f KML output.kml output.gml

Orthorectify an RPC image on a DEM

This one doesn’t apply to planetary imagery, but most Earth images tend to have an RPC camera model attached to them. Download a DEM from USGS’s NED, and then with GDAL you can drape your source imagery over the DEM.

 gdalwarp -of GTiff -co TILED=yes -co COMPRESS=lzw -r cubicspline \
    -rpc -to RPC_DEM=<your DEM> <RPC image> output.tif -dstnodata 0

Building Ames Stereo Pipeline against ISIS on Ubuntu 10.04

This is a guide for advanced bearded users. If you don’t have a beard, don’t try this at home! These instructions will also work for OSX minus the package manager.

Ames Stereo Pipeline is an open source collection of tools for building 3D models from NASA’s planetary satellites. Our software is able to do this by depending on USGS’s ISIS for all the camera models. That saves me a lot of time because I no longer have to program up custom models for all the many cameras that are out there (MOC, HiRISE, MDIS, LROC, ISS, and etc ). The downside is that building ISIS is next to impossible as they expect you to use their binary releases. This means that to compile against their binaries, we must recreate their development environment, down to every third party library.

There are some base tools that you need installed on your Ubuntu Box.

sudo apt-get install g++ gfortran libtool autoconf   \
   automake ccache git-core subversion cmake chrpath \
   xserver-xorg-dev xorg-dev libx11-dev libgl1-mesa-dev \
   libglu1-mesa-dev libglut3-dev

Building an ISIS environment is incredibly difficult to do by hand. Never-mind the difficulty in sanitizing the bash shell so that it doesn’t find libraries in odd places. So a co-worker of mine created an awesome collection of scripts to make this easier. It’s called Binary Builder and it’s available on Github. The commands below checkout the scripts from Github and then run them. What BB is doing in this first step is downloading and compiling all of the dependencies of Vision Workbench and Ames Stereo Pipeline. This means we’re building Boost, GDAL, Zip, OpenSceneGraph, LAPACK, and many others. As you can imagine, this step takes a long time.

cd ~; mkdir projects; cd projects
git clone https://github.com/NeoGeographyToolkit/BinaryBuilder.git
cd BinaryBuilder
./build.py --dev-env

Most likely things will die at this point. Here is where your bearded powers are to be applied! Good luck. When you fix the bug or think you’ve worked it out. You should use the following command to quickly restart.

./build.py --dev-env --resume

You’ll know you’ve had a completely successful ./build.py session when it prints “All done!” and gives you a list of the environmental variables used. Next, let’s clean up by making a BaseSystem tarball.

./make-dist.py --include all --set-name BaseSystem last-completed-run/install

This tarball will house all the headers, libraries, and a copy of ISIS that you need to proceed. It will be your lighthouse when everything else fails. You can also share this tarball with other users who have similar systems. Anyways, it’s time to deploy this BaseSystem tarball into a permanent position.

mkdir ~/projects/base_system
./deploy-base.py BaseSystem-*.tar.gz ~/projects/base_system

Installing Vision Workbench

You’re ready for step 2. This is all pretty straight forward. However you should notice that the deploy-base script produced config.options for both Vision Workbench and Stereo Pipeline. A config.options script is just another way to feed the arguments to ./configure.  When we install Vision Workbench, the base options in config.options.vw should be fine for us.

cd ~/projects
git clone https://github.com/visionworkbench/visionworkbench.git
cd visionworkbench
cp ~/projects/base_system/config.options.vw config.options
./autogen && ./configure
make -j <N Processors>
make install
make check -j <N Processors>

All unit tests should pass at this point. If not, bearded you knows what to do.

Installing Ames Stereo Pipeline

cd ~/projects
git clone https://github.com/NeoGeographyToolkit/StereoPipeline.git
cd StereoPipeline
cp ~/projects/base_system/config.options.asp config.options

We’re going to take a moment to deviate here. At this point you will need to make some modifications to your copy of ‘config.options’ for Ames Stereo Pipeline. You need to modify the variable ‘VW’ to be equal to the install (prefix) path that you used. In this example, it should be set to ‘~/projects/visionworkbench/build’. You can also take this time to flip other bits that you find interesting. For example, there’s a ENABLE_MODULE_CONTROLNETTK variable that you can set equal to ‘yes’ which would enable prototype utilities to manipulate control networks. Once you’re done playing around, finish your build of ASP.

cd ~/projects/StereoPipeline
./autogen && ./configure
make -j <N processors>
make install

You can also run ‘make check’, you just need to have your ISIS3DATA set up. You can fall back to your own install of ISIS and everything should work fine. If it wasn’t clear before, you’ll find the executables in “~/projects/visionworkbench/build/bin” and “~/projects/StereoPipeline/build/bin”. That’s all folks, I hope everything worked out okay.

MER Data Introduction

Getting data from the NASA Planetary Data Services (PDS) can be a little intimidating. The most import thing a person needs to understand about the MER data is that it is sectioned into individual sites that the rovers visited during their trip. These sites are localized on interesting features like an individual bay into a large crater or a particularly large bolder. These sites are further subdivided into sols (Martian days) at the site. When we search for images we limit them by both sites and sols.

Gathering Data

It is now time for us to select a site for which we want to render in 3D. I find MER’s Analyst’s Notebook as a good tool for this task. Here’s what I did:

  1. Go to: http://an.rsl.wustl.edu/mer/
  2. Click on “Opportunity”.
  3. To find a site: click on the “Map” icon on the toolbar, and then click on “Traverse Map”.

Each one of the black dots represents a site location. The number after the slash is a sol number. I compared this map to Google Earth (GE) and decided that I wanted to render “Cape Agulhas”. You can do that too in GE if you click on the flag icon for a MER and select “load rover way points”. Just note that the map on GE doesn’t show as much information as the traverse map on Analyst’s Notebook.

  1. On the “Traverse Map”, click “91/1673”.
  2. Select only Navcam data products on the left.
  3. On the Data Products drop down box, select “Show for sol 1674”
  4. Click “Redraw Map”.

From here we can see that the Navcam did a full panorama. We are now ready to start downloading data from PDS. You can download from Analyst’s Notebook, however I was only able to download a single image at a time. On NASA PDS we can download a ‘wget’ script that will download all Navcam images for us in one go.

  1. Go to NASA PDS’s image search for MER, http://pds-imaging.jpl.nasa.gov/search/search.html#QuickSearch
  2. On the left, under “Select Instrument(s):”, select “NAVCAM.
  3. Next to “Instrument Host ID”, select “MERB/MER1/Opportunity”.
  4. Next to “Product Type”, select “EDR”.
  5. Next to “Image Type”, select “Regular”.
  6. Next to “Eye”, select “LEFT”.
  7. Next to “Planet Day Number”, type 1674 for both the Min and Max text boxes.
  8. On the left, click “Get Results”.

You now have a listing of all the Left EDR NAVCAM. We want all of these; so on the left select ‘WGET’ under download products. Then click download. Move your downloaded ‘atlas_wget_script’ to your work directory. Here is how you start the download in the terminal:

cd $YOUR_WORK_DIR
source atlas_wget_script

Later on I can show how to produce 3D maps from this imagery. However this seems like a good start.

Installing Vision Workbench on Ubuntu

It seems everyone runs into trouble when installing Vision Workbench. Here’s a quick outline on how to get my favorite software on my second favorite platform. We’ll need to start with downloading all the dependencies through Ubuntu’s package manager called Aptitude.

sudo apt-get install build-essential automake libtool
             libboost1.42-all-dev libproj-dev git
             git-completion liblapack-dev ccache

The above command gets you most of the way. There is however one more dependency that we require. We need GDAL, which is a wonderful library for interfacing with many different file types and their geographic information. However Ubuntu only provides an old 1.6 version where Vision Workbench requires the latest and greatest 1.7.

Getting the newer version of GDAL requires an additional repo provided by UbuntuGIS. This can be achieved easily by doing the following:

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
sudo apt-get update
sudo apt-get install libgdal1-dev gdal-bin

If you happen to be one of my unfortunate interns (Muhawahaha) working from a laptop and stuck behind a government firewall. Here’s what Josh has to say about that:

However, if you are on the ARC-WLAN-GUEST network, port
11371 will most likely be blocked which will cause the above com-
mand (and programs like ping) to time out. If this is the case, the
First, add the line ”http://ppa.launchpad.net/ubuntugis/ubuntugis-
unstable/ubuntu lucid main” to your repository. Then, go to
the website http://wwwkeys.eu.pgp.net/ and search for the key
0x314DF160. Copy the text that appears into a local file. Then
apt-key add filename-here. Now you should be able to get the latest gdal. Run apt-get’s update and then install libgdal1-dev.
Alright! We are now ready to get Vision Workbench which as of May is available via a GitHub account. If you want the last stable version, download the source tarball from the official site. But instead I’ll show you how to download the current development code with Git. If you don’t have any plans of ever trying to modify Vision Workbench you can do the following.
git clone git://github.com/visionworkbench/visionworkbench.git
          VisionWorkbench

Others with ambitions of contributing back to the software will need to create a GitHub account and then proceed to fork Vision Workbench. Before you create a GitHub account, if you are unfamiliar with Git then you should read the first 2 chapters of ProGit. Proceed to GitHub and follow all the instructions until you have your ssh-keys setup. Then you’ll want to follow their instructions for how to fork a project. In the future when you want to contribute back to the group it will be performed with a pull request.

From inside your checked out copy of Vision Workbench in the first directory you’ll want to create a config.options file. In this file you’ll want to following contents.

ENABLE_DEBUG=yes
ENABLE_OPTIMIZE=yes
PREFIX=`pwd`/build/
ENABLE_CCACHE=yes
ENABLE_RPATH=yes
ENABLE_MODULE_MOSAIC=yes
ENABLE_MODULE_CAMERA=yes
ENABLE_MODULE_CARTOGRAPHY=yes
HAVE_PKG_GDAL=/usr
PKG_GDAL_CPPFLAGS=`gdal-config --cflags`
PKG_GDAL_LIBS=`gdal-config --libs`

Finally you are ready to compile and install Vision Workbench.

./autogen
./configure
make install

Vision Workbench is now installed in the build directory where the source code is checked out. You’ll probably want to point the environmental variable PATH to the build/bin directory.

If you still have spare cycles, you should run the test suite to find out if Vision Workbench is installed correctly. This is achieved with:

make check

You are now finished. It is now time to party.