Load the Takeo image from the data directory using the auto importer.
%matplotlib inline
import numpy as np
np.set_printoptions(linewidth=500, precision=3)
import matplotlib.pyplot as plt
import menpo.io as mio
takeo = mio.import_builtin_asset('takeo.ppm')
takeo.view()
<menpo.visualize.viewmatplotlib.MatplotlibImageViewer2d at 0x10d780490>
The Takeo image is grayscale, but still has three channels:
print(takeo.n_channels)
print(takeo.pixels.shape)
3 (225, 150, 3)
These are redundant. An easy way to remove them is to explicitly convert the image to greyscale
takeo = takeo.as_greyscale()
print(takeo.n_channels)
print(takeo.pixels.shape)
takeo.view_new()
1 (225, 150, 1)
<menpo.visualize.viewmatplotlib.MatplotlibImageViewer2d at 0x10d874b90>
Since Lucas-Kanade involves matching images between a template and an input, we begin by creating a template to match to. Therefore, we import a warping function and warp the original image to create a template. Create the warp involves defining a set of initial warp parameters, which we will attempt to recover using the Lucas-Kanade algorithm. The warp parameters are defined as the elements of the transform matrix that transforms from target to source. We honour the parameter layout specified in the Lucas-Kanade 20 years on paper: \begin{pmatrix} p1 & p3 & p5\\ p2 & p4 & p6 \end{pmatrix}
where $p5$ and $p6$ are the translation parameters.
from menpo.transform import AlignmentAffine, Affine
from menpo.image import BooleanImage
target_params = np.array([0, 0.2, 0.1, 0, 70, 30])
target_transform = Affine.identity(2).from_vector(target_params)
# printing an affine transform tells us what it does
print(target_transform)
# make a blank (default filled with 0's) greyscale (default: 1 channel) image to guide the warp
template = BooleanImage.blank((90, 90))
target = takeo.warp_to(template, target_transform)
target.view_new()
<menpo.transform.homogeneous.affine.Affine object at 0x10d88d350> [[ 1. 0.1 70. ] [ 0.2 1. 30. ] [ 0. 0. 1. ]] CCW Rotation of 136 degrees about [0 0 1] NonUniformScale by [ 1.151 0.851] CCW Rotation of 133 degrees about [0 0 1] Translate by [ 70. 30.]
<menpo.visualize.viewmatplotlib.MatplotlibImageViewer2d at 0x10d772710>
In order to perform the alignment, we seperate the algorithm from the similarity measure used. Therefore, we begin by creating a similarity measure which defines how we define the error between the two images. For this example, we use the most simple case, the LeastSquares pixels differences.
from menpo.fit.lucaskanade.residual import LSIntensity, ECC
residual = LSIntensity()
Now, given an initial estimate, we can perform the alignment. We can either perform an inverse compositional alignment, or a forward additive. Lucas-Kanade is a gradient descent algorithm, and thus assumes a near global optimum initial parameterisaton. We therefore begin by initialising the image to roughly Takeo's face.
from menpo.shape import PointCloud
# Create the identity 'box' -> representing the area
# that the target image was warped into
corners = target.shape
identity_box = PointCloud(np.array([[0, 0],
[corners[0], 0],
[corners[0], corners[1]],
[0, corners[1]]]))
from menpo.fit.lucaskanade.image import ImageInverseCompositional, ImageForwardAdditive, ImageForwardCompositional
from copy import deepcopy
# Create the initial transform as an alignment transform
# so that we can get more interesting fitting information,
# since we then know the ground truth!
initial_params = np.array([0, 0, 0, 0, 70.5, 30.5])
inital_transform = Affine.identity(2).from_vector(initial_params)
# This is the initial 'box' that we are warping into
initial_box = inital_transform.apply(identity_box)
inital_transform = AlignmentAffine(identity_box, initial_box)
inv_comp = ImageInverseCompositional(target, residual, deepcopy(inital_transform))
for_add = ImageForwardAdditive(target, residual, deepcopy(inital_transform))
for_comp = ImageForwardCompositional(target, residual, deepcopy(inital_transform))
We then perform the alignment, and show the resulting transforms.
%matplotlib inline
# Get Inverse Compositional optimum transform and plot
inv_comp_fitting = inv_comp.fit(takeo, initial_params)
inv_comp_res = takeo.warp_to(template, inv_comp_fitting.final_transform)
plt.subplot(141)
inv_comp_res.view()
# Get Forward Compositional optimum transform and plot
for_comp_fitting = for_comp.fit(takeo, initial_params)
for_comp_res = takeo.warp_to(template, for_comp_fitting.final_transform)
plt.subplot(142)
for_comp_res.view()
# Get Forward Additive optimum transform and plot
for_add_fitting = for_add.fit(takeo, initial_params)
for_add_res = takeo.warp_to(template, for_add_fitting.final_transform)
plt.subplot(143)
for_add_res.view()
# Plot target image we were warping to
plt.subplot(144)
target.view()
# Set Figure to be larger
plt.gcf().set_size_inches(12.0, 5.0)
We can then compute the RMS point error between the original bounding box and the one proposed by our alignment algorithms. Usually, any error less than 3 pixels is considered a convergence.
# Create the target 'box' that the target was warped into
target_box = target_transform.apply(identity_box)
# Setup the fitting objects so we can generate useful results
inv_comp_fitting.error_type = 'rmse'
inv_comp_fitting.gt_shape = target_box
for_add_fitting.error_type = 'rmse'
for_add_fitting.gt_shape = target_box
for_comp_fitting.error_type = 'rmse'
for_comp_fitting.gt_shape = target_box
print('Inverse compositional RMS error: {0}'.format(inv_comp_fitting.final_error))
print('Forward additive RMS error: {0}'.format(for_add_fitting.final_error))
print('Forward compositional RMS error: {0}'.format(for_comp_fitting.final_error))
Inverse compositional RMS error: 1.55166232609e-05 Forward additive RMS error: 1.47437246188e-05 Forward compositional RMS error: 4.08764645057e-05