Harris interest points Comparing interest points

• Harris interest points

• Comparing interest points (SSD, ZNCC, SIFT)

• Scale & affine invariant interest points

• Evaluation and comparison of different detectors

• Region descriptors and their performance

Description regions have to be adapted to scale changes

• Description regions have to be adapted to scale changes

We want to find the characteristic scale by convolving it with, for example, Laplacians at several scales and looking for the maximum response

• We want to find the characteristic scale by convolving it with, for example, Laplacians at several scales and looking for the maximum response

• However, Laplacian response decays as scale increases:

The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases

• The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases

The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases

• The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases

• To keep response the same (scale-invariant), must multiply Gaussian derivative by σ

• Laplacian is the second Gaussian derivative, so it must be multiplied by σ2

Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D

• Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D

Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D

• Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D

The 2D Laplacian is given by

• The 2D Laplacian is given by

• For a binary circle of radius r, the Laplacian achieves a maximum at

We define the characteristic scale as the scale that produces peak of Laplacian response

• We define the characteristic scale as the scale that produces peak of Laplacian response

For a point compute a value (gradient, Laplacian etc.) at several scales

• For a point compute a value (gradient, Laplacian etc.) at several scales

• Normalization of the values with the scale factor

• Select scale at the maximum → characteristic scale

• Exp. results show that the Laplacian gives best results

Scale invariance of the characteristic scale

• Scale invariance of the characteristic scale

Scale invariance of the characteristic scale

• Scale invariance of the characteristic scale

Harris-Laplace (Mikolajczyk & Schmid’01)

• Harris-Laplace (Mikolajczyk & Schmid’01)

• Laplacian detector (Lindeberg’98)

• Difference of Gaussian (Lowe’99)

• invariant points + associated regions [Mikolajczyk & Schmid’01]

Detection of maxima and minima

• Detection of maxima and minima

• of Laplacian in scale space

Fast computation, scale space processed one octave at a time

• Fast computation, scale space processed one octave at a time

• Scale invariant interest points

• Affine invariant interest points

• Evaluation of interest points

• Descriptors and their evaluation

Scale invariance is not sufficient for large baseline changes

• Scale invariance is not sufficient for large baseline changes

Initialize with scale-invariant Harris/Hessian/Laplacian points

• Initialize with scale-invariant Harris/Hessian/Laplacian points

• Estimation of the affine neighbourhood with the second moment matrix [Lindeberg’94]

• Apply affine neighbourhood estimation to the scale-invariant interest points [Mikolajczyk & Schmid’02, Schaffalitzky & Zisserman’02]

• Excellent results in a recent comparison

Based on the second moment matrix (Lindeberg’94)

• Based on the second moment matrix (Lindeberg’94)

Iterative estimation – initial points

• Iterative estimation – initial points

Iterative estimation – iteration #1

• Iterative estimation – iteration #1

Iterative estimation – iteration #2

• Iterative estimation – iteration #2

Iterative estimation – iteration #3, #4

• Iterative estimation – iteration #3, #4

Extremal regions: connected components in a thresholded image (all pixels above/below a threshold)

• Extremal regions: connected components in a thresholded image (all pixels above/below a threshold)

• Maximally stable: minimal change of the component (area) for a change of the threshold, i.e. region remains stable for a change of threshold

• Excellent results in a recent comparison

• Harris interest points

• Comparing interest points (SSD, ZNCC, SIFT)

• Scale & affine invariant interest points

• Evaluation and comparison of different detectors

• Region descriptors and their performance

Quantitative evaluation of interest point/region detectors

• Quantitative evaluation of interest point/region detectors

• points / regions at the same relative location and area

• Repeatability rate : percentage of corresponding points

• Two points/regions are corresponding if

• location error small
• area intersection large

• [K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas,

• F. Schaffalitzky, T. Kadir & L. Van Gool ’05]

Different types of transformation

• Different types of transformation

• Viewpoint change
• Scale change
• Image blur
• JPEG compression
• Light change

• Two scene types

• Structured
• Textured

• Transformations within the sequence (homographies)

• Independent estimation

Good performance for large viewpoint and scale changes

• Good performance for large viewpoint and scale changes

• Results depend on transformation and scene type, no one best detector

• Detectors are complementary

• MSER adapted to structured scenes
• Harris and Hessian adapted to textured scenes

• Performance of the different scale invariant detectors is very similar (Harris-Laplace, Hessian, LoG and DOG)

• Scale-invariant detector sufficient up to 40 degrees of viewpoint change

• Harris interest points

• Comparing interest points (SSD, ZNCC, SIFT)

• Scale & affine invariant interest points

• Evaluation and comparison of different detectors

• Region descriptors and their performance

• Normalized regions are

• invariant to geometric transformations except rotation
• not invariant to photometric transformations

• Regions invariant to geometric transformations except rotation

• normalization with dominant gradient direction

• Regions not invariant to photometric transformations

• normalization with mean and standard deviation of the image patch

Gaussian derivative-based descriptors

• Gaussian derivative-based descriptors

• Differential invariants (Koenderink and van Doorn’87)
• Steerable filters (Freeman and Adelson’91)

• Moment invariants [Van Gool et al.’96]

• SIFT (Lowe’99)

• Shape context [Belongie et al.’02]

• SIFT with PCA dimensionality reduction

• Gradient PCA [Ke and Sukthankar’04]

• SURF descriptor [Bay et al.’08]

• DAISY descriptor [Tola et al.’08, Windler et al’09]

Descriptors should be

• Descriptors should be

• Distinctive
• Robust to changes on viewing conditions as well as to errors of the detector

• Detection rate (recall)

• #correct matches / #correspondences

• False positive rate

• #false matches / #all matches

• Variation of the distance threshold

• distance (d1, d2) < threshold

SIFT based descriptors perform best

• SIFT based descriptors perform best

• Significant difference between SIFT and low dimension descriptors as well as cross-correlation

• Robust region descriptors better than point-wise descriptors

• Performance of the descriptor is relatively independent of the detector

Binaries for detectors and descriptors

• Binaries for detectors and descriptors

• Building blocks for recognition systems

• Carefully designed test setup

• Dataset with transformations
• Evaluation code in matlab
• Benchmark for new detectors and descriptors