Consistency Test

For many registration problems in practice, there is no ground truth available to evaluate the registration. This makes the quantitative assessment of methods quite difficult. Therefore, to test the method quantitatively, a comparative consistency test was performed that does not require knowledge of the actual ground truth. The consistency test is based on comparing registrations obtained using various different, but known, initial starting positions of a given image. If the registrations are consistent then the final registered image will be the same, regardless of the starting position. Consistency is a necessary, but not sufficient condition that all correctly functioning registration methods must possess. This is essentially a measure of the robustness rather than the accuracy [17] of the registration method. Robustness is defined here as the ability to get close to the global minimum on all trials, whereas accuracy is the ability to precisely locate a (possibly local) minimum of the cost function. Ideally a registration method should be both robust and accurate. More specifically, the consistency test for an individual image $I$ involved taking the image and applying several pre-determined affine transformations, $A_j$ to it (with appropriate cropping so that no `padding' of the images was required). All these images (both transformed and un-transformed) were registered to a given reference image, $I^r$, giving transformations $T_j$. If the method was consistent the composite transformations $T_j \circ A_j$ should all be the same, as illustrated in Figure 8.

Figure 8: Illustration of the consistency test for a single image. An image (top) has a number of initial affine transformations $A_j$ applied to it. The resulting images (middle) are then registered to the reference image (bottom), giving transformations $T_j$. Therefore, the overall transformation from the initial image to the reference image is $F_j = T_j \circ A_j$, and these are compared with $T_0$ which is the registration of the initial image directly to the reference image. For a consistent method, all the transformations, $F_j$, should be the same as $T_0$.

The transformations are compared quantitatively using the RMS deviation between the composite registration $T_j \circ A_j$ and the registration from the un-transformed case $T_0$. This RMS deviation is calculated directly from the affine matrices [11]. That is:

$$d_{RMS} = \sqrt{\frac{1}{5} R^2 \mathrm{Tr}(M^\top M) + t^\top t},$$ (6)

where $d_{RMS}$ is the RMS deviation in mm, $R$ is a radius specifying the volume of interest, and $\left( \begin{array}{cc} M & t 0 & 0 \end{array} \right) = T_j \cdot A_j \cdot {T_0}^{-1} - I$ is used to calculate the $3 \times 3$ matrix $M$ and the $3 \times 1$ vector $t$.