Our group has a long history of developing
image registration approaches. In particular, we have been focusing on non-parametric approaches to image registration, where deformations are directly or indirectly parameterized via vector fields. Specifically, we have explored approaches related to the large deformation metric mapping model (LDDMM) and stationary velocity fields for
computational anatomy.
Table of Contents
Image registration and optimal control
A lot of recent image registration work has moved to deep learning approaches which make use of automatic differentiation. Automatic differentiation in turn is related to adjoint optimization approaches and are intimately related to optimal control. We introduced optimal control approaches for the LDDMM model ( [hart09a][hart09b]
) which greatly increased ease of formulating and the solution of related registration problems by simplifying gradient formulation due to the adjoint approach. This approach also paved the way for shooting approaches for image registration and geodesic regression ( [niethammer11]
which parameterize a deformation only via initial conditions as well as for more general models, such as splines for diffeomorphisms ( [singh15]
). Most recently we developed an approach to parameterize neural networks via initial conditions based on this viewpoint ( [vialard20]
).
Image registration in the presence of pathologies or appearance changes
Most registration methods existing today assume that there is perfect correspondence between image pairs to be registered, that transformations are globally smooth, and that image appearance is consistent between modalities or over time. While this is a reasonable assumption for many applications, it is ill suited in case of image pathologies (e.g., tumors, lesions, traumatic brain injury), to capture sliding image motions (e.g., the lung or liver sliding with respect to surrounding tissue), or to model brain maturation during myelination. We have developed multiple approaches to tackle these challenges: 1) by modifying deformation models to make them more realistic ( [pace13]
), 2) by modifying image similarity measures to model anticipated changes of appearance ( [csapo13][cao14]
), and 3) by exploiting population data to infer what should be considered normal and what should be considered abnormal appearance ( [liu15][han18]
).
Deep learning approaches for image registration
Deep learning approaches to image registration dramatically reduce the runtime of registration algorithms as costly numerical optimization is replaced by the evaluation of a pre-trained regression network. This now allows registrations of typical 3D magnetic resonance image volumes in about a second on a modern GPU. We developed the first work on deep-learning for non-parametric medical image registration ( [yang16]
) which predicted the initial momentum of an LDDMM solution, hence was already able to guarantee diffeomorphic transformations. Because this work is based on patch-wise predictions, it can be applied to very large images if needed. This work was extended and extensively analyzed in [yang17]
, showing competitive performance with state-of-the-art optimization-based deformable image registration approaches; it also included deep-network models to predict displacement and velocity fields. Follow-up work focused on learning the registration metric ( [niethammer19]
); jointly learning to predict affine transformations and a nonparametric transformation ( [shen19a]
) (including multi-step approaches and training to encourage symmetric models); as well as extensions of the LDDMM approach to spatio-temporal regularizers (i.e., that move with the deforming images) also within a deep-network approach ( [shen19b]
). We have also worked on deep registration models which jointly learn to segment and register ( [xu19]
) allowing to train a segmentation network with very few samples while improving registration accuracy. Implementations for many of these approaches can be found in our
mermaid and
easyreg registration toolboxes.
