Multi-atlas segmentation (MAS) is a popular image segmen- tation technique for medical images. In this work, we improve the performance of MAS by correcting registration errors be- fore label fusion. Specifically, we use a volumetric displace- ment field to refine registrations based on image anatomical appearance and predicted labels. We show the influence of the initial spatial alignment as well as the beneficial effect of using label information for MAS performance. Experiments demonstrate that the proposed refinement approach improves MAS performance on a 3D magnetic resonance dataset of the knee.