This paper describes a method for model-based averaging of sets of diffusion weighted magnetic resonance images (DW-MRI) under space transformations (resulting for example from registration methods). A robust weighted least squares method is developed. Synthetic validation experiments show the improvement of the proposed estimation method in comparison to standard least squares estimation. The developed method is applied to construct an atlas of diffusion weighted images for a set of macaques, allowing for a more flexible representation of average diffusion information compared to standard diffusion tensor atlases.