Voxel-based analysis provides a simple, easy to interpret approach to discover regions correlated with a variable of interest such as for example a pathology indicator. Voxel-based analysis methods perform a statistical test at each voxel and are prone to false positives due to multiple testing, or when corrected for multiple testing may miss regions of interest. Component based approaches, such as principal or independent component analysis provide an approach to mitigate multiple testing, by testing for correlations to projections of the data to the components. We propose a spatially regularized component analysis approach to find components for image data sets that are spatially localized and smooth. We show that the proposed approach leads to components that are easier to interpret and can improve predictive performance when used with linear regression models. We develop an efficient optimization approach using the Grassmannian projection kernel and a randomized SVD. The proposed optimization is capable to deal with data sets to large too fit all at once into memory. We demonstrate the approach with an application to study Alzheimer’s disease using over 1200 images from the OASIS-3 data set.