Maybe you can use the 2. approach Dustin describes here: How to handle missing values in Gaussian Matrix Factorization