Hi. I’m beginer of Edward and English.
I tried Mixture of Linear Regression (I think we can find this model in Bishop’s Book) with Edward.
My code is here.
I tried two approach but both did not work.
My first approach is with ParamMixture.
I did not use Mixture because I want to get posterior distribution of Cat (Categorical Distribution). I don’t know why it do not work.
My second approach is coding mixture model directly.
I think that it did not work because OneHotCategorical do not have conjugate_log_prob method.
I have two questions.
- Why my first approach do not work? Are there any way works?
- Is there any reason that
OneHotCategorical don’t have conjugate_log_prob?
With PyMC3, Categorical Distribution can be used as index.
(see https://stats.stackexchange.com/questions/185812/regression-mixture-in-pymc3)
But with Edward, slice or tf.gather do not work for ed.Gibbs.
On the other hand, if I use ed.MetropolisHastings, it works. (I reached this method thanks to this post)
But because acceptance rate is very small (less than 0.1), it looks not converge.
The model you’re trying to fit is:
p(y | z = 1) = Normal(mu_1 + x beta_1, sigma_1)
p(y | z = 0) = Normal(mu_0 + x beta_0, sigma_0)
ParamMixture is not appropriate for this model because you need to infer which component each data point came from, not just the mixture proportions.
Assuming only few classes, you could marginalize over z, but that would require a custom RandomVariable in Edward.
You could also do this using expectation maximization, but that is probabily easier expressed in numpy/scipy rather than Edward.
@aksarkar
Thank you for your reply!
Since I want to avoid marginalizing over z, it sounds difficult to infer with Edward.
I will try to implement Gibbs sampler or Mean Field Approximation with numpy/scipy.