Are two mixture models with shared categorical distribution dependent/independent?


#1

Hej,
I’d like two random variables, one that samples from a mixture of Gaussians and along with that a dependent OneHotCategorical random variable that represents that individual Gaussian from the mixture of Gaussians.
Let’s give it a try and first build a mixture of Gaussians:

from edward.models import Normal, Categorical, Mixture, Dirichlet, OneHotCategorical

mu = np.array([0.,1.,2.,3.], dtype=np.float32)
k=4
n=1000
d = Dirichlet(np.ones(k, dtype=np.float32))
c = Categorical(probs = d, sample_shape=(n))
components = [Normal(loc=mu[kk], scale=1., sample_shape=(n)) for kk in range(k)]
x = Mixture(cat=c, components=components, sample_shape=(n))

and another mixture of “deterministic” OneHotCategorical random variables

oneHotProbs = np.eye(k, k, dtype=np.float32)
oneHotProbs[oneHotProbs==0]=-1000
oneHotProbs[oneHotProbs==1]=1000
oneHotComponents = [OneHotCategorical(logits=oneHotProbs[kk,:], sample_shape=(n)) for kk in range(k)]
y = Mixture(cat=c, components=oneHotComponents, sample_shape=(n))

Are x and y coherently using the identical samples from c?
Is there any way to check this? Or is there a simpler way to do this?

Cheers, Rasmus


#2

To test:

sess = ed.get_session()
x_cat_sample, y_cat_sample = sess.run([x.cat, y.cat])
assert x_cat_sample == y_cat_sample

For a more end-to-end test, define the Normal’s in x with well-separated modes. For example, set scale to be 1e-8 and set loc to 0, 1, 2, and 3.

I predict both will pass. c is a Categorical random variable, which specifically means it is a Categorical TensorFlow Distribution associated with a sample tensor c* ~ p(c) in the TensorFlow graph (c.value). This sample tensor is the same in both x and y because c is the same in both x and y.


#3

Actually, the first test will pass but the second test will fail. x and y have a sample tensor (x.value and y.value). These tensors are given by calling {x,y}.sample(): its implementations newly draws a value from the Categorical random variable:


#4

Thank you. I will give it a try.
That the second test fails implicates that a Neural Network cannot be trained on classifying x with y as training output. Since x and y are sampled independently. Right?
Any ideas or suggestions how to create a construction as the above, but with x and y being dependent on each other?
How could this be implemented, I’d love to give it a try.