Error in inference.run() for a mixture model

1

Hello,
I’m a newbie of Edward (and also English). I’m trying to work on a kind of mixture model, which has multiple mixture probability vectors. When I run inference, I get an error and I don’t understand what it says.

The code for my model is following:

import edward as ed                                                                                                                                                                 
import edward.models as edm                                                                                                                                                         
import tensorflow as tf  

phi = edm.Dirichlet(tf.ones([m]), sample_shape=n)                                                                                                                                   
mu = edm.Normal(tf.zeros(1), tf.ones(1)*2, sample_shape=m)                                                                                                                          
x = [edm.ParamMixture(phi[i], {'loc': mu, 'scale_diag': tf.ones([m, 1])}, edm.MultivariateNormalDiag, sample_shape=l) for i in range(n)]

and the code for inference is following:

q_phi = edm.Dirichlet(pos_variables([n, m]))                                                                                                                                        
q_mu = edm.Normal(tf.Variable(tf.random_normal([m, 1])), tf.ones([m, 1]))                                                                                                           
q_z = [edm.Categorical(tf.Variable(tf.random_normal([l, m]))) for _ in range(n)]                                                                                                    
                                                                                                                                                                                    
inference = ed.KLqp({phi: q_phi, mu: q_mu, **{p.cat: q for p, q in zip(x, q_z)}}, data={p: d for p, d in zip(x, data_x)})                                                           
inference.run()

The error I got is following:

Traceback (most recent call last):
  File "minimal.py", line 34, in <module>
    inference.run()
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/inference.py", line 119, in run
    self.initialize(*args, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/klqp.py", line 86, in initialize
    return super(KLqp, self).initialize(*args, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/variational_inference.py", line 74, in initialize
    self.loss, grads_and_vars = self.build_loss_and_gradients(var_list)
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/klqp.py", line 136, in build_loss_and_gradients
    return build_score_loss_and_gradients(self, var_list)
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/klqp.py", line 565, in build_score_loss_and_gradients
    x_copy = copy(x, dict_swap, scope=scope)
  File "/opt/conda/lib/python3.5/site-packages/edward/util/random_variables.py", line 232, in copy
    new_rv = type(rv)(*args, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/edward/models/random_variable.py", line 103, in __init__
    super(RandomVariable, self).__init__(*args, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/edward/models/param_mixture.py", line 80, in __init__
    self._mixing_weights = tf.identity(mixing_weights, name="mixing_weights")
  File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1338, in identity
    result = _op_def_lib.apply_op("Identity", input=input, name=name)
  File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2339, in create_op
    self._add_op(ret)
  File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2033, in _add_op
    "is already used" % op.name)
ValueError: cannot add op with name inference_140101520622984/0/ParamMixture/mixing_weights as that name is already used

I’m using Python 3.5.3, Edward 1.3.2 and Ubuntu 14.04 without GPU.

Thank you for your help.

2

Looks like a bug. Not totally sure how/why copy is attempting to re-build a ParamMixture random variable with a name that’s already taken. I’ll investigate.

3

Thank you very much for your response.

I’ll also look into it.

4

Looks like this is solved in Edward 1.3.3. I’m not sure what commit specifically solved this between 1.3.2 and 1.3.3. However, the following code runs successfully:

import edward as ed
import edward.models as edm
import numpy as np
import tensorflow as tf

l = 100
n = 5
m = 2

ed.set_seed(42)

phi = edm.Dirichlet(tf.ones([m]), sample_shape=n)
mu = edm.Normal(tf.zeros(1), tf.ones(1)*2, sample_shape=m)
x = [edm.ParamMixture(phi[i], {'loc': mu, 'scale_diag': tf.ones([m, 1])}, edm.MultivariateNormalDiag, sample_shape=l)
     for i in range(n)]

q_phi = edm.Dirichlet(tf.nn.softplus(tf.Variable(tf.random_normal([n, m]))))
q_mu = edm.Normal(tf.Variable(tf.random_normal([m, 1])), tf.ones([m, 1]))
q_z = [edm.Categorical(tf.Variable(tf.random_normal([l, m]))) for _ in range(n)]

data_x = [np.zeros(xi.shape.as_list()) for xi in x]

latent_vars = {phi: q_phi, mu: q_mu}
latent_vars.update({p.cat: q for p, q in zip(x, q_z)})
inference = ed.KLqp(latent_vars, data={p: d for p, d in zip(x, data_x)})
inference.run()
5

Thank you very much!
I’ll try with Edward 1.3.3.

6

Hi,
are you sure that the code is working as indented with edward 1.3.3? If I run your code with 1.3.3 (tensorflow 1.2.1), I get the same error that seiyab got.
However this only applies to the first run. But if one runs the same script twice in a single ipython session or defines the model twice before inference.run() the code run successfully.
I’ve tested it on two machines and the behaviour is identical on the two machines, I have however no idea yet as to why it happens.

But just defining the relevant parts twice seems to be a strange but working intermediate solution.

7

Same here.
Additionally, I tried the code from @dustin on a small dataset generated from a mixture of two Gaussians (with n=1), and the results are completely random. Do you have any idea what I am doing wrong or what is going wrong here?

8

@manuel | are you sure that the code is working as indented with edward 1.3.3?

You are correct. It’s the development version of Edward that runs it successfully and not 1.3.3.

@DPmodeler | I tried the code from @dustin on a small dataset generated from a mixture of two Gaussians (with n=1), and the results are completely random. Do you have any idea what I am doing wrong or what is going wrong here?

This issue is not related. ed.KLqp (black box methods for variational inference) are known not to work well with Dirichlet and Gamma distributions. I recommend using an alternative algorithm.