# Error in inference.run() for a mixture model

**URL:** <https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175>\
**Category:** General\
**Created:** [June 14, 2017, 12:24pm UTC](https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175 "2017-06-14T12:24:07Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![seiyab](https://avatars.discourse-cdn.com/v4/letter/s/7feea3/32.png) [@seiyab](https://discourse.edwardlib.org/u/seiyab)\
**Post date:** [June 14, 2017, 12:24pm UTC](https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175/1 "2017-06-14T12:24:07Z")

</div>

Hello,  
I’m a newbie of Edward (and also English). I’m trying to work on a kind of mixture model, which has multiple mixture probability vectors. When I run inference, I get an error and I don’t understand what it says.

The code for my model is following:

```auto
import edward as ed                                                                                                                                                                 
import edward.models as edm                                                                                                                                                         
import tensorflow as tf  

phi = edm.Dirichlet(tf.ones([m]), sample_shape=n)                                                                                                                                   
mu = edm.Normal(tf.zeros(1), tf.ones(1)*2, sample_shape=m)                                                                                                                          
x = [edm.ParamMixture(phi[i], {'loc': mu, 'scale_diag': tf.ones([m, 1])}, edm.MultivariateNormalDiag, sample_shape=l) for i in range(n)]

```

and the code for inference is following:

```auto
q_phi = edm.Dirichlet(pos_variables([n, m]))                                                                                                                                        
q_mu = edm.Normal(tf.Variable(tf.random_normal([m, 1])), tf.ones([m, 1]))                                                                                                           
q_z = [edm.Categorical(tf.Variable(tf.random_normal([l, m]))) for _ in range(n)]                                                                                                    
                                                                                                                                                                                    
inference = ed.KLqp({phi: q_phi, mu: q_mu, **{p.cat: q for p, q in zip(x, q_z)}}, data={p: d for p, d in zip(x, data_x)})                                                           
inference.run()  

```

The error I got is following:

```auto
Traceback (most recent call last):
  File "minimal.py", line 34, in <module>
    inference.run()
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/inference.py", line 119, in run
    self.initialize(*args, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/klqp.py", line 86, in initialize
    return super(KLqp, self).initialize(*args, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/variational_inference.py", line 74, in initialize
    self.loss, grads_and_vars = self.build_loss_and_gradients(var_list)
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/klqp.py", line 136, in build_loss_and_gradients
    return build_score_loss_and_gradients(self, var_list)
  File "/opt/conda/lib/python3.5/site-packages/edward/inferences/klqp.py", line 565, in build_score_loss_and_gradients
    x_copy = copy(x, dict_swap, scope=scope)
  File "/opt/conda/lib/python3.5/site-packages/edward/util/random_variables.py", line 232, in copy
    new_rv = type(rv)(*args, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/edward/models/random_variable.py", line 103, in __init__
    super(RandomVariable, self). __init__ (*args, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/edward/models/param_mixture.py", line 80, in __init__
    self._mixing_weights = tf.identity(mixing_weights, name="mixing_weights")
  File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1338, in identity
    result = _op_def_lib.apply_op("Identity", input=input, name=name)
  File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2339, in create_op
    self._add_op(ret)
  File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2033, in _add_op
    "is already used" % op.name)
ValueError: cannot add op with name inference_140101520622984/0/ParamMixture/mixing_weights as that name is already used

```

I’m using Python 3.5.3, Edward 1.3.2 and Ubuntu 14.04 without GPU.

Thank you for your help.

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [June 15, 2017, 11:33pm UTC](https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175/2 "2017-06-15T23:33:02Z")

</div>

Looks like a bug. Not totally sure how/why `copy` is attempting to re-build a `ParamMixture` random variable with a name that’s already taken. I’ll investigate.

---

<div class="post-metadata">

**Author:** ![seiyab](https://avatars.discourse-cdn.com/v4/letter/s/7feea3/32.png) [@seiyab](https://discourse.edwardlib.org/u/seiyab)\
**Post date:** [June 16, 2017, 4:46am UTC](https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175/3 "2017-06-16T04:46:56Z")

</div>

Thank you very much for your response.

I’ll also look into it.

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 5, 2017, 10:16pm UTC](https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175/4 "2017-07-05T22:16:12Z")

</div>

Looks like this is solved in Edward 1.3.3. I’m not sure what commit specifically solved this between 1.3.2 and 1.3.3. However, the following code runs successfully:

```python
import edward as ed
import edward.models as edm
import numpy as np
import tensorflow as tf

l = 100
n = 5
m = 2

ed.set_seed(42)

phi = edm.Dirichlet(tf.ones([m]), sample_shape=n)
mu = edm.Normal(tf.zeros(1), tf.ones(1)*2, sample_shape=m)
x = [edm.ParamMixture(phi[i], {'loc': mu, 'scale_diag': tf.ones([m, 1])}, edm.MultivariateNormalDiag, sample_shape=l)
     for i in range(n)]

q_phi = edm.Dirichlet(tf.nn.softplus(tf.Variable(tf.random_normal([n, m]))))
q_mu = edm.Normal(tf.Variable(tf.random_normal([m, 1])), tf.ones([m, 1]))
q_z = [edm.Categorical(tf.Variable(tf.random_normal([l, m]))) for _ in range(n)]

data_x = [np.zeros(xi.shape.as_list()) for xi in x]

latent_vars = {phi: q_phi, mu: q_mu}
latent_vars.update({p.cat: q for p, q in zip(x, q_z)})
inference = ed.KLqp(latent_vars, data={p: d for p, d in zip(x, data_x)})
inference.run()

```

---

<div class="post-metadata">

**Author:** ![seiyab](https://avatars.discourse-cdn.com/v4/letter/s/7feea3/32.png) [@seiyab](https://discourse.edwardlib.org/u/seiyab)\
**Post date:** [July 6, 2017, 11:13pm UTC](https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175/5 "2017-07-06T23:13:51Z")

</div>

Thank you very much!  
I’ll try with Edward 1.3.3.

---

<div class="post-metadata">

**Author:** ![manuel](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@manuel](https://discourse.edwardlib.org/u/manuel)\
**Post date:** [July 18, 2017, 6:41am UTC](https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175/6 "2017-07-18T06:41:33Z")

</div>

Hi,  
are you sure that the code is working as indented with edward 1.3.3? If I run your code with 1.3.3 (tensorflow 1.2.1), I get the same error that seiyab got.  
However this only applies to the first run. But if one runs the same script twice in a single ipython session or defines the model twice before `inference.run()` the code run successfully.  
I’ve tested it on two machines and the behaviour is identical on the two machines, I have however no idea yet as to why it happens.

But just defining the relevant parts twice seems to be a strange but working intermediate solution.

---

<div class="post-metadata">

**Author:** ![DPmodeler](https://avatars.discourse-cdn.com/v4/letter/d/b3f665/32.png) [@DPmodeler](https://discourse.edwardlib.org/u/DPmodeler)\
**Post date:** [July 18, 2017, 12:48pm UTC](https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175/7 "2017-07-18T12:48:07Z")

</div>

Same here.  
Additionally, I tried the code from @dustin on a small dataset generated from a mixture of two Gaussians (with n=1), and the results are completely random. Do you have any idea what I am doing wrong or what is going wrong here?

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 19, 2017, 6:35am UTC](https://discourse.edwardlib.org/t/error-in-inference-run-for-a-mixture-model/175/8 "2017-07-19T06:35:58Z")

</div>

> @manuel | are you sure that the code is working as indented with edward 1.3.3?

You are correct. It’s the development version of Edward that runs it successfully and not 1.3.3.

> @DPmodeler | I tried the code from @dustin on a small dataset generated from a mixture of two Gaussians (with n=1), and the results are completely random. Do you have any idea what I am doing wrong or what is going wrong here?

This issue is not related. `ed.KLqp` (black box methods for variational inference) are known not to work well with Dirichlet and Gamma distributions. I recommend using an alternative algorithm.