Multidimensional MDN


#1

Hi,

I’m trying to extend the MDN example to case where both input and output are multidimensional.
Currently I’m getting exception with shape mismatch.
Can you provide simple example that works?

Thanks a lot


#2

Hi,

In order to promote this question, till now I’ve found that when I’m performing next command:
o = Mixture(cat=cat, components=components, value=tf.zeros_like(o_ph))

, the edward infrastructure expects batch shape of cat and components[0] be the same.

While cat has scalar dimension:
TensorShape([Dimension(None)])
(none is number of sample points)

, the components[0] in my case is Normal distribution and its batch_shape is:
TensorShape([Dimension(None), Dimension(2)])
(2 is current dimension of my output)

When creating Mixture, I’m getting error:
ValueError: Shapes (?,) and (?, 2) are not compatible

So, why does Mixture expect cat and component to share dimensions?
And how can I fix it?

Thanks


#3

Hi, I was having this same problem and your forum post was the only other mention of the problem I could find.

Anyway, I’ve since figured it out by going through the tf.contrib.distributions.Mixture code. Apparently distributions infer something about what the different axes you’re working with mean. So if you use a Normal distribution (even if you feed it a loc/scale with two axes), it assumes you want a UNIvariate normal distribution and that you simply have 2 batch dimensions. This becomes a problem later on when you do a mixture, because the mixture sees the categorical distribution has a batch shape of [None] (and event shape of []) and the Gaussians have a batch shape of [None, 2] (and event shape of []). What you want it to think is that the categorical distribution has a batch shape of [None]/event shape of [] and the Gaussians have a batch shape of [None] and an event shape of [2]. To do this, you have to use a different class, namely the ed.models.MultivariateNormalDiag class. This will provide the same distribution mathematically as you had before, but it will realize that the last axis is an event (or depth, whatever you want to call it) axis and not a batch axis.

Hope that helps others with the same problem.


#4

Thanks @kkleidal. I tried the MultivariateNormalDiag in place of Normal and the shape error disappears. However, I’m now getting a confusing error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-ea93ed6724dc> in <module>()
     15 components = [MultivariateNormalDiag(loc=loc, scale=scale) for loc, scale
     16               in zip(tf.unstack(tf.transpose(locs)),
---> 17                      tf.unstack(tf.transpose(scales)))]
     18 y = Mixture(cat=cat, components=components, value=tf.zeros_like(y_ph))
     19 # Note: A bug exists in Mixture which prevents samples from it to have

<ipython-input-3-ea93ed6724dc> in <listcomp>(.0)
     13 
     14 cat = Categorical(logits=logits)
---> 15 components = [MultivariateNormalDiag(loc=loc, scale=scale) for loc, scale
     16               in zip(tf.unstack(tf.transpose(locs)),
     17                      tf.unstack(tf.transpose(scales)))]

~/anaconda3/lib/python3.6/site-packages/edward/models/random_variables.py in __init__(self, *args, **kwargs)
     19     # to use _candidate's docstring, must write a new __init__ method
     20     def __init__(self, *args, **kwargs):
---> 21       _RandomVariable.__init__(self, *args, **kwargs)
     22     __init__.__doc__ = _candidate.__init__.__doc__
     23     _params = {'__doc__': _candidate.__doc__,

~/anaconda3/lib/python3.6/site-packages/edward/models/random_variable.py in __init__(self, *args, **kwargs)
    110       self._kwargs['collections'] = collections
    111 
--> 112     super(RandomVariable, self).__init__(*args, **kwargs)
    113 
    114     self._sample_shape = tf.TensorShape(sample_shape)

TypeError: __init__() got an unexpected keyword argument 'scale'

Any inputs will be greatly appreciated - this is the only relevant thread I could find anywhere.