Numerical issues with Multinomial

2

Just as a heads up - I got something working using the MAP estimator. It seems that once I reduce the learning rate down sufficiently, I can get something that looks sane.

Here’s the code I’m using to do this.

## Model
# N = number of samples
# p = number of covariates in regression model
# D = number of features within each sample
# psi = predefined orthonormal basis to ensure identifiability when performing the softmax transform
G = tf.placeholder(tf.float32, [N, p])
B = Normal(loc=tf.zeros([p, D-1]), 
           scale=tf.ones([p, D-1]))
v = tf.matmul(G, B) 
eta = tf.nn.log_softmax(tf.matmul(v, psi))
Y = Multinomial(total_count=n, logits=eta, 
                value=tf.zeros([N, D], dtype=tf.float32))

## Inference
qB = PointMass(params=tf.Variable(tf.zeros([p, D-1])))
inference = ed.MAP(
    {B: qB},     
    data={G: G_data, Y: y_data}
)
optimizer = tf.train.AdamOptimizer(5e-4, 
                                   beta1=0.9,
                                   beta2=0.999)
inference.run(n_iter=5000, optimizer=optimizer)

Still having a bit of trouble getting the other inference techniques to work. Could be due to difficulties handling the Multinomial. But I’m not entirely sure.

1 Like