L2 regularization of weights


#1

I’m trying to understand how can we use the regularization with Edward models. I’m not very much familiar with tensorflow. Consider the model below,

# prior
w=Normal(loc=tf.zeros((d,c)),scale=tf.ones((d,c)))

# likelihood
y=Categorical(logits=tf.matmul(X,w))

# posterior
loc_qw = tf.get_variable("qw/loc", [d, c])
scale_qw = tf.nn.softplus(tf.get_variable("qw/scale", [d, c]))
qw = Normal(loc=loc_qw, scale=scale_qw)

# inference 
inference = ed.KLqp({w: qw, b: qb}, data={X:train_X, y:train_y})

I notice that Edward uses regularization losses in the loss function.
loss = -(p_log_lik - kl_penalty - reg_penalty)

However, I can’t figure out how to apply the regularization losses to the Edward model. How can we add L1 or L2 regularization to the above model?

Thanks!


#2

Possibly I just answered your question on stackoverflow (I answered a very similar question!)

The normal prior on w is the Bayesian analogue to L2 regularization when optimizing parameters.