Practical experiences using ADVI training Bayesian neural network


#1

Hello!

I am just simply using KLqp, and my neural network model is very simple so it would be using the reparameteric gradient, which is the ADVI algorithm in the ADVI paper.

I have found that

  • using Adam optimizer is really helpful especially in training neural networks
  • using batch training is helpful
  • number of samples doesn’t matter a lot

Please comment below if you find other things helpful!


#2

Hi,

Can I ask how you got that working? I’m a tensorflow beginner and is trying to figure out how to implement a neural network for CIFAR10 with KLqp. However, my model currently won’t train properly and I’m unsure how to debug it.

Thanks!!


#3

Hi,

I think try to fully understand this tutorial is the way to go:

http://edwardlib.org/tutorials/bayesian-neural-network

Hope this helps!