# Examples/bayesian\_nn.py

**URL:** <https://discourse.edwardlib.org/t/examples-bayesian-nn-py/929>\
**Category:** General\
**Created:** [September 7, 2018, 8:36am UTC](https://discourse.edwardlib.org/t/examples-bayesian-nn-py/929 "2018-09-07T08:36:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![liuchenbaidu](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/liuchenbaidu/32/355_2.png) [@liuchenbaidu](https://discourse.edwardlib.org/u/liuchenbaidu)\
**Post date:** [September 7, 2018, 8:36am UTC](https://discourse.edwardlib.org/t/examples-bayesian-nn-py/929/1 "2018-09-07T08:36:29Z")

</div>

in the example.i don’t see any inference algorithm such as SGD,ADM?  
what is the algorithm for the example.

“”"Bayesian neural network using variational inference  
(see, e.g., Blundell et al. (2015); Kucukelbir et al. (2016)).

Inspired by autograd’s Bayesian neural network example.  
This example prettifies some of the tensor naming for visualization in  
TensorBoard. To view TensorBoard, run `tensorboard --logdir=log`.

## References

[http://edwardlib.org/tutorials/bayesian-neural-network](http://edwardlib.org/tutorials/bayesian-neural-network)  
“”"  
from **future** import absolute\_import  
from **future** import division  
from **future** import print\_function

import edward as ed  
import numpy as np  
import tensorflow as tf

from edward.models import Normal

tf.flags.DEFINE\_integer(“N”, default=40, help=“Number of data points.”)  
tf.flags.DEFINE\_integer(“D”, default=1, help=“Number of features.”)

FLAGS = tf.flags.FLAGS

def build\_toy\_dataset(N=40, noise\_std=0.1):  
D = 1  
X = np.concatenate([np.linspace(0, 2, num=N / 2),  
np.linspace(6, 8, num=N / 2)])  
y = np.cos(X) + np.random.normal(0, noise\_std, size=N)  
X = (X - 4.0) / 4.0  
X = X.reshape((N, D))  
return X, y

def main(\_):  
def neural\_network(X):  
h = tf.tanh(tf.matmul(X, W\_0) + b\_0)  
h = tf.tanh(tf.matmul(h, W\_1) + b\_1)  
h = tf.matmul(h, W\_2) + b\_2  
return tf.reshape(h, [-1])  
ed.set\_seed(42)

# DATA

X\_train, y\_train = build\_toy\_dataset(FLAGS.N)

# MODEL

with tf.name\_scope(“model”):  
W\_0 = Normal(loc=tf.zeros([FLAGS.D, 10]), scale=tf.ones([FLAGS.D, 10]),  
name=“W\_0”)  
W\_1 = Normal(loc=tf.zeros([10, 10]), scale=tf.ones([10, 10]), name=“W\_1”)  
W\_2 = Normal(loc=tf.zeros([10, 1]), scale=tf.ones([10, 1]), name=“W\_2”)  
b\_0 = Normal(loc=tf.zeros(10), scale=tf.ones(10), name=“b\_0”)  
b\_1 = Normal(loc=tf.zeros(10), scale=tf.ones(10), name=“b\_1”)  
b\_2 = Normal(loc=tf.zeros(1), scale=tf.ones(1), name=“b\_2”)

```
X = tf.placeholder(tf.float32, [FLAGS.N, FLAGS.D], name="X")
y = Normal(loc=neural_network(X), scale=0.1 * tf.ones(FLAGS.N), name="y")

```

# INFERENCE

with tf.variable\_scope(“posterior”):  
with tf.variable\_scope(“qW\_0”):  
loc = tf.get\_variable(“loc”, [FLAGS.D, 10])  
scale = tf.nn.softplus(tf.get\_variable(“scale”, [FLAGS.D, 10]))  
qW\_0 = Normal(loc=loc, scale=scale)  
with tf.variable\_scope(“qW\_1”):  
loc = tf.get\_variable(“loc”, [10, 10])  
scale = tf.nn.softplus(tf.get\_variable(“scale”, [10, 10]))  
qW\_1 = Normal(loc=loc, scale=scale)  
with tf.variable\_scope(“qW\_2”):  
loc = tf.get\_variable(“loc”, [10, 1])  
scale = tf.nn.softplus(tf.get\_variable(“scale”, [10, 1]))  
qW\_2 = Normal(loc=loc, scale=scale)  
with tf.variable\_scope(“qb\_0”):  
loc = tf.get\_variable(“loc”, [10])  
scale = tf.nn.softplus(tf.get\_variable(“scale”, [10]))  
qb\_0 = Normal(loc=loc, scale=scale)  
with tf.variable\_scope(“qb\_1”):  
loc = tf.get\_variable(“loc”, [10])  
scale = tf.nn.softplus(tf.get\_variable(“scale”, [10]))  
qb\_1 = Normal(loc=loc, scale=scale)  
with tf.variable\_scope(“qb\_2”):  
loc = tf.get\_variable(“loc”, [1])  
scale = tf.nn.softplus(tf.get\_variable(“scale”, [1]))  
qb\_2 = Normal(loc=loc, scale=scale)

inference = ed.KLqp({W\_0: qW\_0, b\_0: qb\_0,  
W\_1: qW\_1, b\_1: qb\_1,  
W\_2: qW\_2, b\_2: qb\_2}, data={X: X\_train, y: y\_train})  
inference.run(logdir=‘log’)

if **name** == “ **main** ”:  
tf.app.run()

---

<div class="post-metadata">

**Author:** ![Focus](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/focus/32/357_2.png) [@Focus](https://discourse.edwardlib.org/u/Focus)\
**Post date:** [September 13, 2018, 7:01am UTC](https://discourse.edwardlib.org/t/examples-bayesian-nn-py/929/2 "2018-09-13T07:01:19Z")

</div>

The inference is KLqp which is minimised by using Adam by default: [https://github.com/blei-lab/edward/blob/master/edward/inferences/variational\_inference.py](https://github.com/blei-lab/edward/blob/master/edward/inferences/variational_inference.py)

---

<div class="post-metadata">

**Author:** ![liuchenbaidu](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/liuchenbaidu/32/355_2.png) [@liuchenbaidu](https://discourse.edwardlib.org/u/liuchenbaidu)\
**Post date:** [September 13, 2018, 7:24am UTC](https://discourse.edwardlib.org/t/examples-bayesian-nn-py/929/3 "2018-09-13T07:24:19Z")

</div>

ths.what you say is right.i find the codes:

# Build optimizer.

96 if optimizer is None:  
97 optimizer = tf.train.AdamOptimizer(learning\_rate)

but if i will change the optimizer,how should i do?

---

<div class="post-metadata">

**Author:** ![Focus](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/focus/32/357_2.png) [@Focus](https://discourse.edwardlib.org/u/Focus)\
**Post date:** [September 13, 2018, 8:10am UTC](https://discourse.edwardlib.org/t/examples-bayesian-nn-py/929/4 "2018-09-13T08:10:33Z")

</div>

You can do it by `inference.initialize(optimizer=tf.train.GradientDescentOptimizer(1e-2))`, see also [http://edwardlib.org/tutorials/batch-training](http://edwardlib.org/tutorials/batch-training) if you want to do batch training as well