# Saving Model Parameters

**URL:** <https://discourse.edwardlib.org/t/saving-model-parameters/24>\
**Category:** General\
**Created:** [March 10, 2017, 5:00am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24 "2017-03-10T05:00:00Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [March 10, 2017, 5:00am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/1 "2017-03-10T05:00:00Z")

</div>

moved from [Saving Model Parameters · Issue #535 · blei-lab/edward · GitHub](https://github.com/blei-lab/edward/issues/535).

> @GhassanMakhoul

> I am working on a variational inference problem, and was wondering how I could snapshot the weights after training. I want to be able to restore then query ‘q’, my approximation distribution, and draw samples from it. I am familiar with saving variables with tf.train.saver. Is there a similarly easy way to save the parameters of my inference model?

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [March 10, 2017, 5:00am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/2 "2017-03-10T05:00:38Z")

</div>

> @dustinvtran

> hi @GhassanMakhoul | `tf.train.Saver` applies to variational approximations too. In your code, you should be handling your own tf variables to parameterize your inference model. After (or during) inference, you can call, e.g.,
> 
> ```python
> saver = tf.train.Saver()
> 
> ```

> sess = ed.get\_session()  
> save\_path = saver.save(sess, “/tmp/posterior.ckpt”)  
> print(“Inference model saved in file: %s” % save\_path)
> 
> ```auto
> This is the [same way you would save model parameters in TensorFlow](https://www.tensorflow.org/versions/master/how_tos/variables/). (All extensions apply, such as saving only a subset of the inference model parameters.)
> ```

---

<div class="post-metadata">

**Author:** ![paul](https://avatars.discourse-cdn.com/v4/letter/p/bbe5ce/32.png) [@paul](https://discourse.edwardlib.org/u/paul)\
**Post date:** [June 21, 2017, 11:33pm UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/3 "2017-06-21T23:33:17Z")

</div>

Hi, I’m trying to add model saving to the example “vae\_convolutional.py”, and I’m having some trouble. Here is my code, with the added code marked with #ADDED CODE:

```python
#!/usr/bin/env python
"""Convolutional variational auto-encoder for binarized MNIST.

The neural networks are written with TensorFlow Slim.

References
----------
http://edwardlib.org/tutorials/decoder
http://edwardlib.org/tutorials/inference-networks
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import edward as ed
import numpy as np
import os
import tensorflow as tf

from edward.models import Bernoulli, Normal
from edward.util import Progbar
from scipy.misc import imsave
from tensorflow.contrib import slim
from tensorflow.examples.tutorials.mnist import input_data

def generative_network(z):
  """Generative network to parameterize generative model. It takes
  latent variables as input and outputs the likelihood parameters.

  logits = neural_network(z)
  """
  with slim.arg_scope([slim.conv2d_transpose],
                      activation_fn=tf.nn.elu,
                      normalizer_fn=slim.batch_norm,
                      normalizer_params={'scale': True}):
    net = tf.reshape(z, [M, 1, 1, d])
    net = slim.conv2d_transpose(net, 128, 3, padding='VALID')
    net = slim.conv2d_transpose(net, 64, 5, padding='VALID')
    net = slim.conv2d_transpose(net, 32, 5, stride=2)
    net = slim.conv2d_transpose(net, 1, 5, stride=2, activation_fn=None)
    net = slim.flatten(net)
    return net

def inference_network(x):
  """Inference network to parameterize variational model. It takes
  data as input and outputs the variational parameters.

  loc, scale = neural_network(x)
  """
  with slim.arg_scope([slim.conv2d, slim.fully_connected],
                      activation_fn=tf.nn.elu,
                      normalizer_fn=slim.batch_norm,
                      normalizer_params={'scale': True}):
    net = tf.reshape(x, [M, 28, 28, 1])
    net = slim.conv2d(net, 32, 5, stride=2)
    net = slim.conv2d(net, 64, 5, stride=2)
    net = slim.conv2d(net, 128, 5, padding='VALID')
    net = slim.dropout(net, 0.9)
    net = slim.flatten(net)
    params = slim.fully_connected(net, d * 2, activation_fn=None)

  loc = params[:, :d]
  scale = tf.nn.softplus(params[:, d:])
  return loc, scale

ed.set_seed(42)

M = 128 # batch size during training
d = 10 # latent dimension
DATA_DIR = "data/mnist"
IMG_DIR = "img"

if not os.path.exists(DATA_DIR):
  os.makedirs(DATA_DIR)
if not os.path.exists(IMG_DIR):
  os.makedirs(IMG_DIR)

# DATA. MNIST batches are fed at training time.
mnist = input_data.read_data_sets(DATA_DIR)

# MODEL
z = Normal(loc=tf.zeros([M, d]), scale=tf.ones([M, d]))
logits = generative_network(z)
x = Bernoulli(logits=logits)

# INFERENCE
x_ph = tf.placeholder(tf.int32, [M, 28 * 28])
loc, scale = inference_network(tf.cast(x_ph, tf.float32))
qz = Normal(loc=loc, scale=scale)

# Bind p(x, z) and q(z | x) to the same placeholder for x.
data = {x: x_ph}
inference = ed.KLqp({z: qz}, data)
optimizer = tf.train.AdamOptimizer(0.01, epsilon=1.0)
inference.initialize(optimizer=optimizer)

hidden_rep = tf.sigmoid(logits)

tf.global_variables_initializer().run()

# ADDED CODE
load_saved_model = True
model_path = r"/tmp/model_vae_edward.ckpt"
sess = ed.get_session()
saver = tf.train.Saver()
if load_saved_model:
  saver.restore(sess, model_path)
  print("Model restored.")
# END ADDED CODE

n_epoch = 1
n_iter_per_epoch = 10
for epoch in range(n_epoch):
  avg_loss = 0.0

  pbar = Progbar(n_iter_per_epoch)
  for t in range(1, n_iter_per_epoch + 1):
    pbar.update(t)
    x_train, _ = mnist.train.next_batch(M)
    x_train = np.random.binomial(1, x_train)
    info_dict = inference.update(feed_dict={x_ph: x_train})
    avg_loss += info_dict['loss']

  # Print a lower bound to the average marginal likelihood for an
  # image.
  avg_loss = avg_loss / n_iter_per_epoch
  avg_loss = avg_loss / M
  print("log p(x) >= {:0.3f}".format(avg_loss))

  # Visualize hidden representations.
  imgs = hidden_rep.eval()
  for m in range(M):
    imsave(os.path.join(IMG_DIR, '%d.png') % m, imgs[m].reshape(28, 28))

  # ADDED CODE
  save_path = saver.save(sess, model_path)
  print("Model saved in file: %s" % save_path)
  # END ADDED CODE

```

The model saves ok, but when I try to load the model (set `load_saved_model = True`), I get the following error:  
NotFoundError (see above for traceback): Key optimizer\_274937448/fully\_connected/weights/Adam\_1 not found in checkpoint

What am I doing wrong?

---

<div class="post-metadata">

**Author:** ![rpmason](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@rpmason](https://discourse.edwardlib.org/u/rpmason)\
**Post date:** [June 26, 2017, 1:49pm UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/4 "2017-06-26T13:49:39Z")

</div>

I am also having the same problem when trying to load a trained model using tf.train.import\_meta\_graph(). I get: KeyError: “The name ‘Normal’ refers to an Operation not in the graph.” Are we meant to only save the tf.variables and not the operations?

Here is what I’m doing:

```
    with tf.Session() as sess:
        # Restore variables from disk.
        saver = tf.train.import_meta_graph('./models/posterior.ckpt.meta')
        saver.restore(sess, tf.train.latest_checkpoint('./models/'))

```

The full error is:

```
Traceback (most recent call last):
  File "/home/rmason/Github/alpha-i/edward-mock-time-series-test/edward_feedforward_network.py", line 87, in <module>
neural_net.run_testing(TRAINING_SERIES)
  File "/home/rmason/Github/alpha-i/edward-mock-time-series-test/edward_feedforward_network.py", line 71, in run_testing
saver = tf.train.import_meta_graph('./models/posterior.ckpt.meta')
  File "/home/rmason/anaconda3/envs/time-series-env/lib/python3.4/site-packages/tensorflow/python/training/saver.py", line 1686, in import_meta_graph
**kwargs)
  File "/home/rmason/anaconda3/envs/time-series-env/lib/python3.4/site-packages/tensorflow/python/framework/meta_graph.py", line 536, in import_scoped_meta_graph
ops.prepend_name_scope(value, scope_to_prepend_to_names))
  File "/home/rmason/anaconda3/envs/time-series-env/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 2584, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/home/rmason/anaconda3/envs/time-series-env/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 2644, in _as_graph_element_locked
"graph." % repr(name))
KeyError: "The name 'Normal' refers to an Operation not in the graph."
```

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 2, 2017, 12:17am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/5 "2017-07-02T00:17:04Z")

</div>

@paul That looks like a bug. It tries to restore Adam optimizer parameters, which are stored under a unique inference name ( `optimizer_274937448`). However, this unique name is different the next time the script is run. Issue raised at [https://github.com/blei-lab/edward/issues/696](https://github.com/blei-lab/edward/issues/696).

@rpmason I personally don’t have much experience with saving/restoring the graph itself. This is certainly worth more investigation.

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 3, 2017, 10:07am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/6 "2017-07-03T10:07:55Z")

</div>

@paul @rpmason Both bugs (restoring optimizer tf.Variables and importing the metagraph) are fixed in [https://github.com/blei-lab/edward/pull/697](https://github.com/blei-lab/edward/pull/697).

---

<div class="post-metadata">

**Author:** ![rpmason](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@rpmason](https://discourse.edwardlib.org/u/rpmason)\
**Post date:** [July 4, 2017, 4:30pm UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/7 "2017-07-04T16:30:10Z")

</div>

@dustin brilliant! Thanks a lot :).

---

<div class="post-metadata">

**Author:** ![edisoncruise](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@edisoncruise](https://discourse.edwardlib.org/u/edisoncruise)\
**Post date:** [July 6, 2017, 2:33am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/8 "2017-07-06T02:33:33Z")

</div>

Hi dustin, I have updated the edward version to 1.3.3. But I still meet this bug. Will recent release fix this? Or I need to modify the files manually following this link [https://github.com/blei-lab/edward/pull/697/files](https://github.com/blei-lab/edward/pull/697/files)?

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 6, 2017, 3:48am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/9 "2017-07-06T03:48:24Z")

</div>

The bug fix is in Edward’s development version and not in 1.3.3. To install that, see [http://edwardlib.org/getting-started](http://edwardlib.org/getting-started).

---

<div class="post-metadata">

**Author:** ![edisoncruise](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@edisoncruise](https://discourse.edwardlib.org/u/edisoncruise)\
**Post date:** [July 6, 2017, 1:20pm UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/10 "2017-07-06T13:20:15Z")

</div>

Thank you dustin. I have installed the development version. But in the example below, I still cannot load the posterior qw, though I can load the placeholder ‘X’. The error message: ‘The name ‘qw:0’ refers to a Tensor which does not exist. The operation ‘qw’, does not exist in the graph."’ Is there anything wrong in my code?

```auto
def build_toy_dataset(N, w, noise_std=0.1):
  D = len(w)
  x = np.random.randn(N, D)
  y = np.dot(x, w) + np.random.normal(0, noise_std, size=N)
  return x, y

###save all the variable
fname='E:\PythonCode\TestEdward\TestEdward\VariablesFiles_ED\MyVariables'

N = 40 # number of data points
D = 10 # number of features

w_true = np.random.randn(D)
X_train, y_train = build_toy_dataset(N, w_true)
X_test, y_test = build_toy_dataset(N, w_true)

X = tf.placeholder(tf.float32, [N, D],name='X')
w = Normal(loc=tf.zeros(D), scale=tf.ones(D),name='w')
b = Normal(loc=tf.zeros(1), scale=tf.ones(1),name='b')
y = Normal(loc=ed.dot(X, w) + b, scale=tf.ones(N),name='y')

qw = Normal(loc=tf.Variable(tf.random_normal([D])),
            scale=tf.nn.softplus(tf.Variable(tf.random_normal([D]))),name='qw')
qb = Normal(loc=tf.Variable(tf.random_normal([1])),
            scale=tf.nn.softplus(tf.Variable(tf.random_normal([1]))),name='qb')

inference = ed.KLqp({w: qw, b: qb}, data={X: X_train, y: y_train})
inference.run(n_samples=5, n_iter=250)

saver = tf.train.Saver()
sess = ed.get_session()
sess.run(tf.global_variables_initializer())
save_path = saver.save(sess, fname)

#################################run in another part to load variables###################
sess=tf.Session()    

loader = tf.train.import_meta_graph(fname+'.meta')
loader.restore(sess,fname)
graph = tf.get_default_graph()
X_load = graph.get_tensor_by_name("X:0")
print('X:',X_load)
qw_load = graph.get_tensor_by_name("qw:0")
print('qw:',qw_load)

```

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 6, 2017, 2:28pm UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/11 "2017-07-06T14:28:58Z")

</div>

Unfortunately random variables aren’t explicitly stored on TensorFlow’s graph. We decided not to as it would require a new data format, similar to how one stores `tf.Tensor`s and `tf.Variable`s.

This implies you need to import the tensor associated to `qw` and then re-build `qw`:

```python
qw_sample = graph.get_tensor_by_name("qw/sample/Reshape:0")
# this wraps the sample to include RV methods
qw = Normal(loc=graph.get_tensor_by_name("qw/loc:0"),
            scale=graph.get_tensor_by_name("qw/scale:0"),
            value=qw_sample)

```

Maybe there’s an easier approach? Contributions/pull requests welcome.

---

<div class="post-metadata">

**Author:** ![edisoncruise](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@edisoncruise](https://discourse.edwardlib.org/u/edisoncruise)\
**Post date:** [July 7, 2017, 1:17am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/12 "2017-07-07T01:17:57Z")

</div>

Thank you dustin. Though not perfect, it can solve my model saving problem.

---

<div class="post-metadata">

**Author:** ![edisoncruise](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@edisoncruise](https://discourse.edwardlib.org/u/edisoncruise)\
**Post date:** [July 10, 2017, 2:49am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/13 "2017-07-10T02:49:15Z")

</div>

Hi dustin. Now I can save and restore model parameters, but I meet some difficulty in reconstructing the inference method. In the code below, when I try to run a reconstructed inference, there is an error:  
ValueError: cannot add op with name optimizer/Variable\_1/Adam as that name is already used

It seems that when I save the model, some inference object are samed, but how can I make a new inference and run it?

import tensorflow as tf  
import edward as ed  
import numpy as np  
from edward.models import Normal

def build\_toy\_dataset(N, w, noise\_std=0.1):  
D = len(w)  
x = np.random.randn(N, D)  
y = np.dot(x, w) + np.random.normal(0, noise\_std, size=N)  
return x, y

##################code to save a linear regression model###########################

fname=‘E:\PythonCode\TestEdward\TestEdward\InferencesFiles\_ED\MyInferences’

w\_true =np.array([0.4,0.3,-0.1]) # np.random.randn(D)  
N = 40 # number of data points  
D = 3 # number of features

X\_train, y\_train = build\_toy\_dataset(N, w\_true)  
X\_test, y\_test = build\_toy\_dataset(N, w\_true)

X = tf.placeholder(tf.float32, [N, D],name=‘X’)  
w = Normal(loc=tf.zeros(D), scale=tf.ones(D),name=‘w’)  
b = Normal(loc=tf.zeros(1), scale=tf.ones(1),name=‘b’)  
y = Normal(loc=ed.dot(X, w) + b, scale=tf.ones(N),name=‘y’)

qw = Normal(loc=tf.Variable(tf.random\_normal([D])),  
scale=tf.nn.softplus(tf.Variable(tf.random\_normal([D]))),name=‘qw’)  
qb = Normal(loc=tf.Variable(tf.random\_normal([1])),  
scale=tf.nn.softplus(tf.Variable(tf.random\_normal([1]))),name=‘qb’)

inference = ed.KLqp({w: qw, b: qb}, data={X: X\_train, y: y\_train})  
inference.run(n\_samples=5, n\_iter=250)

saver = tf.train.Saver()  
sess = ed.get\_session()  
save\_path = saver.save(sess, fname)

####code to reload a model and run a new inference with new data########################

sess=tf.Session()  
loader = tf.train.import\_meta\_graph(fname+’.meta’)  
loader.restore(sess,fname)  
graph = tf.get\_default\_graph()  
X =graph.get\_tensor\_by\_name(“X:0”)

w = Normal(loc=graph.get\_tensor\_by\_name(“w/loc:0”),  
scale=graph.get\_tensor\_by\_name(“w/scale:0”),  
value=graph.get\_tensor\_by\_name(“w/sample/Reshape:0”))

b = Normal(loc=graph.get\_tensor\_by\_name(“b/loc:0”),  
scale=graph.get\_tensor\_by\_name(“b/scale:0”),  
value=graph.get\_tensor\_by\_name(“b/sample/Reshape:0”))

qw = Normal(loc=graph.get\_tensor\_by\_name(“qw/loc:0”),  
scale=graph.get\_tensor\_by\_name(“qw/scale:0”),  
value=graph.get\_tensor\_by\_name(“qw/sample/Reshape:0”))

qb = Normal(loc=graph.get\_tensor\_by\_name(“qb/loc:0”),  
scale=graph.get\_tensor\_by\_name(“qb/scale:0”),  
value=graph.get\_tensor\_by\_name(“qb/sample/Reshape:0”))

print(‘qw’,qw.mean().eval(session=sess))

y = Normal(loc=graph.get\_tensor\_by\_name(“y/loc:0”),  
scale=graph.get\_tensor\_by\_name(“y/scale:0”),  
value=graph.get\_tensor\_by\_name(“y/sample/Reshape:0”))

N = 40 # number of data points  
D = 3 # number of features  
X\_test, y\_test = build\_toy\_dataset(N, w\_true)

inference = ed.KLqp({w: qw, b: qb}, data={X: X\_test, y: y\_test})  
inference.run(n\_samples=5, n\_iter=50) ###The value error occurs here

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 12, 2017, 7:48pm UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/14 "2017-07-12T19:48:28Z")

</div>

Because you import the graph, you can’t use `inference.run()` which also builds the computation to run the graph. There’s more bookkeeping involved as you need to write your own training loop.

---

<div class="post-metadata">

**Author:** ![edisoncruise](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@edisoncruise](https://discourse.edwardlib.org/u/edisoncruise)\
**Post date:** [July 17, 2017, 5:50am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/15 "2017-07-17T05:50:11Z")

</div>

Thank you dustin. But I just want to restore the saved model parameters and then restart the inference process with new updated samples. Is there any convenient way to achieve that?

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 17, 2017, 7:35am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/16 "2017-07-17T07:35:33Z")

</div>

I recommend only restoring `tf.Variables` and not the metagraph itself. You can do so by calling `inference.initialize()`; then restore under your parameter saver, then manually run inference in a loop ([http://edwardlib.org/api/inference](http://edwardlib.org/api/inference)). Basically, you’re replacing the `tf.global_variables_initializer().run()` line.

---

<div class="post-metadata">

**Author:** ![edisoncruise](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@edisoncruise](https://discourse.edwardlib.org/u/edisoncruise)\
**Post date:** [July 17, 2017, 9:27am UTC](https://discourse.edwardlib.org/t/saving-model-parameters/24/17 "2017-07-17T09:27:15Z")

</div>

Thank you dustin, but how to restore tf.Variables without restoring the metagraph? I have tried the following code but it cannot find “qw/loc:0”. Also is it possible to creating qw\_loc without specifying its shape. This will raise a dimension mismatch error in tensorflow.

sess=tf.Session()  
qw\_loc=tf.Variable([])  
qw\_scale=tf.Variable([])  
qw\_value=tf.Variable([])  
mylist={“qw/loc:0”:qw\_loc,“qw/scale:0”:qw\_scale,“qw/sample/Reshape:0”:qw\_value};  
loader=tf.train.Saver(mylist)  
loader.restore(sess,fname)