I am trying to implement LSTM in Edward to do one step ahead prediction. The inputs are samples of time series of a sine wave, while the value I try to predict is one step ahead of the sine wave.
I refer to the example from https://github.com/blei-lab/edward/blob/master/examples/lstm.py
I use a similar lstm_cell function as in that example, but I make, the matrix coefficient wx,wh,b, explicit, so that I can debug easily. I meet two problems:
(1) How to define the output variable? If the function RNN return output as a Normal type, in the inference step, the loss becomes nan. If the function return tf.layers.dense directly, the code can give me some nonnan values, but the inference results are not correct.
(2) I am new to use LSTM to do one step ahead prediction. I am not sure how to generate training examples. Suppose I have a time series with 1000 time steps in total. I cut the time series into continuous small one, say 100 each, then I have about 900 samples. But the problem is that most of these samples are overlapped. I am not sure if there is any better way to generate samples.
def RNN(input,timesteps,hidden_size,scale_out,wx,wh,b):
h = tf.fill(tf.stack([tf.shape(input)[0], hidden_size]), 0.0)
c = tf.fill(tf.stack([tf.shape(input)[0], hidden_size]), 0.0)
hs = []
reuse = None
for t in range(timesteps):
if t > 0:
reuse = True
xt = input[:, t, :]
h, c = lstm_cell(xt, h, c,wx,wh,b, name=“lstm”, reuse=reuse)
theloc=tf.layers.dense(h,1, name=“dense”) ###I use the latest hidden values to calculate output
#######problem code#############
#output=theloc
thescale=tf.fill(tf.stack([tf.shape(input)[0], 1]), scale_out)
output=Normal(loc=theloc,scale=thescale )
###################################
return output
#######Run the inference with the following interface########################
Y=RNN(X,timesteps,hidden_size,scale_out,wx,wh,b)
inference = ed.KLqp({wx: qwx, wh:qwh,b: qb}, data={X: x_train, Y: y_train})
inference.run(n_samples=5, n_iter=20)