# Loss increasing in inference of Poisson Factorization

**URL:** <https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221>\
**Category:** General\
**Created:** [July 6, 2017, 2:15pm UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221 "2017-07-06T14:15:14Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![JHChen](https://avatars.discourse-cdn.com/v4/letter/j/bc79bd/32.png) [@JHChen](https://discourse.edwardlib.org/u/JHChen)\
**Post date:** [July 6, 2017, 2:15pm UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/1 "2017-07-06T14:15:14Z")

</div>

I am try to implement a simple **Poisson Factorization**

```
theta ~ Gamma(a, b)
beta ~ Gamma(c, d)
x ~ Poisson( theta . beta )

```

My codes:

```
from edward.models import Poisson, Gamma
import edward as ed

# Toy dataset
M = (np.random.random((300,200))>0.9).astype(int)

# Model
theta = Gamma(1.*tf.ones([300, 10]), 1.)
beta = Gamma(1.*tf.ones([200, 10]), 1.)
X = Poisson(tf.matmul(theta, tf.transpose(beta)))

# Inference
q_theta = Gamma(tf.nn.softplus(tf.Variable(tf.random_normal([300, 10]))),
                tf.nn.softplus(tf.Variable(tf.random_normal([300, 10])))) 
q_beta = Gamma(tf.nn.softplus(tf.Variable(tf.random_normal([200, 10]))),
               tf.nn.softplus(tf.Variable(tf.random_normal([200, 10])))) 

# Inference
inference = ed.KLqp({theta: q_theta, beta: q_beta}, data={X: M})
inference.initialize(n_print=30, n_iter=300)
tf.global_variables_initializer().run()
for t in range(inference.n_iter):
    info_dict = inference.update()
    inference.print_progress(info_dict)
    if t % inference.n_print == 0:
        print '\tloss: %f'%info_dict['loss']

```

**Why I got the increasing loss? and ended up nan**

1/300 [0%] ETA: 743s | Loss: 1969714.125 loss: 1969714.125000  
30/300 [10%] ███ ETA: 23s | Loss: 5959987.500 loss: 5910604.000000  
60/300 [20%] ██████ ETA: 10s | Loss: 27670352.000 loss: 29153144.000000  
90/300 [30%] █████████ ETA: 6s | Loss: 335439648.000 loss: 430829216.000000  
120/300 [40%] ████████████ ETA: 4s | Loss: 5386785280.000 loss: 5760766464.000000  
150/300 [50%] ███████████████ ETA: 2s | Loss: 88767381504.000 loss: 92775710720.000000  
180/300 [60%] ██████████████████ ETA: 1s | Loss: nan loss: nan  
210/300 [70%] █████████████████████ ETA: 1s | Loss: nan loss: nan  
240/300 [80%] ████████████████████████ ETA: 0s | Loss: nan loss: nan  
270/300 [90%] ███████████████████████████ ETA: 0s | Loss: nan loss: nan  
300/300 [100%] ██████████████████████████████ Elapsed: 3s | Loss: nan

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 6, 2017, 4:58pm UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/2 "2017-07-06T16:58:52Z")

</div>

`ed.KLqp` fares poorly on Gamma latent variables. For scalable Poisson factorization, I recommend EM algorithms such as the one used by @dadaromeo: [https://github.com/dadaromeo/recsys-hpf](https://github.com/dadaromeo/recsys-hpf).

---

<div class="post-metadata">

**Author:** ![JHChen](https://avatars.discourse-cdn.com/v4/letter/j/bc79bd/32.png) [@JHChen](https://discourse.edwardlib.org/u/JHChen)\
**Post date:** [July 7, 2017, 1:20am UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/3 "2017-07-07T01:20:32Z")

</div>

Thanks I will give that a try.

---

<div class="post-metadata">

**Author:** ![mac449](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/mac449/32/18_2.png) [@mac449](https://discourse.edwardlib.org/u/mac449)\
**Post date:** [July 12, 2017, 8:32pm UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/4 "2017-07-12T20:32:44Z")

</div>

Hi JHChen and Dustin,

I’ve been working on Edward model for Poisson factorization with Gamma latent variable using ed.Kqp. I wasn’t getting “nans” for the loss. However, my predictions weren’t right. I’ll also look into EM algorithm as well.

Best,  
Mark

---

<div class="post-metadata">

**Author:** ![TheLeanAcademic](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@TheLeanAcademic](https://discourse.edwardlib.org/u/TheLeanAcademic)\
**Post date:** [July 14, 2017, 9:11am UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/5 "2017-07-14T09:11:49Z")

</div>

Hi Dustin. I’ve run @dadaromeo’s EM algorithm on my own data. Works fine out-of-the-box, but if I change the number of iterations then I get the following error. Any chance you could explain what the error means? The number of iterations was originally set to 500, and if I increase it beyond 500 it appears to fail when it reaches 500…

![](https://cdck-file-uploads-canada1.s3.dualstack.ca-central-1.amazonaws.com/flex035/uploads/edwardlib/original/1X/f179717a5b63814a4e37fde9fe43a0995d4e0734.png)

---

<div class="post-metadata">

**Author:** ![mac449](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/mac449/32/18_2.png) [@mac449](https://discourse.edwardlib.org/u/mac449)\
**Post date:** [July 14, 2017, 9:33pm UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/6 "2017-07-14T21:33:55Z")

</div>

Hi Matthew,

How did you setup the inference procedure for the Poisson Factorization using EM algorithm in Edward?

Best,  
Mark

---

<div class="post-metadata">

**Author:** ![TheLeanAcademic](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@TheLeanAcademic](https://discourse.edwardlib.org/u/TheLeanAcademic)\
**Post date:** [July 16, 2017, 8:50am UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/7 "2017-07-16T08:50:39Z")

</div>

Hi mac. My code is below.  
 ![](https://cdck-file-uploads-canada1.s3.dualstack.ca-central-1.amazonaws.com/flex035/uploads/edwardlib/original/1X/c39e13018168d5e1709133bcb0f3f8e5896cdeaa.png)

---

<div class="post-metadata">

**Author:** ![TheLeanAcademic](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@TheLeanAcademic](https://discourse.edwardlib.org/u/TheLeanAcademic)\
**Post date:** [July 16, 2017, 8:51am UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/8 "2017-07-16T08:51:59Z")

</div>

\*Mark. Sorry …

---

<div class="post-metadata">

**Author:** ![dustin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/dustin/32/134_2.png) [@dustin](https://discourse.edwardlib.org/u/dustin)\
**Post date:** [July 17, 2017, 7:33am UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/9 "2017-07-17T07:33:21Z")

</div>

In the algorithm you set the Empirical approximation as holding `t=500` samples. If `n_iter` is greater than `t`, then the Empirical approximation cannot hold any more samples so the error is raised.

---

<div class="post-metadata">

**Author:** ![mac449](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/mac449/32/18_2.png) [@mac449](https://discourse.edwardlib.org/u/mac449)\
**Post date:** [July 17, 2017, 1:45pm UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/10 "2017-07-17T13:45:29Z")

</div>

Hi Matthew,

Thank you. It was instructive!

---

<div class="post-metadata">

**Author:** ![mbernste](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.edwardlib.org/mbernste/32/254_2.png) [@mbernste](https://discourse.edwardlib.org/u/mbernste)\
**Post date:** [February 21, 2018, 9:05pm UTC](https://discourse.edwardlib.org/t/loss-increasing-in-inference-of-poisson-factorization/221/11 "2018-02-21T21:05:12Z")

</div>

Thank you very much for the support that you’ve given across this forum. Would you mind expounding on exactly why KLqp fares poorly on Gamma latent random variables? More generally, when will KLqp fare poorly and what is the fundamental cause? Or perhaps could you point to a resource that will explain the issue? Thanks!