1 Generator: real inference

The model has the following form:

(1)

N(0,²I_D), d < D. (2)

f(Z;W) maps latent factors into image Y , where W collects all the connection weights and bias terms of the ConvNet.

Adopting the language of the EM algorithm, the complete data model is given by

logp(Y,Z;W) = log[p(Z)p(Y |Z,W)] (3)

+ const. (4)

The observed-data model is obtained by intergrating out Z: p(Y ;W) = ^R p(Z)p(Y |Z,W)dZ. The posterior distribution of Z is given by p(Z|Y,W) = p(Y,Z;W)/p(Y ;W) p(Z)p(Y |Z,W) as a function of Z.

We want to minimize the observed-data log-likelihood, which is

. The gradient of L(W) can be calculated according to the fol-

lowing well-known fact that underlies the EM algorithm:

;W)dZ (5)

. (6)

The expectation with respect to p(Z|Y,W) can be approximated by drawing samples from p(Z|Y,W) and then compute the Monte Carlo average.

The Langevin dynamics for sampling Z p(Z|Y,W) iterates

, (7)

where denotes the time step for the Langevin sampling, is the step size, and U denotes a random vector that follows N(0,I_d).

The stochastic gradient algorithm can be used for learning, where in each iteration, for each Z_i, only a single copy of Z_i is sampled from p(Z_i|Y_i,W) by running a finite number of steps of Langevin dynamics starting from the current value of Z_i, i.e., the warm start. With {Z_i} sampled in this manner, we can update the parameter W based on the gradient L⁰(W), whose Monte Carlo approximation is:

) (8)

(9)

. (10)

Algorithm 1 describes the details of the learning and sampling algorithm.

Algorithm 1 Generator: real inference Input:

• training examples {Y_i,i = 1,,n},
• number of Langevin steps l, (3) number of learning iterations T.

Output:

• learned parameters W,
• inferred latent factors {Z_i,i = 1,,n}.

1: Let t 0, initialize W.

2: Initialize Z_i, for i = 1,,n.

3: repeat

4: Inference step: For each i, run l steps of of Langevin dynamics to sample Z_i p(Z_i|Y_i,W) with warm start, i.e., starting from the current Z_i, each step follows equation 7.

5: Learning step: Update W W +_tL⁰(W), where L⁰(W) is computed according to equation 10, with learning rate _t.

6: Let t t + 1.

7: until t = T

1.1 TO DO

For the lion-tiger category, learn a model with 2-dim latent factor vector. Fill the blank part of ./GenNet/GenNet.py. Show:

• Reconstructed images of training images, using the inferred z from training images.
• Randomly generated images, using randomly sampled z.
• Generated images with linearly interpolated latent factors from (2,2) to (2,2). For example, you inperlolate 8 points from (2,2) for each dimension of z. Then you will get a 8 8 panel of images. You should be able to seee that tigers slight change to lion.
• Plot of loss over iteration.

2 Descriptor: real sampling

The descriptor model is as follows:

, (11)

where p₀(Y ) is the reference distribution such as Gaussian white noise

(12)

The scoring function f(Y ) is defined by a bottom-up ConvNet whose parameters are denoted by . The normalizing constant Z() = ^R exp[f(Y )]p₀(Y )dY is analytically intractable. The energy function is

. (13)

p(Y ) is an exponential tilting of p₀.

Suppose we observe training examples {Y_i,i = 1,,n} from an unknown data distribution P_data(Y ). The maximum likelihood learning seeks to maximize the log-likelihood function

. (14)

If the sample size n is large, the maximum likelihood estimator minimizes the KullbackLeibler divergence KL(P_datakp) from the data distribution P_data to the model distribution p. The gradient of L() is

, (15)

where E denotes the expectation with respect to p(Y ). The key to the above identity is that logZ() = E[ f(Y )].

The expectation in equation (15) is analytically intractable and has to be approximated by MCMC, such as Langevin dynamics, which iterates the following step:

, (16)

where indexes the time steps of the Langevin dynamics, is the step size, and U N(0,I) is Gaussian white noise. The Langevin dynamics relaxes Y to a low energy region, while the noise term provides randomness and variability. A Metropolis-Hastings step may be added to correct for the finite step size . We can also use Hamiltonian Monte Carlo for sampling the generative ConvNet.

We can run n parallel chains of Langevin dynamics according to (16) to obtain the synthesized examples {Y_i,i = 1,,n}. The Monte Carlo approximation to L⁰() is

(17)

,

which is used to update .

To make Langevin sampling easier, we use mean images of training images as the sampling starting point. That is, we down-sampled each training image to a 11 patch, and up-sample this patch to the size of training image. We use cold start for Langevin sampling, i.e., at each iteration, we start sampling from mean images.

Algorithm 2 describes the details of the learning and sampling algorithm.

Algorithm 2 Descriptor: real sampling Input:

• training examples {Y_i,i = 1,,n},
• number of Langevin steps l, (3) number of learning iterations T.

Output:

• estimated parameters ,
• synthesized examples {Y_i,i = 1,,n}.

1: Let t 0, initialize .

2: repeat

3: For i = 1,,n, initialize Y_i to be the mean image of Y_i.

4: Run l steps of Langevin dynamics to evolve Y_i, each step following equation (16).

5: Update _t+1 = _t +_tL⁰(_t), with step size _t, where L⁰(_t) is computed according to equation (17).

6: Let t t + 1.

7: until t = T

2.1 TO DO

For the egret category, learn a descriptor model. Fill the blank part of ./DesNet/DesNet.py. Show:

• Synthesized images.
• Plot of training loss over iteration.