Understanding GANs and their Capabilities to Generate Realistic Images

To develop and modify new data, Ian Goodfellow’s GANs suggest using two neural networks. One is the generator network responsible for creating new instances of data. To put it simply, the process is the inverse of the classification method of neural networks. Unlike the other neural networks, the generator doesn’t take the raw data and map it to predetermined model outputs. Instead it goes backwards from the output in an attempt to construct the input data that would then be mapped to that output. A generator network in GANs, for example, may start off with a matrix of pixels (consisting of random noise) and try to change them in such a manner that a classifier would categories them as a cat.

The second neural network in GANs is called the discriminator. On a scale of 0 to 1, it assesses the quality rating of the generator’s output. If the quality score is less than the desired level, the data is corrected by the generator and sent back to the discriminator. The GAN continues the loop in super-fast fashion until it reaches a point where it can generate data that accurately maps to the intended output.

Overview of GANs: Source

Let’s not wait any longer and go into the implementation specifics of GANs as we go. Here we’ll demonstrate how we can use Deep Convolutional Generative Adversarial Network implementation (DCGAN) to generate realistic images. In our implementation, Tensorflow is used, and some of the principles stated in the DCGAN paper are followed.

We’ll train our DCGAN on Fashion-MNIST dataset comprising of 70,000 28×28 grayscale images with 60,000 images for training and 10,000 images for testing. Plus, it includes 10 classes of products.

So, before we start, let’s import some necessary packages we’ll use in this implementation.

import tensorflow as tf
import tensorflow.keras as keras
import matplotlib.pyplot as plt
import numpy as np
from IPython import display

We’ll use some helper functions to visualize the results in the later sessions.

def visualize_images(sample_images, columns=None):
    '''displaying fake examples'''
    display.clear_output(wait=False)  

    columns = columns or len(sample_images)
    rows = (len(sample_images) - 1) // columns + 1

    if sample_images.shape[-1] == 1:
        sample_images = np.squeeze(sample_images, axis=-1)

    plt.figure(figsize=(columns, rows))
    
    for index, sample_image in enumerate(sample_images):
        plt.subplot(rows, columns, index + 1)
        plt.imshow(sample_image, cmap="binary")
        plt.axis("off")

After downloading the Fashion-MNIST dataset, training images are converted into batches. Then we’ll apply some basic preprocessing steps.

# downloading the Fashin-MNIST dataset
from keras.datasets import fashion_mnist
(X_train, _), _ = fashion_mnist.load_data()

# normalizing the images
X_train = X_train.astype(np.float32) / 255

# reshape and rescale
X_train = X_train.reshape(-1, 28, 28, 1) * 2. - 1.

#defining the batch size
batch_size = 64

# creating batches of tensors before feeding them into the model
data_set = tf.data.Dataset.from_tensor_slices(X_train)
data_set = data_set.shuffle(1000)
data_set = data_set.batch(batch_size, drop_remainder=True).prefetch(1)

Similarly, we’ll build a discriminator network using the following steps:

• Use the convolutional layers by setting the strides at 2 in order to reduce the size or dimensions of the input images.
• Use “LeakyRELU” as an activation function after every convolution operation.
• Flatten the output features and feed them to a dense layer with single neuron activated by “sigmoid” activation function.

The implementation details along with the model summary are as under.

# building the discriminator network

def discriminator_nn():
  discriminator = Sequential([
                              Conv2D(64, kernel_size=5, strides=2, padding="SAME",
                                     activation=keras.layers.LeakyReLU(0.2),
                                     input_shape=[28, 28, 1]),
                              Dropout(0.3),
                              Conv2D(128, kernel_size=5, strides=2, padding="SAME",
                                     activation=keras.layers.LeakyReLU(0.2)),
                              Dropout(0.4),
                              Flatten(),
                              Dense(1, activation="sigmoid")
])
  return discriminator
discrim = discriminator_nn()
discrim.summary()

Model: "sequential_11"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_6 (Conv2D)           (None, 14, 14, 64)        1664      
                                                                 
 dropout_6 (Dropout)         (None, 14, 14, 64)        0         
                                                                 
 conv2d_7 (Conv2D)           (None, 7, 7, 128)         204928    
                                                                 
 dropout_7 (Dropout)         (None, 7, 7, 128)         0         
                                                                 
 flatten_2 (Flatten)         (None, 6272)              0         
                                                                 
 dense_9 (Dense)             (None, 1)                 6273      
                                                                 
=================================================================
Total params: 212,865
Trainable params: 212,865
Non-trainable params: 0
_________________________________________________________________

Now, let’s combine these two neural networks (i.e. generator and discriminator) for a complete architecture of a DCGAN. Here is how we’ll do that.

#combining the generator and discriminator
dcgan = Sequential([gen, discrim])

As the model distinguishes between fake (0) and real (1), we’ll use “binary_crossentropy” in order to measure the loss between the images. Moreover, we’ll optimize the model loss with RMS_prop optimizer. Let’s now compile the model for training.

# compiling the model
discrim.compile(loss="binary_crossentropy", optimizer="rmsprop")
discrim.trainable = False
dcgan.compile(loss="binary_crossentropy", optimizer="rmsprop")

Let’s define a function that trains our GAN on a given batch of images. The process involves two-phase training steps as under:

• First off, we’ll train our discriminator network in order to distinguish between real and fake images.
• Next, the generator network will be trained to produce fake images that should trick the discriminator to map them as real ones.

def train_dcgan(dcgan, data_set, random_normal_dimensions, epochs=20):
    gen, discrim = dcgan.layers
    for epoch in range(epochs):
        print("Epoch {}/{}".format(epoch + 1, epochs))       
        for real_image_samples in data_set:
            # infer batch size from the training batch
            batch_size = real_image_samples.shape[0]

            # Training the discriminator network - first phase
            # creating random noise samples
            noise_samples = tf.random.normal(shape=[batch_size, random_normal_dimensions])

            # generating fake image samples
            fake_image_samples = gen(noise_samples)

            # listing fake and real images by concatinating them
            conc_images = tf.concat([fake_image_samples, real_image_samples], axis=0)

            # Create the labels for the discriminator
            # 0 for the fake images
            # 1 for the real images
            discrim_labels = tf.constant([[0.]] * batch_size + [[1.]] * batch_size)

            # set the discriminator to trainable
            discrim.trainable = True

            # training the discriminator on the batches of conc_images and the discrim_labels
            discrim.train_on_batch(conc_images, discrim_labels)

            # Training the generator netwrok - 2nd phase
            # feeding noise input samples into DCGAN
            noise_samples = tf.random.normal(shape=[batch_size, random_normal_dimensions])
            
            # labelling generated images as real images
            gen_labels = tf.constant([[1.]] * batch_size)

            # freezing the discriminator network
            discrim.trainable = False

            # training the DCGAN with labels set to true

    dcgan.train_on_batch(noise_samples, gen_labels)

# plotting fake image samples
visualize_images(fake_image_samples, 16)                     
plt.show()

Finally, it’s time to train our DCGAN and see the magic. We’ll call the above function to start the training. While we set the number of epochs to 20, you are free to try the higher number in order to achieve more accuracy. It should take around 30 seconds to execute single epoch in a colab environment. You will visualize the results (fake images generated by our DCGAN) after every epoch during the training process.

So, let’s start off the training by simply executing the below lines of code.

#trining the DCGAN for 20 epochs
train_dcgan(dcgan, data_set, codings_size, 20)

After the training, your final output should look like this.