-----SPaCeTriPPin-----

This has been done many times on the web and the method here reveals pretty results and makes use
of Deep Dreams code available in the Tensorflow Examples website.

I call this one cloud farts:



The process works by first downloading the pretrained VGG16 model and importing the Tensorflow api in python:

Code: Select all

import tensorflow as tf
import keras.backend as K
import numpy as np

import matplotlib as mpl
import matplotlib.pyplot as plt
import IPython.display as display
import PIL.Image
import cv2

from tensorflow.keras.preprocessing import imageimport numpy as np

import matplotlib as mpl
import matplotlib.pyplot as plt
import IPython.display as display
import PIL.Image
import cv2

from tensorflow.keras.preprocessing import image
#Download a prelearnt Deep Net
base_model = tf.keras.applications.VGG16(include_top=False, weights='imagenet')


We are going to make another model using Keras which will output just the activations for one specific filter in one specific layer of the net. If we optimise an image with the gradients from this output then we can grow the features within the image. This is essentially what deep dream does but without being filter specific.

We start by creating an image composed of random noise in which to grow our features:

Code: Select all

# Create a random image.
def mkimage(max_dim=None):
 
  imarray = np.random.rand(50,50,3) * 255
  img = PIL.Image.fromarray(imarray.astype('uint8'))
  if max_dim:
    img.thumbnail((max_dim, max_dim))
  return np.array(img)
 
    # Normalize an image
def deprocess(img):
  img = 255*(img + 1.0)/2.0
  return tf.cast(img, tf.uint8)

# Display an image
def show(img):
  #plt.imshow(img)
  display.display(PIL.Image.fromarray(np.array(img)))


# Downsizing the image makes it easier to work with.
original_img = mkimage(max_dim=500)
show(original_img)

# Get some data about the base image
img = tf.constant(np.array(original_img))
base_shape = tf.shape(img)[:-1]
float_base_shape = tf.cast(base_shape, tf.float32)
OCTAVE_SCALE = 1.30


We preset some values such as the shape or size resolution of the base image. Aswell as growing the features in the image we are steadily going to grow their resolution by increasing the size of the image using a scale of 1.3 per iteration until the image is 130x its base size. This means optimizations begin with a low resolution which slowley increases with each iteration, producing larger scaled features.

Here we select which layer and which filter to look at and create a dream_model which outputs just the outputs for that filter:

Code: Select all

# Maximize the activations of this layer's filter
name = 'block4_conv3'  #layer name
filter_index= 46;      #filter index
layer_output = base_model.get_layer(name).output
foutput = K.mean(layer_output[:, :, :, filter_index])  #Mean of the filter acts

# Create the feature extraction model using the the filters outputs
dream_model = tf.keras.Model(inputs=base_model.input, outputs=foutput)


# Example layer block4_conv1 has 512 units with 256 inputs and a 3x3 kernel?
#1792 parameters = 64 units x 3x3 kernel weights x 3 inputs + 64 outputs!!
print(base_model.get_layer('block4_conv1').kernel.shape)
print(base_model.get_layer('block1_conv2').input)

base_model.summary()

Also take a look at all the the layers in the model summary.

To calculate the loss we use the activations relevent for that layer:

Code: Select all

def calc_loss(img, model):
  # Pass forward the image through the model to retrieve the activations.
  # Converts the image into a batch of size 1.
  img_batch = tf.expand_dims(img, axis=0)
  layer_activations = model(img_batch)
  loss = layer_activations;
  return  loss


As the output of the dream_model is the mean over the filters activations we simply return the loss as the models output. In the Deep Dream code the model outputs a tensor over all activations for each layer it uses. Here it's simpler. But you can play around with using either methods.

Here is the Deep Dream Class functions taken directly from the Deep Dream Demo:

Code: Select all

class DeepDream(tf.Module):
  def __init__(self, model):
    self.model = model

  @tf.function(
      input_signature=(
        tf.TensorSpec(shape=[None,None,3], dtype=tf.float32),
        tf.TensorSpec(shape=[], dtype=tf.int32),
        tf.TensorSpec(shape=[], dtype=tf.float32),)
  )
  def __call__(self, img, steps, step_size):
     
      print("Tracing")
      loss = tf.constant(0.0)
      N = tf.constant(1.0)
      tf.cast(N, tf.int32)
      for n in range(steps):
        with tf.GradientTape() as tape:
          # This needs gradients relative to `img`
          # `GradientTape` only watches `tf.Variable`s by default
          tape.watch(img)
          loss = calc_loss(img, self.model)
       
        #Count up the Octaves
        N +=1
        if N > 100:
           N=tf.constant(100.0)
       
        # Calculate the gradient of the loss with respect to the pixels of the input image.
        gradients = tape.gradient(loss, img)

        # Normalize the gradients.
        gradients /= tf.math.reduce_std(gradients) + 1e-8
       
        # In gradient ascent, the "loss" is maximized so that the input image increasingly "excites" the layers.
        # You can update the image by directly adding the gradients (because they're the same shape!)
        img = img + gradients*step_size
        img = tf.clip_by_value(img, -1, 1)
       
        #scale up image size
        img = tf.image.resize(img, [520*(N/100),520*(N/100)])
       

       
      return loss, img

deepdream = DeepDream(dream_model)

I have adapted the function to resize the image for each step of the iteration using the OCTAVES. The image is optimized by simply adding the gradients calculated by using gradient.tape. Which yeilds the gradient of the loss (outputs of model) with respect to the image (pixel values)

The main function which runs it:

Code: Select all

def run_deep_dream_simple(img, steps=100, step_size=0.01):
  # Convert from uint8 to the range expected by the model.
  img = tf.keras.applications.inception_v3.preprocess_input(img)
  img = tf.convert_to_tensor(img)
  step_size = tf.convert_to_tensor(step_size)
  steps_remaining = steps
  step = 0
  while steps_remaining:
    if steps_remaining>100:
      run_steps = tf.constant(100)
    else:
      run_steps = tf.constant(steps_remaining)
    steps_remaining -= run_steps
    step += run_steps

    loss, img = deepdream(img, run_steps, tf.constant(step_size))
   
    #display.clear_output(wait=True)
    #show(deprocess(img))
    print ("Step {}, loss {}, Steps Remaining {},Run Steps {}".format(step , loss, steps_remaining, run_steps ))
    print(loss)


  result = deprocess(img)
  display.clear_output(wait=True)
  show(result)
  print(loss)
  print(img.shape)
  return result



Now just runit and check the result, change the stepsize to get a larger / smaller image. Larger images take more time to process. So you can runit at 20 steps to quickly look at the feature and then grow it large with 100 steps.

Code: Select all

dream_img = run_deep_dream_simple(img=original_img,
                                  steps=45, step_size=0.047)


Here are some examples, see if you can find more by exploring different layers, filters within the pretrained models available on tensorflow. Enjoy.