Midjourney has burst onto the scene as one of the most impressive AI image generators available today. With just a text prompt, it can produce stunning, photorealistic images that bring imagination to life. But how does this futuristic technology actually work?

In this technical article, we’ll unpack the core components powering Midjourney to gain a deeper understanding of how it can turn language into imagery.

The Rise of Generative AI

Midjourney belongs to a category of AI called Generative Adversarial Networks, or GANs which involve two neural networks – a generator and a discriminator – working against each other.

The generator creates new content, while the discriminator tries to detect if the output is real or artificially created. This adversarial setup forces the generator to constantly improve in fooling the discriminator, allowing it to produce increasingly realistic outputs.

GANs represent a shift from passive to generative deep learning. Rather than analysing data, GANs can actively imagine and synthesise original content. When applied to images, GANs enable AI to generate photos from scratch based on a text description.

Midjourney utilizes a specific GAN architecture optimized for high-fidelity image generation. But first, let’s explore the key components that make this possible.

VQ-GANs: Mastering a Latent Space for Images

At the heart of Midjourney lies a pretrained VQ-GAN responsible for mapping textual concepts into image space. VQ-GAN builds upon the original GAN formulation by integrating a technique called Vector Quantization (VQ).

VQ is a compression technique that encodes data into discrete representations. Continuous input data like images gets quantized into a finite set of vectors called codes.

In a VQ-GAN, the generator network learns to map these discrete codes to outputs, like images. The codes provide a compact representation of visual concepts that the AI can recombine to synthesise new content.

The generator starts with random noise vectors and transforms them into discrete code outputs using its neural network layers. These codes are taken from an embedding space, known as the latent space.

The latent space is like a library of visual concepts and features abstracted into code. Denoising layers in the generator cleanly map codes to the output image by removing artefacts.

Meanwhile, the discriminator tries to classify the generated images as real or fake, spurring the generator to improve. Through extensive training, the generator learns to produce sharp, realistic images from any combination of codes.

A major advantage of VQ-GANs is that the discrete latent space enables smooth interpolation between codes to create new outputs. The quantisation also stabilizes training compared to continuous GANs.