Truly multimodal
GPT-4o image generation has officially arrived nearly a year after its initial introduction, showcasing the astounding capabilities of a true multimodal model.
I. Introduction
Last year, GPT-4o was released, with "o" standing for "omni," signifying its emergence as a true multimodal model, rather than simply being plugged into other models. Recently, Gemini and Grok have also announced models capable of directly generating image outputs.
Today, GPT-4o has finally introduced image generation with the capability to produce stunning images. Complex images can now be created effortlessly using simple prompts, and errors can be easily corrected by merely describing them. Many significant improvements have happened in image generation.
OpenAI, which typically does not disclose specific technical details, provided some information about the underlying model in their GPT-4o documentation ("GPT-4o System Card – March 25, 2025"):
"Unlike DALL-E, which operates as a diffusion model, 4o image generation is an autoregressive model natively embedded within ChatGPT."
In this post, we will explore how GPT-4o and other similar models achieve this capability.
II. One Big Autoregressive Transformer
Fundamentally, an autoregressive transformer model predicts the next token based on previously seen tokens:
Tokenization of data: For example, text has its unique tokens, images have their tokens, audio has its tokens.
Transformer sequentially processes tokens: It doesn't differentiate between data types, thus allowing it to learn broad contexts from correlations between different information types.
The simplified workflow is:
Input tokens → Autoregressive Transformer → Output tokens (text/image/audio)
Output tokens → Decoded back into their intended formats (sound or pixel arrays)
III. Key characteristics of the new GPT-4o
OpenAI has even released an image highlighting the fundamental features of integrating image output into their large-language model (LLM):
1. Advantages:
Generates images leveraging the LLM’s existing knowledge (world knowledge): The Transformer model, trained on vast amounts of information, generates images that are richer in context.
Superior rendering of text-image combinations: Easier and more accurate integration of textual information into images, something previously challenging to achieve at high quality with diffusion-based methods (such as DALL-E).
Contextual understanding: The model directly understands and follows even complex semantic instructions, including input from other images.
Unified single stack: Instead of plugging multiple stacks (like GPT + agent + DALL-E), GPT-4o now seamlessly handles both input and output.
2. Challenges:
Significant bit-rate variations across data types:
Text: Large vocab (~50,000 tokens, ~16 bits/token).
Images: Typically fewer tokens (~8192 tokens, ~13 bits/token), encoding large visual information amounts, resulting in significant detail loss compared to original pixels.
High computational cost, sub-optimal efficiency (= compute not adaptive): Autoregressive Transformer models aren't originally optimized for “heavy” data (images, audio), causing computational overload, reduced performance, and higher latency when generating large or high-resolution images. GPT-4o creates highly detailed and sharp images, meaning image generation can sometimes take around 30 seconds, significantly affecting user experience (UX).
3. OpenAI’s solutions to these issues:
Applying compressed representations: Instead of outputting pixels directly, transformers produce compressed tokens, improving computational efficiency while retaining information.
Integrating autoregressive transformers with powerful decoder models: Utilizes stronger diffusion decoders.
IV. Conclusion
GPT-4o represents a major turning point that paves the way for the next generation of AI. Being genuinely multimodal opens up an abundance of possibilities for user experiences, potentially leading to more creative applications such as multimedia AI personal assistants.
Intensive and costly computation remains a future challenge; optimization efforts must continue. However, with computational power increasing rapidly, an advanced virtual assistant like Samantha no longer seems too distant.