How Does OpenAI's DALL-E Work?
Ever thought a machine can draw exactly how you describe it?
Updated
On this page
Updated November 2026. This was first published on 4 June 2021. The idea is the same. Any number below is what the original piece said then. It is not a new count.
Key takeaways
- Ever thought a machine can draw exactly how you describe it?
- OpenAI recently launched their text to images generation model called DALL-E.
- The name is inspired by the movie WALL-E.
Ever thought a machine can draw exactly how you describe it?
OpenAI recently launched their text to images generation model called DALL-E. The name is inspired by the movie WALL-E.
The model is based on neural networks and is a 12 billion parameter version of GPT-3.
But you must be thinking, what is GPT-3, right?
Well, GPT 3 is a model from OpenAI which can generate write-ups and codes on instruction.
Similarly, DALL-E uses the same algorithm to generate images instead of write-ups. After developing the model, OpenAI found that it has a diverse set of what it can do. It can even create a cartoon version of animals and objects.
It has a talent for combining unrelated concepts after understanding the texts. So, you can get an image of a carrot walking a dog in a tutu or an avocado-shaped armchair.
A few years back, it was unimaginable for a machine to understand what exactly we want and give it to us.
This blog will provide you with an idea about DALL-E, how it works, and how it affects our recent ML.
What is DALL-E, and What Does it Do?.
Back in 2020 July, DALL-E’s creator OpenAI launched their model GPT-3. GPT-3 shook the AI communities all over the world. People started calling it the baby steps towards Artificial General Intelligence (AGI).
GPT 3 can generate human-like texts, including computer codes, poems, stories, and more.
People were stunned by the high-quality writings. Social media started gossiping about this magic bot.
And DALL-E is just an extension of GPT-3, but instead of showing results in the form of texts, it displays images.
In OpenAI’s blog, you can find a couple of images generated by DALL-E.
For instance, with the text prompt “a store front that has word “openai” written on it,” DALL-E showed something like this.
If designers have this option with them, they won’t need to go out for inspiration.
DALL-E’s work is to start showing images when there is a text prompt. But, it uses the OpenAI model “CLIP” to rank the pictures. It tries to identify the best images and shows them to the user.
So, when you are working with DALL-E, you will not get only some random mediocre pictures but only the best.
Presently, no one exactly knows how DALL-E is built except the creators. They didn’t publish any paper on it yet. But, there is a replication of DALL-E by Phil Wang (aka Lucidrains).
But it’s easy to understand that the core uses the same new neural network used on almost all modern ML models.
Transformers came into the scene in 2017. They are neural networks that can be used in huge data sets efficiently. These Transformers were somewhat responsible for the revolution in natural language processing. Even models like BERT, GPT-3, and more use Transformers.
They have also helped in improving the Google search results and even in predicting protein structures.
Generally, all language models come with colossal text data sets. They collect the data from platforms like Wikipedia or use the web crawl information.
But, what’s unique about DALL-E is, it uses a mix of pixels and words. Soon we will know what data set this model used to generate these fantastic results.
But you must note one thing that is DALL-E is a decoder-only Transformer. It means all the texts and images pass together in one token stream to the decoder.
So, the model may look something like this.
In the above picture, as you can see, the encoder receives the dog image. It then designs a “latent space.” Latent space is the space due to the compression of image data where similar data points are together.
The VQ-VAE then uses a batch of vectors which we may call vocabulary for convenience. The encoder has to present an output vector into the latent space which is existing in the vocabulary. So, we are providing the encoder with a set of choices.
The vector then goes into the decoder, which redesigns the dog image.
DALL-E uses the training of VAE where the encoder, decoder, and vocabulary work together.
When a simpleton like you and me sees these images, we would be shocked. But for an engineer, it can lead to tons of questions.
The first question that might strike an engineer is whether the DALL-E created the pictures or it just copied them from somewhere.
So, to prove that DALL-E produces its creations, OpenAI instructed the model to render some weird images that may not be available anywhere.
So, the weird prompt was to generate “a professional high quality illustration of a giraffe turtle chimera.”.
It isn’t easy to imagine that they could have provided the data to exactly show something like this unless they were ready with it. But if they used random prompts like this, then the model’s work is quite impressive.
Ready to See More of DALL-E in the Future?
DALL-E may have been a massive thing if it came before GPT-3. it still has surprised many AI experts. What is unique about DALL-E is its ability to generate images of things that the author may not even anticipate.
In simple words, we won’t know how weird DALL-E can get.
Both DALL-E and GPT-3 are two examples of the greatness of deep learning. It shows how extensive the neural network’s training is where unlabeled internet data can be put together to create something meaningful. In one way, what we see here is an example of “self-supervised learning,” which is also highly versatile.
Although it may seem like the incoming age of general intelligence, the truth is it’s not so. It’s not that difficult to trick these models unless OpenAi has something else in the store that they didn’t say.
We will get to know more about DALL-E when they release the model in public. Then we can play around and learn more from this.
Questions and answers #
What is the main point? #
Ever thought a machine can draw exactly how you describe it?
What else did the original piece say? #
OpenAI recently launched their text to images generation model called DALL-E.