The Art of Prompting: Why “Prompt Engineering” Is a Skill of the Future
While it might not seem like typing words into an A.I. bot is a skill, the history of photography suggests that it is — and that it’ll be around for decades to come.
Should A.I. Art Win Awards?
A recent New York Times article once again brought A.I. art, and all the controversy behind it, into the limelight. The article describes how Jason Allen, a Colorado-based game designer, won first place at an art fair by submitting an A.I.-generated image. Allen made the image using an A.I. tool called Midjourney. The extent of his contribution was essentially typing a collection of words into the program. According to Allen, he carefully crafted the prompt (text) used to produce the final art image. Submitted under the digital art/digitally manipulated photography category, some judges weren’t aware that the image was created using A.I. until after the awards were given. However, after they learned of its true origins, they still felt like it should win first place for the category.
The story sparked a heated debate on Twitter as to whether the art should have won. Does prompting an A.I. to create art take skill? Should Allen receive any credit? One user on Twitter laughed at the claim that typing words could even be regarded as a skill, saying “I love how they speak of ‘refining their prompt’ as if it was some sort of skill.”
Others claimed that Allen was by no means an artist, but rather a collaborator taking credit for the A.I.’s work.
These criticisms sound completely valid based on our current cultural understanding of A.I. art. However, they also resemble the early criticisms of photography.
Similarities to Photography
As the New York Times article aptly points out, the introduction of A.I.-generated images resembles the invention of photography. When photography became a widespread tool to capture images, many voices also criticized the tool, seeing it as a threat to painting. Charles Baudelaire, a French poet and art critic, claimed that photography was by no means an art. He’s quoted as saying photography was a “refuge of failed painters with too little talent.” Though some may still argue that photography takes little talent or it is not a true art, it’s now widely regarded as an artful skill. Learning to capture lighting, using a camera’s lens, etc. — these are all new skills that are now part of a photographer’s repertoire. Photography can also be considered a collaboration with the camera. The creator isn’t making the scene, they are capturing it using the device.
Just like photography, A.I.-generated images are capturing a moment — they are capturing a scene in a trained machine learning model. Francois Chollet, a renowned machine learning programmer and the author of several books on A.I. and machine learning, also compared A.I. art to photography. He tweeted “Image generation is a form of photography. Photography in a latent space that interpolates between hundreds of millions of images. When you take a photo, you don’t ‘create’ the picture, you take it. You find the scene you want, and you capture it the way you want. It’s curation.”
Chollet goes on to say that “Instead of walking around in the real world and taking pictures of it, you can now walk around in a latent space that interpolates past human creations, and take pictures of it. Latent space photography. And just like photography, it’s art. It requires the eye of the artist.”
In the case of photography, those that quickly adapted to the new technology often profited. The artists that shifted from painting to photography saw success in a burgeoning field, while those particularly skilled in painting continued to thrive in their domain. The introduction of photography made portraits accessible to all, not just the elite that could pay for fine artists. Similarly, perhaps A.I.-generated art is making fine, one-of-a-kind art accessible to all.
Returning to the art fair competition, Allen essentially entered a photography image into a painting competition. He used a cutting-edge tool that requires a different set of skills. When photography was first invented, it wasn’t used to compare to paintings, it was another art entirely. That is why I think that there should be a separate ‘A.I.-generated’ category at the competition, that way the tool is the same for all contestants, and it’s comparing similar skill sets.
Honing the Skill of Prompt Engineering
As a machine learning artist myself, I also believe that prompt engineering (typing text into an A.I. to create an image) is a skill. Perhaps it’s less of a physical skill than photography, but it’s essentially capturing an image from the A.I. Since reading the New York Times article, I too have started using Midjourney to create A.I.-generated images. Sometimes the images created are incredible works that totally inspire me, but other times they look a bit underwhelming. I see others on the Midjourney Discord with incredibly long prompts that are finely detailing exactly what they want in the image. We’re all new photographers learning how to use a camera, but in this case our tool is the A.I. program. It’s inspiring to me. I feel like I’m new on the scene of the latest skill. I already see a few artists on Fiverr offering their “prompt engineering” skills to curate images.
In the weeks that I’ve spent playing with the A.I. Midjourney, I’ve seen my ability to engineer prompts slightly improve. There’s even a channel on the Midjourney discord for community members to swap knowledge about prompt engineering. It was from this chat that I learned that you can specifically tell Midjourney to remove certain things from the image. For one image I experimented with the prompt “forest in the mountains — ar 16:9”. The ‘ar’ parameter specifies that the aspect ratio is 16 by 9. The output is shown below:
Using the negation parameter I was able to remove pine trees from the generated image:
This is one small example of how prompt engineers are using the features available in Midjourney as well as honing their ‘artistic eye’ to create new artworks.
What Will Prompt Engineering Look Like in the Future?
As A.I. advances, I anticipate that “prompt engineering” will continue to evolve. Currently, there are many parameters that one can use to tune an image on Midjourney — just take a look at the documentation that the creators have provided. Through ending parameters, you are able to specify aspect ratio, which machine learning model version to use, style of the photo, any weightings to other images, etc. Future machine learning models may have many of these parameters, which one could compare to features on a camera. The artist needs to master these features to know how to perfect the output.
Perhaps the method of “prompt engineering” will shift to talking to an A.I. assistant. I could easily see Alexa or Google Assistant having an art engine — just speak the words you want to your Google Assistant, and it’ll create the image. Then, you can adjust the image with subsequent voice commands. Perhaps Google Assistant will even ask you if you like the image. These features may diminish the ‘skill’ that it takes to prompt engineer, but it remains a skill nonetheless. Similarly, taking a picture on an iPhone is somewhat of a skill (I don’t just ask anyone to take a picture of me and my friends when we need help). There are many features to help you when taking an iPhone photo (autofocus, portrait mode, etc.), but still only those that have an ‘artistic eye’ take good photos. Similarly, only those with a level of artistic sense will be able to succeed in the world of prompt engineering.
It will also be interesting to see how the term changes over time. Prompt Engineering someday will not just refer to creating images, but also prompting A.I. to write papers, send emails, or even write screenplays (i.e. “Hey Google, could you write a screenplay that mixes the movies ‘Fox and the Hound’ and ‘Mad Max: Fury Road’ with a soundtrack featuring new songs that sound like David Bowie). We may use personalized A.I. engines. “Hey Google, send an email to my boss using my style of writing that says I can’t make it to work today.” “Hey Google, use my previous artworks to create a new art of a sunset that looks like I painted it.”
If A.I. programs continue to see enough innovation, perhaps someday we will see prompt engineers using A.I. assistants to create new shows, new movies, new music, new books, new car designs, or new city planning designs. Maybe that idea terrifies you, or maybe it inspires you. In the end, these A.I. tools are meant to inspire and uplift artists if they choose to embrace it.
As Chollet puts it, “Reports of art’s death are greatly exaggerated. New tools don’t kill art, they expand it.” Photography showed us that artists will continue to adapt to new tools. Prompt Engineering is the newest tool; artists will thrive despite — and maybe more so because of — its appearance on the scene.
