Imagine spending a weekend afternoon with friends at an art museum: nodding with crossed arms, desperately searching for something insightful to say. The vast majority of paintings you stroll past are immediately forgotten, but some stick in your mind. As it turns out, the paintings you remember are likely the same ones everyone else does.
There’s a scientific term for that: image memorability. “It’s the idea that, essentially, there are some intrinsic patterns that make some content more memorable than others,” says Camilo Fosco, a PhD student studying computer science at MIT and the CTO of Memorable AI, a startup that uses machine learning to test how engaging content will be for advertisers and creators. In other words, certain pieces of art have that je ne sais quoi—and now a team of scientists is using AI to figure out what it is.
In a study published earlier this month in the Proceedings of the National Academy of Sciences, University of Chicago researchers Trent Davis and Wilma Bainbridge show that the memorability of artwork is not only consistent across people, but predictable by AI. In an online experiment, they pulled about 4,000 paintings from the Art Institute of Chicago’s database, excluding anything the institute labeled “boosted,” or especially famous. Over 3,200 people viewed hundreds of images so that each painting was seen by about 40 people. Then the volunteers were shown the paintings they had seen mixed in with ones they hadn’t and asked whether they remembered them or not. People were really consistent—everyone tended to remember (or forget) the same images.
Using a deep learning neural network called ResMem, designed by data scientist Coen Needell as part of his master’s thesis in Bainbridge’s psychology lab, the research team was able to predict how likely each painting was to be memorable. ResMem roughly mimics how the human visual system passes information from the retina to the cortex, first processing basic information like edges, textures, and patterns, then scaling up to more abstract information, like object meaning. Its memorability scores were very highly correlated with those given by people in the online experiment—even though the AI knew nothing about the cultural context, popularity, or significance of each artwork.
Counterintuitively, these findings suggest that our memory for art has less to do with subjective experiences of beauty and personal meaning, and more to do with the artwork itself—which may have major implications for artists, advertisers, educators, and anyone hoping to make their content stick in your brain. “You might think art is a very subjective thing,” says Bainbridge, “but people are surprisingly consistent in what they remember and forget.”
Although the online experiment was an intriguing start, she continues, “it’s more interesting if we can predict memory out in the real world.” So along with Davis, then an undergraduate double-majoring in neuroscience and visual arts, Bainbridge recruited 19 more people to actually wander through the museum’s American Art wing as though they were exploring with friends. The only requirement was that they see each piece at least once. “Especially as an artist myself, I wanted the results to apply to the real world,” says Davis, who is now the lab’s manager. “We wanted it to be a natural and enjoyable museum experience.”
Upon leaving the exhibit, each participant took a memory test on their phone. As it had done for the online experiment, ResMem strongly predicted which paintings people would remember.
What did these standout paintings have in common? Well, they were more likely to be large, or surrounded by larger pieces. But they didn’t share a subject, historical period, color palette, or emotional theme. So Bainbridge’s team pressed harder to try to find out what people were picking up. In a third experiment, 40 additional online participants rated the beauty, emotional tone, familiarity, and interestingness of each painting that the people from the second test saw in-person at the Art Institute. The first three factors—as well as basic visual features like color, brightness, and amount of clutter—turned out to be unrelated to memorability. “The only thing that was actually linked to memorability was how interesting people felt the piece was,” says Bainbridge.
But it’s hard to say what “interesting” means. It’s a vague term that can connote anything from curiosity to veiled distaste. Bainbridge suspects that what people find interesting has to do with how the artwork interfaces with human culture. Some of the most memorable paintings, according to both human participants and ResMem, were either humorous or vulgar. One of the highest-scoring paintings, for example, features two oblong potatoes dangling from a string in a suspiciously testicular fashion. “We’re going to pick up on that,” Bainbridge says—even if a neural network can’t tell you why.
Fosco has noted the same thing in his own work at Memorable AI—what he calls a “clear correlation between weirdness and memorability.” And Zoya Bylinksii, a senior research scientist at Adobe, came to a similar conclusion in a recent study of people’s aesthetic judgments of artwork. Using methods similar to the Chicago team’s, she found that while people tend to rate natural landscapes as the most beautiful, they aren’t the most memorable. “We don’t remember something because it’s beautiful,” she wrote in an email to WIRED. “We remember it because it stands out, because it’s strange, because it’s unlike what we’ve seen before.”
In previous studies, Bainbridge found that the brain—particularly parts of the visual stream and the medial temporal lobe—reacts differently to memorable images than to forgettable ones, even when people aren’t doing a memory task. She thinks that the brain quickly calculates which visual inputs to prioritize, and which can be thrown away. But we still don’t know what exactly the brain (or an artificial neural network, for that matter) does to separate these experiences from others.
ResMem, Needell says, “is a black box, for sure.” In a previous study, he tried to figure out what the model was thinking by generating pictures that maximally activated each of its components. (It’s sort of like figuring out your pet’s favorite toy by waving each one and seeing which excites them the most.) The results were baffling: psychedelic swirls of object fragments and rainbows. “One is really stuck in my head—it just pulled out the bottom half of people’s faces and turned it into a fractal pattern.”
While “interesting” images may tend to be more memorable, says Bainbridge, many other factors influence whether artwork lingers in your mind. Very strong negative emotions, like disgust or fear, will make an experience stick, she says. It’s possible that the researchers didn’t see that particular effect in this study because none of the paintings were truly grotesque or horrifying (as these characteristics would be uncommon in a museum like the Art Institute).
The brain also tends to prioritize surprising, new, and unusual things. “We remember the past so we can predict the future better,” writes Bylinskii. She speculates that certain works of art may become culturally famous because they stand out—whether by being stylistically unique, touching on an unusual subject, or violating expectations. “For these reasons, they get ingrained in people’s brains,” she says.
Now Davis and Bainbridge are trying a new approach to figuring out what makes art memorable: asking artists for help. Bainbridge’s lab is running a contest challenging artists to create their most memorable and forgettable artwork. Submitted pieces will be shown in a gallery, and viewers’ memories will be tested. The pieces that people are most likely to remember (or forget) will win, and hopefully provide some clues about what makes art pop. (Submissions are open to any US-based artist willing to ship their work to Chicago by January 1, 2024.)
Beyond seeing whether artists can figure out what makes an image enduring, Davis hopes to study how memorability is shaped during the art-making process. Artists must include at least five photos of their work in progress. “Adding a brushstroke here or there changes memorability,” Davis says, “so we’re using ResMem to track changes in memorability with every change to the painting.”
Any time we learn about how the human brain prioritizes information, Bylinskii writes, “it creates opportunities for manipulation, toward good or bad outcomes.” ResMem is copyrighted by the University of Chicago, so corporations can’t use it to, for instance, make catchier advertisements. “That helps us go to bed at night,” Bainbridge says. But companies like Fosco’s are already using their own deep learning models to help clients make subtle changes to ad content to boost click rates and recall. Fosco also envisions educators harnessing this science to make slide decks and infographics easier for students to remember.
AI that can predict how much a piece of art will stick with a viewer—and possibly give artists the power to fine-tune their work to cater to their audience—might sound scary to visual artists. Along with generative AI tools like Dall-E, people might fear it will hinder their creative process or expression, Davis says. Davis is often asked whether he’ll apply these findings to his own art, but he says he tries to avoid allowing his neuroscience insight to seep into his creative world. He envisions ResMem as a tool gallery curators and artists could use to hone the presentation of their work, but not as a replacement for their own creative direction.
While exploiting the power of memorability has the potential to threaten artists and anyone who consumes entertainment, Bylinskii believes that figuring out what makes an image stick can also arm people against manipulation. “The solution is not to create less knowledge,” she writes, “but to make the knowledge so widespread that others can recognize when it’s being used against them.”
