ESSAYS Mousse 60

Light Play: Twisting Reality and Deepening Narrative through Augmentation

by Nora N. Khan


Meriem Bennani: FLY installation view at at MoMA PS1, New York, 2016. Courtesy: the artist and MoMA PS1, New York. Photo: Pablo Enriquez


While the jury is still out on the promise of virtual reality applications and devices in an artistic context, more and more contemporary artists in the last two years have been using augmented reality to compelling ends. Museums, even those famously slow at adopting technology, are commissioning artists working with augmented reality. Where virtual reality and game simulations are closely bound within the all-subsuming frameworks established by their technologies, augmented reality feels crucially different. Augmentation deploys lightness and play to highlight, reframe, and rearticulate visual narratives through subtle gestures and edits.


What is augmented reality, and the practice of augmentation? Broadly, augmented reality encompasses a range of flexible digital practices, tools, and techniques that layer media, including images, writing, chat programs, and virtual objects, atop one’s existing visual environment. The augmentation takes place on screens; through holograms, apps, chat rooms, and drawing programs; and through viewing devices like helmets, headsets, or glasses.

Designers of augmented reality devices, such as Microsoft’s HoloLens, are deploying artistic and creative approaches as they consider the aesthetic and theoretical impact of their products on digital visual culture. Early in writing this piece, I spoke to Yasaman Sheri, one of the first designers on the Microsoft HoloLens team who helped create the first operating system for Windows Holographic. In contrast to mobile or desktop computing design, Sheri had a chance to think outside of two dimensions. She had to think of how a user moves through space, and the many stimuli they have to process. She told me how the design experience allowed her a “chance to define refine and design frameworks and new paradigms for how to interact and think with holographic objects in spatial computing,” drawing on research in peripheral vision, ontology, object-ness, and spatial memory while “really considering the behavioral, cultural, and philosophical aspects of interaction design.”

Designing for augmentation feels “more like designing for behaviors and psychology,” she says, as the human brain is predisposed to interacting with tangible physical objects. Users look away from a hologram and then back, swiftly, expecting the object to be there; building on this, Sheri explains, holographic gestures, like hands waving or signaling, do a great deal of emotional and psychological work. In Copenhagen, Sheri teaches students how augmentation is a matter of expanding on “not only vision and sound, but also olfaction, taste, proprioception, touch, and haptics.” Extending every sense, augmentation adds another layer of possible experience in which holographic objects hold as much primacy as physical ones.Augmented reality’s practical applications are most evident in medicine, real estate, and teaching. Advanced augmentation could let you, through one seamless interface, call your parents in California while you scout the chemical composition of the walls at a construction site while sending diagrams of that site to a business partner in Shanghai. Basic augmentation happens most familiarly on social media, through user manipulation of stickers, emoji, and cartooning (as found in Snapchat and Instagram) layered atop existing images.

As an emerging art form with its own vocabulary, augmentation is used for powerful storytelling and compelling framing of existing work, opening up unseen elements of our world. Four artists with a proven and evolving engagement with digital visual culture—Meriem Bennani, Ian Cheng, Tuomas A. Laitinen, and Martine Syms—are currently positioning augmentation and augmented reality techniques as an essential layer atop their respective inquiries. Through scanning, cartooning, and archiving, each artist uses augmentation to expand the viewer’s sense of spatial potential and haptic range, deepen their access to both the work and space surrounding it, and think about pressing social and political themes in an arguably more effective fashion than full immersion. Further, each of these four contemporary artists uses augmented reality and augmentation in a radically different fashion, some creating surreal, original augmented worlds to pull viewers through both futurist and historical narratives, and others deploying, through AR’s unique affordances, daggers of vital critical commentary that could not be accessed otherwise. In examining each one’s approach to AR, I started to gain a portrait of the vast potential of digital augmentation.


Meriem Bennani, FLY (still), 2016. Courtesy: the artist and SIGNAL, New York


A first insight: AR is different from other technologies like VR that are themselves the content, and is instead an effective tool “for manufacturing emotions, tension, and amplifying subtleties around a central narrative structure,” as Meriem Bennani describes. Bennani’s practice has unfolded over time as a series of experiments in augmented realization, from her delightful cartooning in vignettes on Instagram to her solo installation at MoMA PS1, FLY. She deploys augmented techniques—sound effects, animations, cartoons—to develop narrative humor around her personally sourced images of daily life in Morocco. She tracks graphics and animations over real-time images, intervening to modify the meaning in postproduction and animation.

Her edits result in a continuously expanding imaginative dream world that asks viewers briefly to suspend disbelief. On Instagram, Bennani invites followers in on wildly creative stories that unfold and close in ten to fifteen seconds. In one frame, she spins out of one city into a new one, in a dramatic Disney whirl. A cashier working at Dunkin’ Donuts, going about her business, suddenly sports a layered donut crown digitally rendered atop her hijab. Older Moroccan women coyly tell stories about love as they are vanished into vases and then magicked back into frame again. We learn from her family and the strangers her eye lands on via moments of their days edited to highlight their warmth, wit, compassion, and humor.

Bennani situates augmentation as related to a long tradition of experimental editing in film. “Coming from animation, I do feel like the only way I know how to convey emotions, tone, mood, or story is through intensive intervention and labor around existing material,” she writes. “I like the idea of shifting the ruleset of a place I experience with its many realities, which a Western audience might read with a number of assumptions,” Bennani writes. “Placing scenes from my Morocco into an augmented or modified reality allows for the narrative to develop within an imagined version of a country, which is very similar to the old trick of literature set around the creations of utopian or dystopian places.”

Although her approach to augmentation is extremely intuitive, and her video formats have changed in tandem with the increasing complexity of her interventions, Bennani avoids theorizing the practice too heavily: “An effective augmentation is similar to a good joke; you just know when it lands right because it makes people laugh.” The key to its success is its lightness; a failed augmentation is often one that is overly done, “heavy handed, and painful to watch.”

The augmented world also allows viewers to experience affect on new registers and in different configurations, which they might not have had access to before. Augmented emotion could come, for example, out of the tension between the viewer’s awareness and the lack of awareness of the people in the film. As Bennani describes, her “augmentations often place the people in the frame in a vulnerable position, because you can’t help but think about them not being aware of the special effect in the time of shooting.” This constant time rift between real time and AR time creates a mesmerizing game for audiences, who are pulled into the comedy, and into the maker’s universe, as each augmented gesture reveals the camera holder’s thinking and interpretive process.

Bennani’s augmentation demands a great deal from a critical and attentive viewer. Consider her use of timing. Watching, the audience might quickly grow to expect more special effects and entertainment, at which point the artist might withdraw or withhold. “I like this seduction,” the artist muses, “because it keeps me alert in my choices, and viewers as well in their watching.” AR allows viewers to be more attuned to their own (real) world. They are given a crucial distance that allows for criticality and space to process Bennani’s reaction.

Viewers might find themselves caught off-guard by the emotional impact of Bennani’s playful pieces, finding them intensely charged or highly emotional. Her augmentations are narrative enhancers, conveying vulnerable personal moments through focused gesture. “Many people think of AR as cold and computer-generated,” she notes, but the similarities between her augmented world and the original “preserve the emotional charges,” deepening and amplifying them.


Tuomas A. Laitinen, 1 (flyer version), 2016. © Tuomas A. Laitinen. Courtesy: the artist and Helsinki Contemporary, Helsinki. This image works as a target for the AR application Arilyn. Get Arilyn from Apple Store or Google Play.


Augmentation, by externalizing thinking into cartooning and layered play, in joyful, slight twists of the visual field, offers a novel mode of access to discourse and criticality. These techniques introduce small shifts in people’s perceptions and reveal our current technological state as polysemic, rife with competing meanings—a product of our increasing ease with using competing tools, devices, and media all at once.

Augmentation’s lighter touch sidesteps debates over “how to engineer empathy” through immersion, which has placed unrealistic moral demands on digital visual culture to make us feel or think or to parse reality in a certain way. The idea that a virtual world must be fully immersive to be considered successful is a tired one. Both critical discourse around games and consumer reviews of virtual reality and mobile games frequently insist on the importance of how completely one is transported and consumed by a world. Further, immersion is a privileged position. Who has the time for full immersion? Most people have to work, and cannot sink eighty hours into a game, no matter how evocative its world. More people might, though, have time for slight shifts layered atop a version of hyperreality and hyperreal perception.

Yasaman Sheri points out that designing for empathy is “more of a social challenge” than a technological one. Seeing from another’s point of view works better “as a metaphor; I feel skeptical that empathy would be achieved from such a literal solution,” she tells me. “While my eyes may be seeing the same images as you may be seeing, my mind makes sense of it in a different way, as I was raised differently, think differently, and may have different habits of seeing and ultimately believing.”

Bennani is also skeptical of claims that AR or VR should encourage or foster any kind of emotion: “We are so desperate for a more empathetic alternative to today’s human interactions, but I am not sure how AR can be more efficient than good film. I find it absurd to expect for any emotional reparation to take place within it. The acronyms ‘AR’ and ‘VR’ have become synonymous with living in the future.” The technological hype and excitement is quite removed “from feelings of empathy and social activism,” she adds, though AR can be useful for education, “a first step toward empathy.”

Augmentation demands attention, a sharpened quality of seeing, and an active balance between multiple competing modes of vision. The power of augmented reality is to reveal how our perceptive habits are formed by our values. AR’s play and games point obliquely at our assumptions and prejudices, gently directing the viewer along multiple, often competing modes of interpretation. The effect is a moral challenge posed through aesthetics.


Tuomas A. Laitinen, 1, 2016, installation view at Bucharest Biennale 7, Bucharest, 2016. © Tuomas A. Laitinen. Courtesy: the artist and Helsinki Contemporary, Helsinki. Photo: Tuomas A. Laitinen


Finnish artist Tuomas A. Laitinen choreographs just such a progressive sharpening of attention through pr0.bes (2016), an AR series developed at first as a billboard for the Bucharest Biennale. Playing with scale, space, and time, Laitinen uses AR to explore a political narrative, namely how scanning technologies and capitalism are layered one atop another. AR formed an entry point into unseen narratives and images, the “pictorial ambience unseen without technical tools, hidden behind the surface appearance of (any)body or thing,” he told me.

Laitinen created a meta-scanner that changes one’s mobile device not into a game, but into a “mediator for the story,” in the case of the billboard, the scanning of trucks for evidence of human trafficking in crossings at borders. He notes that AR opened up the story to be more experimental, inhabiting different modalities, itself a kind of biological system, a “stacked narrative that has multiple layers that are porous and seeping through one another.”

The artist is excited for AR’s potential as a guerilla tactic in museum installations. There is, he says, a “beautiful underlying subversive potential here: imagine a situation where one would go to a history museum and there is an alternative critical history of colonial objects made through an unofficial AR application.” AR could have deep promise as a critical intervention in a world of surfaces, guided and managed by corporations. He imagines one might use it to scan the embedded structures, the flows of money and influence supporting a powerful artistic institution to gain an intimate familiarity with its social and financial infrastructure.

Where else are people willing to be led by augmentation? Pokémon Go is one clear suggestion that people entranced by an AR world will follow its augmentation anywhere, into temples and schools and parts of their neighbourhood they’ve never been. Ian Cheng had to think quite a bit about such augmented leading in his Emissary Forks For You, a Google Tango tablet project first installed at Migros Museum, Zurich, and further developed for the Liverpool Biennale.

Cheng is an artist and a thinker to whom many scholars look for guidance and articulation of the aesthetic and philosophical developments offered by emerging technologies, whether through his apps (such as Bad Corgi (2016) for the Serpentine in London) or live simulations, as in EMISSARIES (2017), his massive show at the Museum of Modern Art, New York, in which four years of his live simulations unfolded continuously to create a universe. Viewers could walk into any “time” to watch his stunning artificial children interact, breed, and socialize across an uncanny primordial landscape. Shiba Emissary, a recurring character in all of Cheng’s works, leads us through each stage of this world.


Ian Cheng, Emissary Forks For You, 2016, Liverpool Biennial 2016, Cains Brewery, Liverpool, 2016. Courtesy: the artist; Liverpool Biennial, Liverpool; Pilar Corrias Gallery, London; STANDARD (OSLO), Oslo. Photo: Mark McNulty


In Emissary Forks For You, the user follows a digital Shiba Inu via a tablet he or she carries throughout the museum space. At the Biennale, the user was intended to crash and tumble into the viewing space before and around other artworks without regard for the viewer. The other artworks were obstacles in the space to be navigated around. Cheng’s augmented world offers a cheeky intervention, an interruption of the high status of artworks, rendering them mere objects elevated through the logics of a self-contained world, with its prescribed rules establishing value.

This AR experience is opportunity to “see the world as layered, that at any given moment we are living in one story among many layered stories,” Cheng tells me. He compares it to a plot in China Miéville’s 2009 novel The City and the City, in which “two cities are geolocated in the exact same place, but citizens of each are blind to the other. AR makes any given reality less sacred, but in doing so, it doesn’t abolish meaning. It is just a forcing function to see multiple meanings, and for you to be in the position to make a choice about which meaning you want to live with, and shape you. Perhaps it lets you even see what different contexts you actually share.”

I asked him how his ontology of artificial beings has changed through his use of augmented reality, and what new entry points into the generative narrative of Emissary Forks For You AR allows for. For one, he says, in sending his Shiba Emissary out on this new augmented path, the viewer could enter the simulation space. And further, “it was a way to see if a character and its basic AI model could survive in a different context, outside, and have a physical relationship to us people.” The Shiba trains us to have a relationship with it, understand it as an artificial piece of art that is “so alive it wants to learn and play with us.”

We continue to follow the dog because of its layering atop our own, and arguably, because of this layering, we can begin to understand it as vibrant, “opinionated, and extreme in personality, an AI you can actually have an interesting relationship to, no matter how stupid or smart.” We start to sense that the Shiba’s context, ontology, and perspective might be just as complex and diverse and rich as our own. In contrast to the dominant model of AI, the “neutral concierge AI who is devoid of overt beliefs” and offers us no meaningful relationship, Cheng intends his sentient virtual dog, leading the user through AR space, to “both lower our expectation for AI to be human-like, but not strip it of its wonder.”


Ian Cheng: EMISSARIES installation view at MoMA PS1, New York, 2017. Courtesy: the artist and MoMA PS1, New York. Photo: Pablo Enriquez


I found the AR dog primes me mentally, in accretive steps, to have a different understanding of all the clattering artificial objects that fill the space around it. In trusting and believing in the Shiba, and depending on its guidance through the augmented space, I become more intimate with the noises and feeling of digital life forms as they move (as I move along with them). I begin to respect their capacity for beauty, and surprise.

Hearing Cheng and Laitinen speak about augmentation as a way to truly break up a museum space, I sought out Jocelyn Miller, assistant curator at MoMA, who has worked in depth with Bennani, Cheng, and other digital artists over the years. This past year, she curated Martine Syms’s lauded show Projects 106, which featured the AR app WYD RN? (What You Doing Right Now?), developed with Brent Freaney at Special-Offer in Los Angeles. WYD RN? offers the most serious entry that I have yet encountered in augmented reality as a critical framework. It seems a natural next step in deepening Syms’s practice, which has for many years thoughtfully explored issues of individuation and race and gender in a highly mediated digital age.

Miller proudly described how WYD RN? developed, and called Syms a rarity for having deployed an augmented reality “to very deliberate, intelligent, productive ends,” not as a “frothy gimmick but as connective tissue, [but] coming directly out of the media she has worked with throughout all of her practice, that helps viewers most directly inhabit her work and subject matter.” Syms sought a way for her first feature film, Incense Sweaters & Ice (2017), to “live in space.” AR was a means to have every element of the show continue to layer atop and deepen “the film’s narrative, production, and characters, as a mechanism for viewers to think more deeply about how they themselves experience their own individuation as (sometimes shifting) roles they assume and perform in public, on screens, and in feeds.” Viewers will also, Miller continues, consider “how this may or may not be distinct from who they are under conditions of anonymity, or obscurity, in private.” Early on, Miller describes, Syms conceived an app with augmented reality technology to help viewers interact with the suite of photos Syms printed onto found vintage Black film posters.

For Syms, the digital zeitgeist ever frames both the personal and the historical. The WYD RN? app is triggered by the photographic images in Projects 106 using facial recognition technology. Each sync triggers a rapid layering, atop one image, of family photos and personal archives, which the viewer feels lucky to have a privileged intimacy with. We are treated to a remixed rendition of “For Sentimental Reasons” by Nat King Cole. Other images contain a flow of the best one-liner reaction memes from Real Housewives of Atlanta’s shade masters NeNe and Phaedra, images of Tyra Banks chastising models on Top Model, and hilarious group text conversations about going out for karaoke, the need for some D, and whether to text one’s ex back.


Martine Syms, WYD RN?, 2017, augmented reality iOS app made in collaboration with Brent David Freaney at Special—Offer, Los Angeles. © Martine Syms. Courtesy: the artist and Bridget Donahue, New York


As you walk through the gallery space to watch Incense Sweaters & Ice, you are being seen by others watching your phone; your intimacy with your screen becomes itself a subject. “The app was the perfect way to close and complicate this loop,” Miller elaborates. But “it was important both for practical and also conceptual reasons that the show be able to exist independently from the AR experience,” which is “one of many recursive threads tying together the characters of the film, their stories, and the way their subjectivity is constructed, alongside the viewer’s subjectivity, in space.”

AR feels perfect for a millennial visitor to a museum in 2017: she is moving, as Miller describes, in a more choppy way from rooms to screens, performance installations, and sculpture, all while looking at other bodies in the gallery space, in a way that mirrors how she moves through digital life. “The tension that arises from oscillating between those spaces in a public setting is one that is extremely lifelike,” the curator explains. AR apps work to “simulate the experience of contemporary media literacy and behavior, which is completely nonlinear, and all about a constant negotiation between spaces, places, and screens,” she adds.

The use of AR to flow through, negotiate, and recontextualize space might be one of its most compelling uses today, a time when social and political divisions seem insurmountable. Sheri notes that the future design of all mixed reality, augmented reality, and virtual reality platforms will have to ask serious questions: “Are spaces and objects owned? Who has the right to see what? Does everyone see the same object? How are ‘objects’ perceived? By whom? As we alter our senses and perception, we are altering our relationship to objects, communication, and meaning of trust.” Designers have a responsibility to “consider consequences of the underlying systems” they make.

Cheng has pointed out that AR experiences can help delegitimize hateful spaces, like the Westboro Baptist Church. Others argue that this equalizing of objects allowed briefly in the augmented reality landscape is ethically problematic, as when eager youths playing Pokémon Go wander into parts of town (read: “the other side of the tracks”) they have no familiarity with, a terrible exercise in privilege. Cue hand-wringing about the tools’ flattening out of context of environments, neighbourhoods, and groups in cities that are already unseen, a continuance of a wider extant political and economic un-seeing by the supposedly neutral eye of technology.

But like Bennani, Syms, and Laitinen, Cheng insists that AR is simply a tool; it is, as ever, people who make ethical or unethical decisions. “I wouldn’t give up Shakespeare and science fiction just because the technology of writing also allows for hate speech and unfair laws to be written,” he jokes. Seeing that there is “much lower existential ambiguity from developing AR than, say, genetic modification, going to Mars, or AI,” and as AR development becomes easier and easier, he imagines that there will be a dizzying variety in AR uses, in as-yet unimaginable and nested arrangements. He offers artists a beautiful manifesto to augmentation as a medium that can only generate more creativity and expressive possibility:

“Like movies, novels, music, there will be 90% junk, some of which, as you say, will be made with the incentive to amplify blindness… But that 90% junk space is a sign of a health, because it means the 10% that is good is going to be mind-blowingly good. Our taste and barometers for what is good or interesting or useful AR will have a space in culture to develop very high. True inequity comes when there is a limit of options and a limit on the ease of production and experience. If it is hard to play with, it is probably not expressive; it is probably not very freeing.”


Nora Khan is a writer of fiction and criticism focusing on digital art and philosophy of emerging technology. She is an Eyebeam Research Resident and acting editor at Rhizome. Her criticism won a Thoma Foundation Arts Writing Award in 2016, and has been published or is forthcoming in 4Columns, Art in America, Conjunctions, Flash ArtGlass BeadAmerican Literary Review, Spike Art Quarterly, California Sunday, The Village Voice, Rhizome, POSTmatter, and After Us. She collaborates frequently with artists internationally, with essays commissioned for Katja Novitskova and Yuri Pattison by Chisenhale Gallery, Sternberg and Mousse. This past summer, Primary Information published Fear Indexing the X-Files, an essay-book Nora wrote with Steven Warwick. She has given talks at Triple Canopy, Gray Area Festival, transmediale, Whitney Museum of American Art, UCLA, New Museum, NYU, and New School.


Originally published on Mousse 60


Related Articles
Mousse 74
Attack of the Killer Grannies: Pauline Curnier Jardin
(Read more)
Blue Monday Forever: Alan Michael
(Read more)
Book Reviews by McKenzie Wark
(Read more)
Mousse 74
Beaded Affairs: Igshaan Adams
(Read more)
Mousse 74
The Swamp of Masculinity: Florian Krewer
(Read more)
A Sculpture Looking at You Whilst Touching Itself: Jesse Wine
(Read more)