New ReBlink exhibit at Toronto’s AGO takes augmented reality beyond video games

When most people think of technologies like virtual reality and augmented reality they tend to think of video games, but there are plenty of other ways to experience these modern mediums ranging from QR codes scanned on your phone to art installations in galleries. An example of the latter is on display now through December 3 at the Art Gallery of Ontario.

Entitled ReBlink, the exhibition is the work of Toronto-based digital artist Alex Mayhew, along with a team of OCADU students and assistance from his business partner Ian Kelso. Mayhew has digitally reinterpreted – or, as he prefers to say, re-mixed – several key pieces within the gallery’s collection of Canadian and European paintings so that they can be viewed augmented reality.

All visitors need do is download an app on their phone or tablet, then hold up the screen in front of specific painting to watch them magically break free of their two-dimensional canvases, often in surprising ways. They’ll see the paintings’ classic subjects in a fresh and modern light, perhaps with an arm protruding from the frame holding a camera and taking pictures of gallery patrons, or standing like a colossus among the planets and satellites of our solar system.

Post Arcade had an opportunity to briefly chat with Mayhew and his partner Kelso via email on the eve of the exhibition’s debut. They discussed the motivation for the work, what visitors will experience, and the growing accessibility of tools and technology that make this sort of digital art possible.

Impossible Things

" data-medium-file="http://wpmedia.business.financialpost.com/2017/07/screen-shot-2017-06-19-at-14-12-33.png?w=300" data-large-file="http://wpmedia.business.financialpost.com/2017/07/screen-shot-2017-06-19-at-14-12-33.png?w=640" class="size-large wp-image-1402037" src="http://wpmedia.business.financialpost.com/2017/07/screen-shot-2017-06-19-at-14-12-33.png?w=640&h=436" alt="" width="640" height="436" srcset="http://wpmedia.business.financialpost.com/2017/07/screen-shot-2017-06-19-at-14-12-33.png?w=640&h=436 640w, http://wpmedia.business.financialpost.com/2017/07/screen-shot-2017-06-19-at-14-12-33.png?w=1280&h=872 1280w, http://wpmedia.business.financialpost.com/2017/07/screen-shot-2017-06-19-at-14-12-33.png?w=150&h=102 150w, http://wpmedia.business.financialpost.com/2017/07/screen-shot-2017-06-19-at-14-12-33.png?w=300&h=204 300w, http://wpmedia.business.financialpost.com/2017/07/screen-shot-2017-06-19-at-14-12-33.png?w=768&h=523 768w, http://wpmedia.business.financialpost.com/2017/07/screen-shot-2017-06-19-at-14-12-33.png?w=1024&h=698 1024w" sizes="(max-width: 640px) 100vw, 640px">

George Agnew Reid’s painting “Drawing Lots” is one of several classics that have received augmented reality treatment at the AGO under Mayhew’s guidance.

Post Arcade: What led to the creation of this exhibit? What ideas are being explored?

Alex Mayhew: The painting above (George Agnew Reid’s Drawing Lots) was the main inspiration for this show. Being burnt out after completing a very big digital entertainment project, I took solace at the AGO just looking at the old masters: a kind of calming antidote to the chaotic experience of contemporary city life. The George Reid painting is a place free from computers, where people simply play and talk with each other. I found a deep connection with the painting through a reflection of the present from that image of the past. Yet I saw so many people — particularly younger visitors — just glancing at the work for mere seconds and then walking away. I later learned that the average time someone actually looks at a painting in a gallery — and this includes the masterpieces, like the Mona Lisa — is just sixteen seconds. 

I had dabbled with early augmented reality before when I made the Time Tremors AR Treasure Hunt for the AGO in 2013. But so much had changed for the better and I wanted to push things beyond the surface that I felt was only being scratched up to now.  

That is where the inspiration for ReBlink came from.  

I became so enamoured with the idea for ReBlink that I decided to build a working prototype completely on spec – without the AGO even knowing I was doing it. I then showed it to Keri Ryan, the Associate Director of Interpretation and Visitor Research at the AGO, and luckily she saw the potential and pitched it to all the key decision makers. The AGO is a wonderfully progressive institution, but this was something very different even for them. I believe the key motivator for the AGO is the ability to get visitors, particularly younger audiences that consume media and images so voraciously, to pause, slow down, and really engage with the work.

It is all about the juxtapositions of realities. By looking at the past with a present day lens you can provide a link to the past. And in turn the past can then provide us with an understanding of the present. Augmented reality, and especially mobile AR, is the perfect platform to trigger this kind dialogue.   

I then teamed up with my business partner Ian Kelso and we enlisted star students and graduates from OCADU. Technical Director and Artist Hector Centeno and lead production artist  Saffron Bolduc-Chiong were absolutely key, as well as talented 3D modellers David McKenna and Yifat Shaik.  

The show’s overarching theme is “past versus present.” Or, more aptly, “the past through a modern day lens.”  Perhaps this is particularly apt because of Canada 150 as it presents an opportunity to reflect. The specific themes that I felt could help make that contrast include images of self, the effects of technology on our lives, the loss of innocence through media, scientific advancement, pollution and our consumption of food.

A visitor takes in the augmented reality version of a painting at the AGO’s new ReBlink exhibit.

Can you describe what attendees will see?

Ian Kelso: I love to just stand and watch people’s eyes as they take in ReBlink experiences for the first time. There is an absolutely real sense of magic to seeing each painting come to life. We call these particular AR experiences “re-mixes.” Just like with music, the augmented work is strongly rooted in the original, but is “modernized” in a deliberate way to give visitors a new layer of context, one that they can perhaps better identify with. 

Some of the experiences also play with a sense of presence. For example, in the re-mix of Cornelius Krieghoff’s 1850 Village Scene in Winter we are transported from an idyllic and peaceful landscape of a tiny Quebec settlement to a dystopian environmental disaster area where a masked guard in a hazmat suit stands sentry in front of the painting and leers at you wherever you go in the gallery. It’s kind of scary, yet kind of awesome at the same time! 

The AGO will have iPads in the main gallery where five of the works are on display. To experience the others, some of which are my personal favourites, visitors will need to bring their own smartphone or tablet. The ReBlink app is available for download for free on both iOS and Android. Because the paintings are so huge, and because the images are so high resolution, the download size is quite large. With most fast home connections though it should only take minutes. Speeds on the free public WiFi at the AGO can be fast but they can also be variable depending on traffic. 

Why were these specific paintings chosen?

Mayhew: Keri Ryan and I explored the AGO collection and decided together. Decisions were based on which paintings had the most ‘interactive’ potential and where there was a particular contrast with present day life. 

Toronto-based digital artist Alex Mayhew and business partner Ian Kelso at the AGO.

What do you think the original painters would think of augment reality? 

Mayhew: We have no way of telling what the original artist would feel. The most contemporary work in the series is The Marchesa Casati by Augustus John which was painted almost 100 years ago. Certainly the Marchesa Casati herself would have loved the attention. She is famously quoted as having always wanted to be “a living work of art.”‘ When you see the augmented reality re-mix, you will see we have kind of granted her that wish!   

Some people may take offence to the way we have “altered” and played with these masterworks. We can say that every piece we created came from a place of love. We highly respect the themes and aesthetics of the original pieces and we crafted our interventions with meticulous attention to detail whilst trying to stay true to the spirit of the original. 

Kelso: I think it is safe to day that none of the artists of the original paintings in our exhibition would have had even an inkling of the kind of the technologies that we take for granted today. But I’d like to think that any creative person would be very excited with the new and powerful forms of expression and communication that AR and VR are capable of. I started my digital career at the dawn of the commercial web in the early 1990s, and I was truly thrilled to just go to work each day where something new always seemed to happen. I spent several years as an Internet evangelist telling everyone who would listen that “a new age is coming!” Over twenty years have passed, and I suddenly have this feeling once again. 

Do artists working in AR, VR, and MR need to be highly adept with technology, or are there accessible tools that allow less savvy creators to explore these new mediums?

Mayhew: There are accessible and easy-to-use tools. Anyone can create AR within an hour using tools like Blippbuilder or Aurasma. However, for more complex projects like ReBlink, technical knowledge is absolutely necessary. So the answer is to team up with a very clever and talented developer or learn how to code.  

Kelso:  As with the introduction of any new digital medium, the basics will become more and more accessible. Anyone can now build a video game using tools like Scratch. But you won’t likely use those tools to suddenly create the next Zelda or Minecraft or even Pac-Man overnight. And for the next few years I think we are going to see continual transformation and disruption. There will be many new hardware standards, constantly improving but competing mobile frameworks, more powerful engines and lots of AI integration. All I can say is that the digital artists and developers we have been working with coming out of OCADU, are amazingly able and prepared to take on the challenge of integrating all of these variables to create something brilliant. 

Ubisoft Toronto’s big bet: A new spin on toy-based video games with Starlink

My daughter and I sat on a white leather couch in a small space within Ubisoft Toronto Inc.’s office made up to feel like your average living room. I watched, fascinated, as she tinkered with a collection of hand-painted prototype plastic starships used by designers to test Ubisoft Toronto Inc.’s Starlink: Battle for Atlas, a game that puts players in control of well-armed, highly maneuverable, and completely customizable spacecraft.

As she altered her ship – which was cleverly mounted atop her controller for easy access – its onscreen doppelgänger transformed to match in real time. If she pulled off a wing or a gun, it would likewise disappear from the ship shown on the TV. When she popped them back on – sometimes adding wings on top of wings, or making guns face different directions – they would magically appear on the virtual ship exactly as she positioned them, occasionally causing it to briefly loll left or right until the craft’s automated systems restored balance.

My kid and I have played plenty of toys-to-life games – video games with a physical toy component – but neither of us had ever seen anything quite like this; toys designed for extensive physical modularity replicated instantly within the virtual world.

Add in a plot-driven space opera story set on a series of huge, open, and completely circumnavigable alien planets designed with a level of sophistication more common in adult-oriented games – potentially stretching the game’s target audience beyond the category’s traditionally youthful demographic to include older players – and StarLink feels like it could be something special and unique when it eventually launches in the latter part of next year.

A screenshot from StarLink: Battle for Atlas, Ubisoft Toronto’s first original IP.

“We were given an amazingly wide open mandate from HQ,” said the game’s producer, Matt Rose, referring to Ubisoft Toronto’s parent company Ubisoft Entertainment SA, based in Montreuil-sous-Bois, within a suburb of Paris in France. “There’s a fundamental core belief within Ubisoft that comes right from [president, CEO, and co-founder] Yves Guillemot that has to do with entrepreneurship. They empower teams to take bold risks.”

Indeed, StarLink is risky for many reasons. Not only is it a brand new IP in an industry that thrives on sequels and known licenses, it’s also a game that will help to define Ubisoft Toronto.

Founded in 2010, Ubisoft Toronto is located in a large, old brick building in the city’s quietly bustling Junction Triangle neighbourhood. It’s the result of a partnership with the government of Ontario. It was the first major game development studio in the city, and the province bet big that it would help kickstart Toronto’s then-fledgling game industry the same way Ubisoft’s massive Montreal studio assisted in turning that city into a world class hub of game development. The province offered up to $263 million over 10 years to help the new studio grow to 800 employees. Seven years later, Ubisoft Toronto is home to some 600 game industry professionals. More than that, its people are helping steward the local industry, regularly offering talent, resources, and leadership for local conferences, workshops, and industry organizations.

The government’s plan seems to be working. The game industry in Ontario is now booming. As of 2015, the last time the Entertainment Software Association of Canada issued a comprehensive report on the industry, there were 100-plus game studios employing more than 2,500 people with an average salary of over $70,000. 

However, in the seven years since Ubisoft Toronto first opened shop, it has yet to produce something it can call its own. It’s supported the development of half a dozen major games headed up by Ubisoft Montreal, including several entries in the popular Far Cry series, but the only game on which Ubisoft Toronto has acted as lead is 2013’s critically successful but financially disappointing Tom Clancy’s Splinter Cell Blacklist, a sequel within a long running franchise.

StarLink: Battle for Atlas, however, is Ubisoft Toronto’s baby through and through.

The studio has dedicated nearly 100 staff to its production. It’s eschewing Ubisoft’s famously efficient distributed development process – which sees discrete tasks disseminated to sister studios around the globe – to develop StarLink almost entirely in-house. It’s even created its own hardware team to design proprietary connection and communication technology to enable the toys’ innovative modularity, beginning with existing staff members who happened to have a background in industrial design or dabbled in chemical etching on circuit boards.

“Once we knew there was something special to our idea and wanted to pursue it we built a dedicated hardware team right here,” said Rose. “We had all of these people come out of the woodwork from within the studio, and we hired experts from many of the biggest toy companies in the world and built a world class team with dozens of years of experience.”

Establishing a hardware team changed the way the studio’s seasoned software engineers thought about game design. “The meeting between the toy company experts and people within our own team and our concept artists has been incredible,” said Laurent Malville, the game’s creative director. “You have to rethink the way you work. The concept artist might create a ship with seven colours instead of five, and then the industrial designer explains that each time you add a colour it increases the cost. So art and cost and quality are all intertwined. And it was extremely important for us to be the one driving things and maintaining the quality of the process.”

One of the modular toy ships designed for StarLink: Battle for Atlas.

Of course, the upshot of retaining complete control over the development effort means that Ubisoft Toronto is fully on the hook for how the game eventually turns out.

Some analysts aren’t bullish on its chances, perhaps because the toys-to-life category appears to have cooled off a little over the past couple of years. Disney Interactive Studios Inc. mothballed its ambitious Disney Infinity franchise in 2016, and this year will be the first in half a decade that Activision isn’t launching a new game within its Skylanders line, the series credited for popularizing the toys-to-life segment.

“The category is yesterday’s news,” said Michael Pachter, a managing director of equity research at Los Angeles-based Wedbush Securities. “I don’t think that Ubisoft expects dramatic success.”

IDC gaming research director Lewis Ward isn’t as quick to dismiss new toys-to-life games. “The general outside view is that the segment’s in trouble, but that’s a little simplistic,” he said, adding that Disney may have exited simply because it wanted to go back to its traditional business model, which is licensing its properties to publishers rather than developing games itself.

“Bringing something unique to toys-to-life has potential value,” said Ward. “It’s a pretty big bet for Ubisoft on original IP, and especially the development team based in Toronto. When I think of Ubisoft I don’t normally think of kids games. So in that sense it’s a new muscle they’re trying to work up, and that’s ambitious.”

In the end, the key to StarLink‘s success – as is often the case with new video game properties – may simply end up being whether it’s fun to play. And whether it can find a big enough audience. Ubisoft Toronto’s managing director, Alexandre Parizeau, is confident that his team has hit on something different, something that will not only shake up the toys-to-life segment, but grow it to include older gamers.

“The feedback from players and retailers is positive,” said Parizeau. “Innovation has the potential to inject new life into the toys-to-life category. Toys-to-life introduced many new players to video games and these players have now evolved, and so have their expectations. [Our game] combines a modular starship breakthrough with a dynamic living star system that completely changes the way you play.”

He might be right.

My own 12-year-old daughter has grown up playing toys-to-life games. She still loves them, but as she gets older she’s looking for more from her interactive entertainment. And as I watched her playing the pre-alpha version of StarLink: Battle for Atlas, it seemed like it might be satisfying her new itch for more sophisticated games. She responded to the game’s emotive human characters and their problems, her imagination was captured by the vast and exotic alien landscapes, and her hands eagerly manipulated the starships as she experimented with their seemingly endless configurations.

During the car ride on the way home from our visit to Ubisoft Toronto I noticed her gazing absently out the window. I asked what was on her mind.

“Why are they waiting to launch it for over a year?” she asked. “I’d pay to play that game and buy those ships right now.”

A screenshot from StarLink: Battle for Atlas, Ubisoft Toronto’s first original IP.

Digital sales to account for nearly 93% of all video game sales by 2021: report

The global digital game market is set to grow from €48.1 billion (CAD$71 billion) in 2017 to €73.1 (CAD$108 billion) in 2021, according to new projections issued by European digital industry observer IDATE DigiWorld Research. What’s more, digital sales will account for nearly 93 per cent of all game sales within four years.

The Montpellier, France-based firm’s newly released data forecasts the quickest growth of digital revenue in the home console market, climbing from €6.2 billion this year to €12.2 billion in 2021. This shouldn’t be surprising, given Sony Interactive Entertainment Inc., Microsoft Corp., and Nintendo Co. Ltd.’s collective push to make more content – both full games and add-ons – available within their respective digital stores.

According to the researchers, the growth in digital sales should prove sufficient to offset declining sales of physical media, including game discs and game cards, in the home console market. “The home console market is still growing inside life cycles of six or seven years,” Laurent Michaud, head of IDATE DigiWorld’s video game practice, told Post Arcade via email.

Michaud also confirmed that his projections, which forecast a gradually declining annual growth rate across all categories of digital game sales, will spike back up in 2021, which is when he believes the next generation of home consoles will launch.

But consoles make up only a fraction of the market. According to IDATE DigiWorld’s data, the biggest streams for digital game sales will continue to be computers and smartphones, though their growth won’t be quite as pronounced over the next four years. The former will increase from €17.3 billion in 2017 to €19.9 billion in 2021. The latter will see stronger progression, moving from €16.5 billion this year to eclipse computer online game revenue as early as 2018 before hitting €22.4 billion in 2021.

Growth in tablet game revenue, meanwhile, was calculated separately from phones, and is set to increase by about a billion euros each year from 2017 to 2021, eventually reaching €10.4 billion in 2021.

Also measured discretely from other forms of digital game sales is revenue for “artificial reality games,” which combines a range of technologies, including virtual reality, augmented reality, and mixed reality. This area is expected to see the steepest growth of any category, starting at just €0.1 billion this year but hitting €5.8 billion by 2021.

Handheld console digital revenue will see more sluggish growth, moving from just €1 billion this year to €1.7 billion in 2021.

IDATE DigiWorld’s research highlights how the growth of connectivity in game devices has impacted physical media. With virtually all game playing devices now connected to the web, players can purchase and download almost any game or game related content they want instantly rather than making a trip to a game shop.

“The Internet allows links between games and devices, whatever the segments,” said Michaud. “[Digital access to] games will soon be completely ubiquitous.”