While It’s Compiling is a continuing series of interviews with experts across a range of bleeding-edge technologies and practices, exclusive to Skills Matter. Be sure to subscribe to this blog for future interviews, or follow us on Twitter.
Find out who we’ll be interviewing next, and get a chance to put your questions forward with the hashtag #whileitscompiling.
We had a fantastic start to the year when we hosted the first ever iOSCon here at our headquarters in London, bringing together some of the world’s leading iOS experts including Boisy Pitre, Affectiva’s Mobile Visionary and lead iOS developer.
Boisy’s work has led to the creation of the first mobile SDK for delivering emotions to mobile devices for the leading emotion technology company and spin-off of the MIT Media Lab. We were delighted to get the opportunity to interview Boisy while he was here.
You can find the link to his talk from iOSCon at the bottom of this post, and all the talks here.
Hi Boisy, thanks for joining us for this year’s iOSCon. Can you tell us a little about yourself and the work you’ve been doing with Affectiva?
Sure. Currently I’m with Affectiva, an MIT media lab start-up based in Boston. We have an interesting technology which analyzes people’s facial expressions to determine their emotional state. The technology was developed, based on the research that one of the co-founders, Rana El Kaliouby, had pioneered in the affective computing field. The applicability of that technology was originally targeted towards the market research industry to help measure consumers’ emotional connections to brands and media.
About a year ago, Affectiva decided to expand their technology to mobile devices and tap into other industries beyond their current market – including gaming, healthcare, education and others. So, I came on board to lead this mobile initiative; and worked with some brilliant engineers to shrink the existing technology, which had a significant server component, down to the iOS platform. The Affdex Mobile SDK is the outcome of that effort. It does all the processing and reporting of emotional data on a frame-by-frame basis back to the app, right on the device – eliminating the need to connect to a server.
So it’s built on a lot of research – was iOS the natural progression and the natural platform to go to? What sets it apart from other platforms?
iOS was the initially targeted platform. In hindsight, I believe this was the right choice, as targeting iOS devices has been a bit easier due to the commonality of hardware and software; and it allowed us to get the SDK to market pretty quickly. Although we initially focused on iOS, I knew we were going to eventually develop an Android piece as well; which we’re almost done with, in fact. For the Android SDK, we hired team members who love and play in that sandbox. My philosophy is that for a company to be successful in a mobile strategy they should have experts that specialize in a particular platform.
In terms of applications going beyond the obvious marketing and advertising aspects, what are the real-world applications that exist now? Is there anything particularly interesting or exciting that Affectiva is working on right now?
As far as I’m concerned, it’s the wild west for apps that want to take advantage of emotion technology. It reminds me of the introduction of the iPhone in 2007. The idea of touching your device was not new, but Apple started democratizing it with the iPhone. That was the first really breakthrough moment in mobile. The second important breakthrough moment in mobile was the introduction of voice as input – again, Apple democratized how we interact with our phones and our devices when they offered Siri. I see emotional analysis having that same potential in mobile. Like voice, it give us a way of controlling the device and for the device to understand you better and offer you more choices.
So what type of apps can take advantage of this technology? Well obviously the low-hanging fruit would be games, where you’re interacting with the game – you want to have your emotions maybe control the game or have the game respond to your emotions in some way to adjust the level of intensity of play.
Health is another big opportunity where I think this technology can bring value. Take emotional health and well-being, for instance… there’s so much research pointing to the fact that our emotions have an impact on our health for better or for worse. So there’s a whole avenue of possibilities in that regard.
Then there’s the fun stuff. Imagine an app that analyzing your photos on your device to determine the emotional content to get an overall feel of your pictures. Or an app which changes music or colors on the screen while it watches your facial expressions. . Approaches like that can certainly lead to some interesting applications.
Of course, Affectiva is pursing app ideas at the moment based on this technology, but I cannot comment on them at this moment.
You mentioned that 2007 was the introduction of the first device, and how it’s moved-on, especially with Siri. Do you think that for someone such as myself, as a user of this device, are things going to continue coming out in stages, or is there anything around the corner that’s going to be as big and as ground-breaking as touch, or voice? Is there anything that’s going to jump out?
Emotion recognition technology has the potential to be that huge leap which brings in completely new way of interacting with our devices, whether we’re sensing emotions using the camera or through some other sense or mechanism. Having technology understand us better and gather deeper insights into our own emotions, through analysis at specific points in the day as we’re using apps, is a significant break-through in computer-human interaction.
And it’s a different level of interaction that liberates us. Just like touch liberated us from typing on tiny keyboards, and added a new paradigm of full natural touch with swiping. Emotional expressions in our face are instinctive; and they too can be a form of input and control, but they can also be a great form of feedback to us. I really think this will raise awareness of how we see ourselves in the world, as well as how we interact with others.
We hosted Droidcon last year, with devices such as the interactive mirror that could recognise your emotions in the morning. There is a huge interest currently in the Internet of Things, in connected devices and so on. Going beyond the iPhone or Android devices themselves, do you do much in terms of reaching out into connected devices?
Certainly. This technology can reach beyond just the device in your hand. For example, the automotive industry has expressed interest in our technology. That industry may be easier to break through on the Android side of things than it is with iOS, as iOS is a lot more compartmentalised and controlled by Apple. But certainly that’s one industry which could benefit from emotional analysis – just imagine driving along and your car wants to know if you’re falling asleep or paying attention or distracted; it’s looking at locations for safety, and again, health and well-being.
You touched on the fact that Apple and iOS is compartmentalised and controlled a lot more than Android. Do you think that’ s a drawback? Is this holding developers back on iOS or does it create an environment to focus ideas and energies?
Keep in mind that I’m coming from the Apple perspective as that’s the sandbox I play in. I completely understand and buy into Apple’s reasoning for why they do things. I’m also looking at this from a developer point of view.
We all know that Android exists on many, many mobile devices. It can be ported, unlike iOS, to phones, tablets, and other devices. The trade-off for such sheer ease of portability is the “fragmentation issue” which leads to complexity in development. At some point it becomes too massive for developers to support each of those devices. They must pick and choose their device support carefully.
I believe this is getting better as Android matures, but compared to Apple’s unified, streamlined hardware upgrading approach, it’s still a mess.
Apple’s approach, while certainly much more restrictive, brings a sense of order to the device chaos that permeates Android. If anything, I would argue that developers fare better in the Apple ecosystem because of these controls. But that is my opinion, of course.
Finally, in terms of Affectiva and how you work on a day-to-day basis – what’s the structure there? How does the team work?
We have two engineering teams, one dedicated to Android and the other to iOS. Each has an engineering lead. Both engineering teams interface with the science team, which concentrates specifically on the core technology of emotional classification. As science improvements are made, they are provided to engineering, which integrates the changes and improvements into the SDK code base. Agile is the foundation development methodology we use to organize and account for our work across the teams. This constant, connected cycle allows us to quickly iterate so that we can test, examine performance, etc.
Watch Boisy’s talk from iOSCon 2014
What if your iPad or iPhone could detect your emotional state and respond in a way that enhances your day? What if an app could deliver soothing content when you’re feeling upset, or play your favourite song when you’re feeling happy? Find out how you could achieve this in Boisy’s talk!
You can see the rest of the Skillscasts from iOSCon 2014 here.
While It’s Compiling is a series of interviews with experts across a range of bleeding-edge technologies and practices, exclusive to 
The Skills Matter Blog 