AR Face Filters

GitHub Repo
Snapchat interview project

Blow A Kiss: AR experience with Unity MARS

Instruction

1. Open your mouth to blow a kiss! 2. Target the heart to spin it faster! 3. Wink to change the color of the heart!

Process

Idea

I had an initial idea to make an AR effect that users can interact with the environment object with the user’s interaction input. Most users hold a phone while playing AR apps. So I avoided touch input, using the most accessible interaction inputs, opening mouth, and closing eyes.

Once I decided on facial expressions for interaction, I could come up with the idea of kiss and wink. The idea moved to the ‘Face Blowing a Kiss 😘’ emoji and the look and feel of the AR effect. On top of that, it would go along with the romantic effects of Valentine’s day.

Structure

Unity provides complete access to components of the development, compared to Spark AR or Lens Studio. Even though Spark AR and Lens Studio provide better features focusing on AR effects development, they aim at patch/block-based development. That limits control over elements while programming. I found that Unity MARS can also provide essential features for AR effect making, such as face landmark tracking and facial expression events. So decided to work on Unity MARS for the project.

Trial & Errors

AR Face Mask

I made a reflective mask on the detected facial mesh. I intended to show the light change on the face according to the interaction input. It worked with existing project dependency and setting. However, the facial mesh edge was too distinctive compared to other commercial AR filters; you can see the border of the forehead and chin. I decided not to include this effect in this project.

Reflection Probe in AR

I tried to include reflection probes for a more realistic visual effect in the scene. However, the AR reflection probe only worked with the rear camera. I could not adopt it for the front user camera scenario.

Further Steps

New interaction input for world AR scenario.
I’d like to try body pose input, location of people, GPS, and landmark anchoring.

Assets Credit

Cartoon Kiss_cjohnstone.wav, trijohnstone, Freesound
Heart Symbol Low Poly Free low-poly 3D model, VARRRG, Cgtrader
3D model heart, honyzirbu, Turbosquid

---

Snapchat Lenses

Neon Party

Neon light moves according to face position. The lights on the depth mask works like face painting.

Whack-A-Mole

Whack-a-mole game using rear camera.

1. User can position moles before game starts
2. Whack a randomly jumping mole
3. Highscore tracked.

Role

Concept, Programming, Design

Credit

Mole by Eucalyp from the Noun Project (royalty paid)
Skate by Komiku
SFX UI Button Click by suntemple
Retro Bonus Pickup SFX by suntemple
Referee whistle blow, gymnasium.wav by SpliceSound