Changing someone's gender or race on screen traditionally requires lengthy hours in front of a make-up mirror. But new software that can take a live video feed of a person talking and make them look and sound like somebody else could change that.
Psychologists rather than moviegoers are the first to see the benefits of the new technology: putting it to use in experiments that test how a person's gender affects the body language of others.
The software was developed by computer scientist Barry-John Theobald at the University of East Anglia in the UK and Iain Matthews, formerly at Carnegie Mellon University and now at Weta Digital in Wellington, New Zealand. They were approached by psychologists at three US universities searching for a way to switch the apparent gender of volunteers talking to each other through video conference software.
At that point not even Hollywood studios had access to such technology. For example, after Oliver Reed died during filming of the 2000 movie Gladiator the studio had to re-write his character's part and use existing footage of Reed to "act out" his character's demise . But even creating that 2-minute snippet took an estimated $3.2 million and five man-years to stitch footage over the face of another actor, frame by frame.
So, having worked on processing human faces in video for many years, Theobald and Matthews set about creating software to speed up the process.
They recorded video of volunteers performing 30 different facial expressions such as frowning, smiling and looking surprised. For each expression, the positions of key facial features, such as the eyes, nose and corners of the lips, were manually labelled.
That annotated footage was used to "train" software to recognise the face of each individual featured in the set. Once trained on a person in this way, it can closely track every move of their face in video footage.
Those movements can then be transferred onto the face of another "known" person by calculating how the recipient's features need to change to take on each new expression.
Doing that and displaying the transformed face takes just 150 milliseconds, fast enough to allow a conversation over video link to continue in real time. To complete the effect, a person's voice can be manipulated to match their new face.
Psychologists at the universities of Notre Dame, Indiana; Pittsburgh, Pennsylvania; and Virginia have already used the new software to test ideas about body language and gender.
Volunteers were asked to chat to one another in a video conference, but did not know if the face they saw was really that of the person they were talking with – or indeed if the other volunteer was seeing their own true face.
The results suggest that our body language during conversation is more reactive to that of others than it is to their physical appearance, says Theobald. "We've shown you can present a female as herself or as a male, and the other participant's behaviour doesn't change," he says. The results will soon be published in the Journal of Experimental Psychology.
Next, the team plans to use the system to test the effects of changing someone's race instead of their gender.
In the long term, Theobald also thinks film studios could benefit from the technique. "What we're doing is the same effect [as used in Gladiator], but in real time with no manual input." So far the software can't deal with complexities like variable lighting, he adds, but that ability can be programmed in.