Result

Moving Camera view

Moving Camera view

Fixed Camera view

We can even do more! 

Any video + any view you like!

New instrument!

Demo

Comparison with existing method

Ours Scream Lab, NCKU
Camera View Any view you wish  Any view you wish 
Figure From real people 👍🏿 Generated from 3D artists using tools like Unity  
Figure motion Limited by SMPL parameters more delicate 👍🏿
Time In an hour 👍🏿 Much more than a hour

faq

Questions asked by professors during project presentation

Why not directly capture a human playing an instrument? It seems unreasonable to me to separate the human performer from the instrument unless the application has a specific purpose.

We want to capture only the human’s appearance and make it into an animatable 3d model so that

○  We can animate the model to play any piece of music and any instruments

○  The person being captured doesn’t have to know how to play instruments

I am curious about the timing performance of the proposed system. How long does it take to create a new scene from scratch?  

Train Avatar(<20 min) + Generate Video(~20 min)(~0.9 sec per frame)