Analysis of Dynamic Gesture Recognition with Machine
Find Out More公共空間有很多與行人互動的裝置，藉由不同於以往使用鍵盤滑鼠而是簡單的手勢來進行遊戲，這種遊戲 進行的方式會讓使用者與觀看遊戲進行的人覺得很新鮮、有趣並為日常生活多些樂趣，我們的團隊就是想做出使用手勢操作的遊戲,並希望在進行遊戲時去製造出這些公共空間裝置所提供的樂趣。
利用Unity強大的物理引擎及物件操作的容易度高，來製作遊戲，在第一個遊戲中我們 使用Unity的Cube去組合出各種不同形狀的俄羅斯方塊，然後藉由三度空間座標來判斷方 塊的位置來安置方塊並進行是否已經放滿的確認，第二個遊戲則是使用了網路上完成3D模 組賽車來進行賽車遊戲的製作，使用Unity的力場推動金幣，並對道路進行移動使車子有移 動的效果。
Leap motion是利用紅外線裝置，其裝置支援利用手掌和手指動作來進行輸入，可以由 Library API獲取手部的相關資訊。在第一個俄羅斯方塊的遊戲，我們利用抓取手掌的位置 和手握產生的球半徑，辨識使用者手抓取方塊的動作，並利用手部的旋轉角度加上法向量 判斷旋轉手勢，另外也利用API內建的手揮動和手指點取辨識輔助遊戲的進行。而在第二 個賽車遊戲中，利用雙手手掌位置連線的角度實作方向盤的轉動。
利用API實作完旋轉手勢後，發現Leap motion在辨別是否是旋轉手勢及旋轉軸上不夠 精準，我們利用machine learning SVM的概念和Leap motion API抓取手部資訊，對旋轉 動態手勢的60個frame進行各種feature的分析，抓取60個frame之間的角度偏移量和法 向量當作feature，進行即時的曲線擬合，並實作手勢的優化。
First we decided to use Unity3D as our game engine. We dig into the Leap Motion API Unity documentation. Besides, we choose C# as the implementation language. We found that some part of the Unity document is deprecated or missing, so we check on the C# as supplement.
Next Step!Statistic data of Leap Motion sensor is retrieved in Unity C# script via API. Also, implementation of simple gestures such as grab, swipe and poke is done within the script as well. For the rotate gesture, parts such as filtering and preprocessing data is done. Additionally, components of game object is build in C# as well, mainly game scenes, shape of various bricks, and other user interface gadgets such as timer, menu, and workflow of the entire game.
Next Step!
In order to capture a rotation in real time during the game, the system continuously keep track of the data stream. We apply a queue-window approach, which keeps track of the most recent 60 frames and stores them in a queue. At first we submit the result to SVM upon each frame, but we faced severe performance issue and caused the game to freeze. We then decided to preprocess the API data in order to filter out patterns that is unlikely a rotation. When the preprocessing is done and a possible rotation pattern is detected, the system will submit the window queue to the SVM.
We use Matlab to implement machine learning and SVM. Since some porting issues on calling Matlab functions in Unity C#, we train data and build the model in Matlab first, then the Unity C# script retrieve the model matrix. Effort is made to trial and error in choosing features for machine learning. Features closely related to rotation such as rotation angle and palm normal are selected. Training and other hard works are done in Matlab. The Unity script is then left with relatively easy tasks, which yields good performance in real time gaming. Whenever the preprocessing system submits the queue, the SVM script will perform parabolic curve fitting and compute the scores cooperating with the model matrix produced by Matlab. Special effort is made to lighten system loading by sampling data every 4 frames. The 60 frame queue is then replaced by a 15 frame queue before submit. The database of training data is categorized into 7 sets, x/y/z for rotation axis in 3- dimensional space, and clockwise/counter-clockwise for each, and lastly the “not rotation” set. Each set we have more than 100 pattern samples, thus a total of more than 700 samples.
Next Step!
For Tetris, user is presented with a menu which one can “poke” on a button in order to select an option. During the game, user stretches his right hand to grab, rotate and push down a brick. If the floor is filled with bricks then the bricks of the bottom level can be cleaned. Also one can uses his left hand to swipe left or right and the angle of view is changed. In the Race Car game, user uses both hands to control the steering wheel to change driving direction. Collect coins to reach high score and dodge obstacles. Crash into them will lead to lost in health point. Swipe up your right hand to change to sniper mode. Push your hand forward to shoot obstacle to fire.
Demo!
Haiya
Christine
cheng Shaung
