Kinect Augmented Reality – Part 1

At Digital Tactics we are keen to explore new technologies and enjoy tinkering with interesting ideas and pushing the boundaries. We have always been interested in the Kinect and I was assigned to take a look at using the Kinect to create our own version of the Augmented Reality Sandbox.

The Kinect contains 3 sensors sensors – an RGB color VGA video camera, a depth sensor, and a multi-array microphone.

For the Augmented Reality Sandbox we’re only interested in the depth sensor which is an infrared emitter and monochrome CMOS depth sensor that allows us to gather point cloud data of the sandbox. This works by measuring the distance of each point of invisible near-infrared light and measuring its “time of flight” after it reflects off the sand. The Kinect is effectively capturing 9 million data points per second over a 3 to 16 foot range.

The Kinect is a cheap and effective way to measure depth over relatively small distances and can be picked up for as little as £20 on eBay in 2016. Traditional scanning tools which use LIDAR (Light Detection And Ranging) which send laser pulses to accurately measure surfaces over many miles, cost between £15,000 and £150,000.

After a little research we realised the majority of the sandboxes found on YouTube are running the same software. As great as it is to set a sandbox up using an off the shelf software, we’re coders and we do like a challenge so we decided to do what hasn’t been attempted previously and create the sandbox visualisation in-browser using three.js. Whilst the sandbox itself has not been previously visualised using three.js, visualising the Kinects point cloud has.

First I needed to connect the Kinect into my PC, and for our Kinect, this requires a power adapter, which we picked up on Amazon for around £9. I currently use CentOS 7 as my main development operating system, so I chose a driver and library (libfreenect) that supports Linux. This also has the added benefit of possibly running the sandbox using a Raspberry Pi or similar.

Once installed, libfreenect has some binaries that enable you to check if the Kinect is working, shown in this screenshot

That’s it for part 1, please check out part 2 where we start getting the Kinects point cloud data into our browser!