Construction - Data Collection

A key aspect of the idea of a mirror world lies in its construction - the difference between a mirror world and any other virtual environment is that at least some part a mirror world is built from data collected from the real world, whereas other virtual environments can be entirely fabricated.

On this page, I will examine some of the methods of data collection used in the construction of mirror worlds; specifically, mirror worlds as they relate to augmented reality.

Di-Ocular 3D Scanning

3D scanning is a particularly straightforward means of construction - using a sensor consisting of two IR cameras and a projector that projects a non-repeating pattern on the space to be scanned, raw depth and color data is recorded for individual points in a scene. these points, collectively, form a "point cloud", which is already a pretty in-depth visual representation of a physical space:
semantic_map.jpg

At this point, algorithms can be applied to this point cloud to create a solid 3D model from the point cloud, to which physics simulations can be added, leaving a digital representation of a physical space.

This particular method of construction is shared by numerous 3d scanning solutions. Intel's realsense, Microsoft's Kinect, Google's Project Tango (which enables such cool augmented reality apps as Ikea Place) - all of these devices use this method.

Strengths

  • Allows for very high fidelity 3D scans, resulting in detailed mirror worlds
  • Easy to use, and there are very cheap options available (the xbox kinect can be had for less than $50)
  • The sensors are very versatile - they can be used for everything from motion capture to large scale scanning of areas

Weaknesses

  • Optical based methods of construction are often time consuming
  • computationally expensive
  • Can be overkill - more information than needed

Body Tracking

Sometimes, you do not need the level of fidelity afforded by bi-optical 3D scanning - instead, you might only need to know where in a room one can walk without bumping into something. In these scenarios, the processing overhead of bi-optical 3D scanners make them a less-than-ideal solution.

With body tracking, you construct your mirror world based on input from a person or some other being who is physically occupying this space.

The above video shows an example of body tracking based construction in the form of the HTC Vive. It's a Virtual Reality headset that allows the user to walk around a physical space in order to explore a virtual world based on the confines of the room; to walk around a mirror world.

There are two kinds of body-tracking based construction; outside in, and inside out. Outside in body tracking refers to a situation in which the sensors that are used to track a person as they move around a space are placed in the space itself (a-la Oculus Rift/psvr), while Inside out refers to a situation in which the sensors are on the person, without any need for additional accessories in the space (a-la windows mixed reality headsets/hololens). Each has different strengths and shortcomings;

Outside In

Strengths

  • Cheaper, and relatively easy to accomplish (by comparison) with existing technology;
  • user friendly
  • more accurate (as of today)
  • lower latency(as of today)
  • computation can be done on a powerful pc, rather than on devices worn the user's person

Weaknesses

  • Hard to scale - because the sensors must be placed around the space, both the size and the contents of the mirror world you can create using this method are very limited (too many objects in the space causes occlusion errors)

Inside Out

Strengths

  • Scalability - because the sensing and computation takes place entirely on devices on the user's person, you can use it in any sized space
  • As VR and AR technology progresses, the demand for mobility without sacrificing quality continues to rise - inside out tracking is likely to become more and more prevalent in the coming years

Weaknesses

  • Requires advanced computer vision software - not quite caught up to the outside in alternative
  • All computation must take place on devices worn on the user's person

The Vive is a crossover of sorts - the headset does the computation, like in a inside out situation, but it is necessary to have lighthouse accessories all over the space you're trying to digitize.

Sources

Benchoff, Brian. “3D Scanning Entire Rooms with a Kinect.” Hackaday, Hackaday, 10 Dec. 2015, hackaday.com/2015/12/10/3d-scanning-entire-rooms-with-a-kinect/.

Buckley, Sean. “This Is How Valve's Amazing Lighthouse Tracking Technology Works.” Gizmodo, Gizmodo.com, 19 May 2015,
gizmodo.com/this-is-how-valve-s-amazing-lighthouse-tracking-technol-1705356768.

Cui, Yan, and Didier Stricker. “3D shape scanning with a Kinect.” ACM SIGGRAPH 2011 Posters on - SIGGRAPH 11, 2011, doi:10.1145/2037715.2037780.

Langley, Hugh. “Inside-out v Outside-in: How VR tracking works, and how it's going to change.” Wareable, Wareable, 3 May 2017, www.wareable.com/vr/inside-out-vs-outside-in-vr-tracking-343.

Ricci, Alessandro, et al. “The Mirror World: Preparing for Mixed-Reality Living.” IEEE Pervasive Computing, vol. 14, no. 2, 2015, pp. 60–63., doi:10.1109/mprv.2015.44.

Back to Mirror Worlds

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License