Virtual reality not only consists of putting on an HMD unit, but it also makes use of a series of tracking technologies to read the movements of the player or at least the specialized control knobs in VR. That is why we have decided to list the different tracking mechanisms for VR that have existed and how each of them works.
Since the beginning of Virtual Reality, it has been designed for the immersion of the player in a virtual environment. Which means that the so-called telepresence has not only to be achieved from the audiovisual point of view. A 1: 1 correlation is also sought between the player’s movements and the environment. Which today is something that we have very, very green and is pure science fiction today. The day we have a Virtual Reality environment in which we can move freely without problems and interact is still a utopia
All this does not mean that there is no road map in order to arrive at this idea of freedom. Which is related to the development of a series of technologies that have been evolving in recent years. Many of them are already in the different virtual reality systems, but others we will see in the short, medium or long term.
That is why we have decided to make this article where we are going to list all the existing technologies that are used for tracking in Virtual Reality.
Tracking Inside-Out and Tracking Outside-In
There are two ways to categorize user tracking in VR, depending on where the tracking sensors are located in VR.
- If these are fixed somewhere in the room, but outside the HMD unit we call it Outside-In tracking.
- On the other hand, if the sensors are inside the virtual reality or HMD helmet, they are called Inside-Out sensors .
Both types of tracking systems are not exclusive and can be combined to obtain greater precision when following and interpreting the movements of the virtual reality user.
Cameras and computer vision
Two types of cameras are used today to track movements, the first of which are the classic RGB cameras that take an image in which each pixel represents a color from the RGB spectrum. In other words, a traditional camera. Although this is not the only type of camera used. Since they are complemented with time-of-flight cameras in some cases or infrared cameras for another in which each pixel of the image does not mark the color, but the distance in which each object in the image is located.
All this is combined today with what we call Computer Vision, which consists of neural networks that emulate the human vision system in its operation. This is the same technology that is used for smart cars so that they can drive alone and it also makes it easier for us to park. Although it has other uses. For example, it is the one that allows NVIDIA Broadcast technology to discard the background and leave us alone when we make a live video one of the most famous streaming platforms.
But what is the use of Virtual Reality? Thanks to the fact that we can know the distance of the objects in an Outside-In tracking we can know the relative position with respect to the camera. In the case of the Inside-Out technology, it is more interesting, since it is what makes it possible to discard the player’s hands from the rest and through a trained neural network to be able to interpret the different movements of the player’s hands.
This is also possible to do from an Outside-In system, but since a higher resolution camera would be necessary due to the distance, it is more feasible to do it from the cameras installed in the HMD.
Inertial systems for tracing in virtual reality
Inertial systems are nothing new and have been used in virtual reality tracking for some time, at least from the beginning in the form of MEMS devices. The clearest cases being the accelerometer and gyroscope that allow the movement of the control knobs and the HMD itself or virtual reality helmet to be positioned with greater precision .
The accelerometer’s job is to measure acceleration, which is nothing more than the change in velocity over a specific period of time. And how can knowing it help us to position an object? Well, due to the fact that speed is the variation of the displacement in time. With this information, the system can not only predict where the object is, but where it will be at a certain point in time.
The gyroscope on the other hand measures the degree of inclination of the object, together with the accelerometer it allows to trace the movement of the objects. Although both require another observer, in the form of an external viewer and that is why an external camera is needed like the one we discussed in the previous section. With all this, the system can now triangulate the position of objects in space and know where they are at all times. In some cases, to increase precision, sensors such as ultrasound microphones are used as a sensor or magnetometers or better known as compasses.
Eye tracking in virtual reality
One of the problems that Virtual Reality has is the inconsistency between what the player should see and the image shown on the HMD screen, since the only movement that is read is the movement of the head, this causes the image that is displayed on the screen does not correspond to the movement of the user’s eyes.
The idea of eye tracking is the installation of a high-frequency infrared camera that allows you to follow the movement of the viewer’s eyes and make it coincide with the camera’s point of view when rendering. In such a way that what the user sees corresponds to the movement of their eyes and is a way to reduce the lack of coordination in the movement that causes the famous dizziness. All this coordinated with the data obtained by the rest of external tracking systems.
Due to the fact that in Virtual Reality the user’s eyes are hidden by the HMD unit that he or she wears on their head, these types of sensors are located inside the Virtual Reality helmet.
