Blurring the seams between real and virtual

Scientists across NTU's campus are using artificial intelligence and virtual reality to augment human abilities.

Although artificial intelligence (AI), virtual reality (VR) and augmented reality (AR) might seem like the hottest new trends, each of these three technologies actually has roots that go back several decades. The reason these buzzwords have re-entered mainstream consciousness is simple: new streams of data and cheaper consumer hardware.

“VR was very popular in the 90s. The current renaissance is due to more cost-effective hardware solutions opening up new possibilities to use VR in homes and making it accessible to small and medium enterprises (SMEs), not just industry giants,” says Prof Wolfgang Müller-Wittig, Executive Director of Fraunhofer Singapore, a research centre in NTU.

The mission of Fraunhofer Singapore—the first Asian subsidiary of German applied research organisation Fraunhofer-Gesellschaft—is to help companies translate the latest research in visual computing into solutions that make sense for them, whether they’re a start-up or multinational corporation (MNC).

“It doesn’t matter if it is mixed or purely virtual environments; we see a huge potential for visualisation to address the primary human sense of sight through visual storytelling,” Prof Müller-Wittig adds.

Gearing up for industry 4.0
In particular, VR can contribute towards Singapore’s goal of staying competitive as a high-quality manufacturing hub, Prof Müller-Wittig continues. As the demand for personalised products increases, production is becoming more flexible, complex and need-specific. VR allows large MNCs to build complete digital replicas of their factories, giving them unprecedented insights into how operations can be optimised.

Less well-resourced SMEs can also harness VR and AR to take part in the fourth industrial revolution, Prof Müller Wittig says. Even on a less ambitious scale, VR and AR can help companies extract patterns to predict when and where maintenance will be required and provide training to employees and technical support to customers.

One of the many active collaborations Fraunhofer Singapore has with the Building and Construction Authority (BCA) Academy involves testing the use of VR and AR to reduce the cost of in-house training in fields that employ complex machinery and processes. For example, one project uses VR to provide context-sensitive information on materials and components during training.

Providing another perspective on the use of VR in industrial settings is Mr Niels de Boer, Programme Director of the Centre of Excellence for Testing & Research of AVs-NTU (CETRAN), who says VR can be used to test out scenarios that would be too difficult or risky in real life.

Unlike applications in games such as Pokémon Go, mistakes or weaknesses in industrial AR or VR could become a matter of life and death. Thus, researchers at CETRAN have developed a virtual autonomous car as well as a virtual model of Singapore to understand the way humans and autonomous vehicles would interact in the real world.

“We have effectively created two separate virtual systems, using one to understand how the other will respond. This allows us to test the vehicle and expose it to many scenarios, which we otherwise wouldn’t be able to do,” Mr de Boer says. “And if you have enough computing power, it is possible to test many more scenarios than would be possible in a physical system.”

Apart from needing to be very precise and robust, industrial applications of AR and VR have the additional requirement of operating in real time.

“Our main point is human-centred visual solutions, so people should be able to interact in real time,” Prof Müller-Wittig says. “Developing industrial applications is not like working on special effects where there is time to work on the animation.”

When real and virtual worlds collide
“The real challenge for research in AR is to be able to mix real worlds and virtual worlds together in real time,” says Prof Nadia Magnenat Thalmann, Director of the Institute for Media Innovation (IMI) at NTU.

“What we see today are simple virtual worlds with limited interactivity and digital avatars that have little autonomy. There are very few fully autonomous virtual humans that are aware of the actions of the user.”

At IMI’s BeingTogether Centre, researchers are working to achieve realistic, three-dimensional telepresence by capturing and conveying contextual information such as shapes, people, material and lighting in real time. The ultimate goal, Prof Thalmann explains, is to create an immersive virtual space that enables geographically-distant people to feel as though they were in the same room.

To achieve this aim, there are considerable technological hurdles to be cleared. One key challenge, she says, is replicating real people in virtual environments as avatars, relaying their expressions and gestures in real time. The next challenge is bringing these avatars into a common virtual environment, giving real users the illusion that they are together and sharing the same space.

“The situation becomes even more complicated if social robots are stand-ins,” she says. “The robots must not only be able to perceive and understand the speech, actions and emotions of the real humans ‘teleported’ into the virtual environment, but also know how to react in an appropriate manner. All these challenges are on our research agenda and are being pursued through various interdisciplinary subprojects.”

A data-driven approach to education
Beyond telepresence, such realistic environments could make a real difference in medical and healthcare training, says Assoc Prof Nabil Zary, Acting Director of the Medical Education Research and Scholarship Unit at the Lee Kong Chian School of Medicine (LKCMedicine), NTU’s joint medical school with Imperial College London.

With this aim in mind, LKCMedicine’s e-learning ecosystem has been designed to be an intelligent and immersive learning environment. For example, its Joint Education Data Infrastructure is built on state-of-the-art data-centred architecture to capture user-centric learning data from the medical school’s ecosystem and generate actionable insights. Based on these insights, and together with other schools at NTU and with strategic industry partners, LKCMedicine educators develop AI applications to provide students with personalised learning experiences.

In particular, novel immersive environments and interfaces with a focus on AR, VR and augmented virtuality— where user interaction takes place in a virtual space—are being trialled in the context of the LKCMedicine Multiverse initiative. An Education 4.0 initiative, Multiverse aims to develop interconnected physical and digital spaces that allow for immersive learning experiences leveraging a continuum of technologies such as AR, mixed reality (to see the world differently) and VR (to see a different world).

Security made smart
In addition to its use in simulated work and learning environments, AI offers the promise of safer data use. When it comes to cybersecurity, for example, machine learning will be a real game-changer, says Prof Thambipillai Srikanthan, Executive Director of the Cyber Security Research Centre @ NTU. “The key advantage of machine learning algorithms is that they adapt as time progresses. Hence, they can easily respond to variations in attacks.”

This versatility is crucial for dealing with present-day cyber attacks that are constantly evolving, he adds. Furthermore, AI techniques that involve automation can surpass human ability to spot intrusions, providing high quality security at a lower cost.

But it is not a matter of man versus machine, Prof Srikanthan emphasises. Rather, AI can be thought of as a tool that augments and enhances human abilities. For example, the classification capabilities inherent in AI techniques can help security experts categorise incoming threats, increasing the speed at which they can analyse each threat and recover from unwanted breaches.

“Machine learning has made significant contributions to cybersecurity, particularly for objectives like file classification, spam detection and malware identification,” Prof Srikanthan says. “In addition, we are also working on building context-aware classifiers that will incorporate additional information such as device type and location to make cybersecurity even more intelligent.”

A means to an end
As an early adopter of AI, VR and AR technology, NTU will be able to provide models of how these technologies can be used to help train doctors and other healthcare professionals in the face of limited resources, says Assoc Prof Zary.

“The success that we have achieved in moving our e-learning environment into a smart learning environment is just the beginning of the journey. The next step is positioning ourselves as one of the leading players in AI in medical education,” he says.

For Prof Müller-Wittig, who has used AR technology extensively in the medical e-learning and digital heritage domain, technology is likewise simply a means to an end, albeit an important one. “I think it is important that partners and potential customers can see beyond what is currently possible and nearly done. We shouldn’t be hindered in our vision for the future due to a lack of technology that will come sooner or later.”

“Our daily business is to inspire, to help those with domain expertise imagine how technology can be used in their field,” he says. “Thus, having a dialogue and active listening is a very important part of our process.”

The article appeared first in NTU’s research & innovation magazine Pushing Frontiers (issue #13, June 2018).