NTU and Peking University score breakthrough in image search technology for WeChat

Published on: 16-Jul-2014

A joint team from Nanyang Technological University (NTU) and Peking University, China, has come up with a new image recognition and search technology which will soon lead to more creative mobile apps.

This new technology was developed by the Rapid-Rich Object Search (ROSE) Lab, a joint initiative by NTU and Peking University, in collaboration with Tencent, a leading provider of comprehensive Internet services in China.

Mobile app developers can use the new technology, available free online, to create new apps on the popular 
​​​​WeChat platform, also known as Weixin in China.

WeChat is the world’s fastest-growing social application, according to market research firm GlobalWebIndex. As of March 2014, the combined number of monthly active users of WeChat stands at 396 million.

The new image recognition and search software allows for faster processing of information compared to any similar software currently available. Mobile developers could develop a WeChat app using the software’s photo search function, enabling users to retrieve information on almost anything they point their camera at.

For example, such apps will allow users to take a photo of an iconic statue or a landmark and to immediately read its rich history with a tap of their screen.  Or they can take a photo of a bag or shoe and search for its exact model, 
​price and availability in stores near them.

Professor Alex Kot, Director of NTU’s ROSE Lab, said the new technology is made possible through the efforts of the joint ROSE-Tencent team. The ROSE Lab signed a five-year research agreement with Tencent Technologies in November last year to embark on joint research and development.

“Together with team members from Tencent’s Pattern Recognition Centre, we developed a breakthrough method 
that will allow quick searching of images in a huge database containing millions of photos without compromising the user experience,” said Prof Kot, who is also the Associate Dean (Graduate Studies) for NTU’s College of 
​Engineering.

“With its large user base, Tencent offers a wealth of data for training and testing our mobile visual search technologies for real user applications, so we expect that more technologies to come from this collaboration.”

Ms Juliet Wang, Managing Director of the Tencent Technology Management Committee said, “Tencent attaches great importance to developing the research collaboration with top universities. The partnership with NTU ROSE 
​​Lab explores a new model of academia-industry collaboration.’’ 

Tencent’s WeChat Pattern Recognition Centre recently released the “Weixin Chat Smart Platform” – a publicly available software development kit (SDK) targeted at third-party developers.

The image recognition software developed by ROSE researchers is one of the “Weixin Chat Smart Platform” SDK’s two modules, with the other being audio recognition. The platform works on various types of devices, including Apple’s IOS and Google’s Android operating systems.

How the new image recognition technology works

Currently, when a photo is taken on a mobile phone, the entire photo is sent to the server to be analysed and to search for matching photos and related information on the database. This consumes a lot of data bandwidth and server resources, worsened by multiple independent developers, each with their own dataset. For a system ​with a 
few hundred million users, the system infrastructure will be overloaded and the search will be slowed down.

In this newly designed database system developed by ROSE and Tencent, the algorithm in the WeChat image recognition software will analyse and extract relevant bits of information (descriptors) from the photo taken.

Compact descriptors are used to search the server database for similar images and related information. By letting 
the mobile device handle the task of extracting the feature descriptors, the load on the server is reduced and data bandwidth usage is kept to a minimum, thus increasing the speed of the search.

Rigorous testing by Tencent showed that the jointly-developed image retrieval techniques are better in search performance and data retrieval, compared to other state-of-the-art approaches.

Singapore’s first WeChat app competition

To commemorate the initial success of its collaboration with Tencent, NTU is planning to host Singapore’s first competition​ to develop the most creative and innovative apps for WeChat, using the new Smart Platform software development kit.

Specially organised for the Singapore market, the competition is expected to be held in the second half of this year. This will be Tencent’s first competition based on the WeChat Smart Platform in South-East Asia.

Pushing the frontiers of new media

NTU launched ROSE in November 2013, together with two other research centres in interactive digital media, the Active Living for the Elderly Research Centre (LILY) and the Multi-plAtform Game Innovation Centre (MAGIC).

While ROSE focuses on developing innovative search technologies, LILY aims to develop 
​cost-effective technologies to assist the elderly to lead a healthy, comfortable and independent life, with MAGIC poised to champion efforts in the R&D, education, and advancement of digital gaming in Singapore.

These three centres are part of NTU’s New Media peak, one of five Peaks of Excellence or interdisciplinary areas of research in which NTU aims to put a global stamp. The other four peaks are Sustainable Earth, Future Healthcare, Innovation Asia, and the best of East and West.

***END***

Media contact:

Lester Kok
Senior Assistant Manager
Corporate Communications Office
Nanyang Technological University
Tel: 6790 6804

Email: [email protected]

About Nanyang Technological University

A research-intensive public university, Nanyang Technological University (NTU) has 33,500 undergraduate and postgraduate students in the colleges of Engineering, Business, Science, Humanities, Arts, & Social Sciences, and its Interdisciplinary Graduate School. It has a new medical school, the Lee Kong Chian School of Medicine, set up jointly with Imperial College London.

NTU is also home to world-class autonomous institutes – the National Institute of Education, S Rajaratnam School of International Studies, Earth Observatory of Singapore, and Singapore Centre on Environmental Life Sciences Engineering – and various leading research centres such as the Nanyang Environment & Water Research Institute (NEWRI), Energy Research Institute @ NTU (ERI@N) and the Institute on Asian Consumer Insight (ACI).

A fast-growing university with an international outlook, NTU is putting its global stamp on Five Peaks of Excellence: Sustainable Earth, Future Healthcare, New Media, New Silk Road, and Innovation Asia.​

Besides the main Yunnan Garden campus, NTU also has a satellite campus in Singapore’s science and tech hub, one-north, and a third campus in Novena, Singapore’s medical district.

For more information, visit www.ntu.edu.sg

About ROSE Lab

With the proliferation of mobile internet devices, the Rapid-Rich Object Search (ROSE) Lab was conceived to address the growing need to expand search beyond text’s limited capability in describing real-world objects.  The Lab aims to build visual search technology on media cloud platforms to support real-time applications with scalability to user demand.

The ROSE Lab was jointly set up between the Nanyang Technological University (NTU), Singapore, and Peking University (PKU), China, drawing upon each university’s strength in media, computer vision, and cloud computing technologies

The lab is physically located on a premise provided and maintained by NTU, in Singapore. Faculty and students from both universities have been participating in the research activities, inside and outside of the physical lab space.  The ROSE Lab is part of NTU’s New Media peak of excellence.