/enri-thumbnails/careeropportunities1f0caf1c-a12d-479c-be7c-3c04e085c617.tmb-mega-menu.jpg?Culture=en&sfvrsn=d7261e3b_1)
/cradle-thumbnails/research-capabilities1516d0ba63aa44f0b4ee77a8c05263b2.tmb-mega-menu.jpg?Culture=en&sfvrsn=1bc94f8_1)
7e6fdc03-9018-4d08-9a98-8a21acbc37ba.tmb-mega-menu.jpg?Culture=en&sfvrsn=7deaf618_1)
Landmark Search
With the rising adoption of smart phones and tablet PCs, mobile multimedia services such as visual search and augmented reality are now becoming popular. To meet the increasing demands for searching landmarks on mobile devices, we have built up a low-bit rate mobile visual search system on a million-scale database of landmark images.This system can extract compact visual descriptors as low as 512 bytes on mobile clients and conduct efficient landmark retrieval on this million-scale database.
One highlight of the system is the compact descriptor, which is proposed to address the challenges of data transmission, power consumption, computing capability and memory limitation on mobile platform. In a mobile visual search scenario, it may take several seconds to directly send a JPEG query image over a slow wireless link and the power consumption will be high. To improve the user experience, we propose to directly extract compact visual descriptors on the mobile end. The main advantages of using the low bit rate descriptor are two-fold. First, the descriptors are expected to be compact and discriminative to reduce overall query delivery latency. Second, the extraction of the compact descriptor is surprisingly efficient and ensures impressive power saving. Therefore, the compact descriptor is the best choice for mobile search systems.
Recent research, development and standardization of compact descriptors for visual search involve numerous industry efforts from STMicroelectronics, Samsung, Qualcomm, Huawei, NEC, etc. In particular, this topic relates to an ongoing MPEG CDVS (Compact Descriptor for Visual Search) standardization. Our PKU team plays as a leading contributor and one of the draft editors of the standard.
In the mobile landmark search system, our mobile client sends compact descriptors of variable size (512 ~ 16k bytes) to the server end, on which efficient visual search are conducted on a million-scale database of landmark images. By using the compact descriptors, our system demonstrates several advantages:
- Transmission cost over the mobile wireless network is reduced.
- The power consumption is greatly reduced by sending only low bit-rate query descriptors, as data transmission is one of the most power-consuming modules on mobile devices.
- The million-scale image database can be effectively stored and indexed on the server with compact descriptors.
- The visual search algorithm is extremely fast and accurate when using compact descriptors, which outperforms the state-of-the-art.
- The system is relatively robust to partial occlusions, changes in vantage point, camera parameters and lighting conditions.
The Future
Compact visual descriptors extraction is a fundamental technology for mobile visual search and mobile augmented reality. Our research team will continue to develop more powerful compact visual descriptors, targeting at ultra-fast feature extraction, extremely low computation complexity and low power consumption, as well as robustness.
Future work also includes the extension of mobile visual search to mobile augmented reality applications. Compact descriptors are more promising for augmented reality and some new components such as registering 3D scene and tracking technology should be considered.