Enabling the expansion of 'street view' technology across China
Cardiff University’s research expertise in the development of image processing and manipulation techniques has enabled the rapid expansion of street view technology in China, which is now accessed more than six billion times a day.
Researchers from the School of Computer Science and Informatics, working with colleagues at Tsinghua University in China, used image processing algorithms including image graph matching and meta-filters, to enable the efficient processing of street view images.
They collaborated with Tencent Holdings Ltd - the primary street view provider in China - to expand the multinational conglomerate’s Street View service, which was facing limited coverage in China due to restrictions preventing external companies from accessing the sector.
Impactful research
The work of the computer scientists has been key to enabling the rapid expansion of Tencent’s Street View technology across China, providing quicker, higher-quality, and more accurate images and location services.
The Tencent Street View and location service is now accessed over six billion times a day, widely employed by commercial organisations including DiDi – the leading ride sharing service in China, and Mobike – a global bike-sharing service.
The growth of Street View also supported its integration into commercial applications within Tencent and beyond. This includes Tencent’s flagship product WeChat, the most widely used mobile instant messaging app in China with an estimated one billion unique monthly users, and QQ, Tencent’s PC-based messaging app. Street View allows users to quickly share images of their location, scan environments to pinpoint their location, and improve accuracy of a user’s location.
The improved location accuracy enabled by this research has provided commercial users with a greater level of accuracy and quality that enhance the delivery of their products, as well as providing the companies with more accurate location data.
The enhanced image and location data has also delivered additional benefits for Tencent’s development of autonomous driving vehicles in China.
Our integral role in research-led impact with Tencent was recognised by the Chinese government’s highest science and technology honours.
About the research
Cardiff University's expertise in the development of image processing and manipulation tools was instrumental in overcoming key technical challenges in four main areas:
Identifying boundaries and stitching images together
Our researchers developed techniques based on graph matching allowing images captured by individual cameras to be stitched together robustly and efficiently to form Street View panoramas, while research in image extrapolation ensured boundary consistency in the composite panoramas.
Novel image enhancement to improve panoramas
The individual images used to construct panoramas are usually captured from a wide range of viewpoints and under differing lighting conditions, making them challenging to align and blend together. Our research proposed techniques to improve the quality of the generated panoramas consistently across a wide range of input images with minimal manual input, including the use of meta-filters that approximate multiple image filters and PatchNets, a hierarchical representation of structural and appearance characteristics of image regions.
Panorama completion and blending
Limitations of the capture devices leads to missing content at the bottom of panoramas. For Street View applications, pedestrians and sensitive text also need to be removed. Existing techniques for image completion do not work well due to the large distortion introduced when combining individual images into panoramas. We introduced a novel technique that balances speed and quality by morphing sections of an image to regular dimensions, applying traditional completion methods, before warping the image back to the original perspective.
Identifying useful geographic and street information
Although techniques to detect and classify traffic signs exist, this is challenging when the area of interest occupies a small fraction of a larger image. Our research provided a key step in improving accuracy by determining a tighter bounding box around objects of interest, leading to the creation of a new public dataset of 25,000 panoramas from Tencent Street View. This image information provided the basis for Tencent location services that correct inaccurate GPS location data.
The combined research led to a Cardiff University-developed suite of key image processing techniques that underpinned the production, expansion, and use of Street Views in Tencent products and supported its massive expansion and use in recent years.
Meet the team
Publications
Miao Wang, Yu-Kun Lai, Yuan Liang, Ralph R. Martin, Shi-Min Hu. BiggerPicture: data-driven image extrapolation using graph matching. ACM Transactions on Graphics, vol. 33, no. 6, pp. 173:1-13 (2014)
Shi-Sheng Huang, Guo-Xin Zhang, Yu-Kun Lai, Johannes Kopf, Daniel Cohen-Or, Shi- Min Hu. Parametric meta-filter modeling from a single example pair, The Visual Computer, vol. 30, no. 6-8, pp. 67A3-684 (2014)
Shi-Min Hu, Fang-Lue Zhang, Miao Wang, Ralph R. Martin, Jue Wang. PatchNet: A Patch-based Image Representation for Interactive Library-driven Image Editing, ACM Transactions on Graphics, vol. 32, no. 6, Article No. 196 (2013)
Zhe Zhu, Jiaming Lu, Minxuan Wang, Songhai Zhang, Ralph R. Martin, Hantao Liu, and Shi-Min Hu. A Comparative Study of Algorithms for Realtime Panoramic Video Blending, IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 2952-2965, (2018)
Zhe Zhu, Ralph R. Martin, Shi-Min Hu. Panorama completion for Street Views, Computational Visual Media, vol. 1, no. 1, pp. 49-57 (2015)
Zhe Zhu, Jiaming Lu, Ralph R. Martin, and Shi-Min Hu. An Optimization Approach for Localization Refinement of Candidate Traffic Signs, IEEE Transactions on Intelligent Transportation System, vol. 18, no. 11, pp. 3006-3016 (2017)
