Smartphones get even smarter with mobile AI

Source: Huawei. Zhou holds a stand with an
embedded Kirin 970 chipset.

Huawei's new flagship phones tend to come with new chips that facilitate new capabilities. The recently-announced Kirin 970 mobile artificial intelligence (AI) platform is expected to appear in the upcoming Huawei Mate 10 smartphone, bringing better connectivity, computing, audiovisual experiences, and battery life.

"We don't think a chipset is just for benchmarks," said Eric Zhou, Product Marketing Director, Huawei Wireless Terminal Chipset Business Unit during a deep dive of the Kirin 970 in Singapore. "The important thing is the user experience. We don't just make chips, we make a total solution in a chipset to provide (a better) user experience."

The Kirin 970 is a technnical wonder, packing 5.5 billion transistors into a chipset. It has to survive on battery energy of 2,000-4,000mAh that has been allocated for the entire device.

"The central processing unit (CPU) has to handle task scheduling, load balancing, and memory allocation. The graphics processing unit (GPU) is for user interface rendering and graphics processing. The neural-network processing unit (NPU) has to handle AI computing. The image signal processor (ISP) and digital signal processor (DSP) are doing camera and image processing," Zhou explained.

The NPU is a new feature for Huawei's mobile architecture, and appears in a smartphone chipset

for the first time. It draws on AI learning software like mobile versions of TensorFlow and works with Android AI API, processing information differently from a CPU or GPU. A GPU delivering the same performance as the Kirin 970 NPU would be 10x the size, while a CPU doing the same would have to be 56x the size, Zhou said. Using the GPU or CPU to do the work would also consume battery life more quickly.

Computing

Clement Wong, Director of Global Product Marketing for the Huawei Consumer Business Group, said in an earlier overview of the Kirin 970 conducted at Huawei's headquarters in Shenzhen, China that performance receives a boost when AI is used to handle task scheduling, load balancing, and memory allocation.

"It will run fast and stay fast," Wong said, observing that phones tend to slow down as more apps begin running in the background, and as software demands rise over time. With AI capability, a  phone would be able to manage background apps and prioritise resources so that the phone continues to run smoothly, he noted.

"NPUs, because of their learning capability, will have even more improved operations," he said. 

Connectivity

The Kirin 970 comes into play in enhancing phone features like connectivity, shared Zhou. While a mobile phone connects to one network when it is stationary, a moving phone has to deal with all kinds of connectivity challenges, from interference and Doppler effects* to successful handovers from network to network. Radio spectrum is also fragmented globally, with different countries making use of different bands for communications. The 4G standard alone spans 29 bands of spectrum in different countries, with  different operators use different spectrum frequencies for 4G. "Many factors can affect the channel and make the connection unstable," Zhou said.

The ultimate challenge is providing communications services on a high-speed rail network, Zhou said. The Beijing-Shanghai high speed rail network covers 1,318 km and hosts 80 million passengers a year. There are 900+ cellular handover points throughout the entire ride. AI technology such as the Kirin 970 can help to keep Internet and voice connections stable, Zhou said.

Zhou explains that retaining stable Internet and voice connections at speeds of up to 350kmh is possible with an AI platform. Huawei has experience with this not just for China's "Fuxing" trains which run at 350kmh, but also Japan's Shinkansen, with top speeds of 270kmh.

Zhou noted that the a Huawei phone equipped with Kirin 970 performed better than an iPhone 7 in VoLTE voice (left) and data transmission tests (right). The white bars show how the Kirin 970 phone performed against the iPhone 7 for (from left) calling success ratio, called success ratio, call on hold success ration, 4G network ratio in a China Mobile network, data connection success rate, and time delays for pings (for this last chart, a lower score is better).

Huawei has worked to optimise performance across the entire rail network in China, conducting field trials, negotiating with various network operators to tweak their networks and enhancing connections to base stations, and now has a new tool: the Kirin 970.

To improve data transmission speeds and unlock the maximum speeds in networks around the world the Kirin 970 supports advanced communications technologies: 4x4 multiple input, multiple output (MIMO), 3CC carrier aggregation (CA), and 256 quadrature amplitude modulation (QAM)**. "We have to use different technologies to reach the top speeds of the network," Zhou said. 

The Kirin 970 also facilitates better dual-SIM functionality. Huawei's dual-SIM phones used to accept one 4G+ SIM card and a second 2G card, Zhou explained. Last year, the company managed to enable a 4G+ SIM card working simultaneously with another 2G or 3G card. This has helped users in countries where 2G networks have shut down to allow reuse of the spectrum for faster networks. Singapore closed down its 2G network in April 2017; in Australia, Optus was the last operator to switch off its 2G network, in August 2017; KT in Korea stopped 2G services as far back as 2012.

Today, the Kirin 970 supports dual SIM cards both running 4G for both cards to deliver the same quality of voice. One card can run up to 4G+ while the other can be on a 2G, 3G or 4G network. This allows VoLTE to be supported, Zhou said. With VoLTE there is no need to maintain separate networks for voice and data. It provides better voice quality and battery life for consumers, and allows spectrum to be used more efficiently.

Audiovisual capabilities

Further audio quality improvements can be achieved through AI-brokered noise reduction. Huawei is investing in neural network techniques for speech recognition, including in dynamic convolutional neural networks (DCNN) long short-term memory (LSTM), and time delay neural networks (TDNN, for instance. In a demonstration in a car, Huawei showed that speech recognition had an accuracy of 80% to 92%. This is relatively high given that a vehicle can be quite noisy, causing problems for speech recognition engines.

"With deep learning we can find out how to distinguish the human voice from noise for better voice calls, speech recognition, and sound recordings," Zhou said.

Zhou noted that it will be a natural progression for AI in photography towards helping the photographer take better pictures, for example in scenarios which typically occur faster than the hand can react. "You have to use all your knowledge – your training, what you saw, your real-time analysis to focus the camera at the right place in the right time," he said.

AI in photography is expected to address other photographic misses we all live with, such as overexposed shots, out-of-focus pictures, and unintentional motion blurs, Zhou added. Images could become sharper, and move from being beautiful to artistic. AI can help to manage composition, clarity, colour, and contrast, adjusting different aspects of the camera mechanism on-the-fly depending on the subject of the picture. "With the NPU, the Leica dual camera can recognise the object and then adjust lighting accordingly," Wong said in an overview of the Kirin 970 conducted at Huawei's headquarters in Shenzhen, China.

The Kirin 970 and AI will improve noise reduction, light detection and intelligent recognition of other salient scene details such as low light or stage lighting, for an enhanced low light camera strategy. There are motion detection sensors to distinguish static, slow, medium and fast motion, as well as an image signal processor (ISP) to manage camera and image processing, and hardware-based face detection, Zhou said. The ISP enhances throughput by 25%, and camera processing, carried out in parallel, is 30% faster.

In another demonstration of a moving subject, Zhou showed that the Kirin 970 took a clearer picture than the iPhone 8 Plus. Backlit subjects are brightened, while an improved face recognition algorithm uses a 3x3 grid instead of a single rectangle to obtain more exact data and optimise skin tones.

Source: Huawei. A comparison between pictures taken with the iPhone 8 Plus and a Huawei phone with a Kirin 970 chipset. The calendar held by the man in the image is visibly clearer in the picture taken with the Huawei phone.

The camera will also fire up more quickly. A zero shutter lag (ZSL) feature for colour-monochrome dual cameras eliminates lag after the shutter has been pressed, while focusing has also been enhanced with a four point-hybrid autofocus capability, self-calibrating focusing, face-tracking, and adjustments for point lighting, as well as flat areas.

"The camera helps us see more," Zhou said, suggesting that the future is about an augmented reality where we have more information about our environment than ever before. While Kirin 970-backed phones will be able to "see" to adjust white balance, exposure, focus and reduce noise in pictures, they will also be able to "see" more clearly with Huawei's dual camera design, low light and motion detection capabilities. Finally, they are also able to recognise, analyse and understand scenes in real-time, allowing them to determine what is happening in environments, identifying faces, objects, even mood and actions, Zhou said.

About to begin the image recognition test. The Kirin
970 chipset is the square chip to the right of the centre
of the dark motherboard.

In an image recognition test, the Huawei Kirin 970 has been clocked at 2,005 images per minute (ipm) against an iPhone 8 Plus at 889 ipm and the Samsung Galaxy S8 at 95 ipm. Watch the Instavideo of the test, in which the Huawei phone with a Kirin 970 chipset comes in firstm the iPhone 8 Plus ranks second, and a Samsung Galaxy S8+ comes in last.

With such performance, use cases for the Kirin 970 can include recognising a serving of food, estimating its weight, and coming up with a calorie count, Zhou suggested.

Third party applications can also be expected as Huawei has already seeded the Huawei developer community with Kirin 970 hardware and software, Zhou added. "We will open up the AI computing power of the Kirin 970 to developers and partners – we need more developers to do more interesting things for AI," Zhou said.

Explore:

Read the TechTrade Asia blog posts about the launch of the Kirin 970 and Huawei's smartphone focus

Hashtag: #华为OPENDAY

Note: Part of this report stems from a Huawei-hosted visit to China. The company sponsored accommodation and travel costs.

*The Doppler effect refers to how sound frequency or wavelength changes depending on the distance of the recipient from the source of the sound.

**4x4 MIMO has the same bandwidth as 2x2 MIMO but increased peak rates. It improves the user experience in weak signal scenarios; carrier aggregation improves capacity, allows higher data rates, and can make use of fragments of spectrum. 
3CC CA aggregates the networks of three participating carriers (component carriers, CC) to offer more bandwidth to users. User peak rate thus increases linearly with the number of carriers. 256 QAM is a method of combining multiple signals into one channel to increase bandwidth and improve the efficiency of spectrum utilisation. 256 QAM has a 33% increase in peak rates compared to 64 QAM.