What are WeChat applications and iPhone 7 that can bring 3D audio?

Lei Feng network (search "Lei Feng network" public concern) by: The author of this article Chen Xiaoliang, doctor of engineering, the founder of Sound Technology. Lei Feng network exclusive article, reproduced please contact the authorization.

The internal measurement of the WeChat application number is undoubtedly a heavy news. The media headlines have been covered in these days, and various fields are speculating about the important influence of the WeChat application on APP. Although we cannot confirm the details of WeChat application in terms of audio support at the moment, WeChat still has a lot of surprises in audio development in recent years.

There are two main methods for WeChat audio support. One is a WeChat built-in browser and the other is a JS-SDK . These two methods are different in the development and application scenarios. JS-SDK mainly supports audio recording and identification. However, these two functions are severely restricted by WeChat itself and have poor extensibility. Currently, they are mainly applied to the WeChat Native SDK integration scenario.

Prior to the launch of WeChat applications, the one that really brought developers' imagination was the WeChat browser. The WeChat browser was also very rubbish, and the old version had poor support for H5 and CSS3. At one time, the developers were given a "Mobile IE". In April 2016, the WeChat browser finally upgraded to the X5 Blink kernel, and the Android and IOS platform compatibility was basically the same. At that time, it provoked a wave of heat in various development communities.

What audio features are supported by WeChat Browser? We tested with html5test and the results are as follows:

As you can see, WeChat Browser fully supports the most important Web Audio API interface . The Web Audio API is a Javascript interface used in H5 to process and synthesize sounds. This API can be used to write game sound engines and to implement functions such as editing and processing sounds like various music production software.

In fact, the audio playback on the Web has always been relatively backward, before H5 mainly rely on Flash or QuickTime plug-in form. The audio experience before this, basically depends on the underlying APP driver or hardware built-in sound effects. The H5 Audio tag is of great importance, allowing basic streaming audio playback.

However, the Audio tag only provides developers with the ability to read, play, pause, and adjust the volume of a sound file. It does not handle more complex audio applications. For complex web-based games or interactive applications, especially the audio mixing, processing, filtering, sound effects, positioning and other functions implemented in game audio engines and desktop audio processing applications, the Web Audio API is required for processing.

The Web Audio API includes modular audio routing to support complex mixing, high dynamic range audio processing, multi-sampling and low-latency audio processing, dynamic sound generation, sound envelope and fade support, channel separation and merging Audio and video separation support, network real-time communication support, on-site audio processing support, dynamic compression and frequency domain processing, audio visualization support, nonlinear effects and filter support, and acoustic signal parameter generation. These functions are listed, is it a bit similar to PS software in the audio field? In fact, using these APIs, we can quickly create a "beauty" WeChat application. The following figure is a screenshot of the visual effects of the Web Audio API.

Even the Web Audio API directly supports various scene space sound effects, such as just switching between different scenes with just a few lines of H5 code, as if you were in a real scene and enjoying an immersive sound experience. These scenarios include: large/small rooms, churches, concert halls, caves, tunnels, foyers, forests, open-air theaters, sounds through a distant room at the entrance, and filter effects. Well, you can almost abandon professional audio plug-ins! Even in the future, hardware will only open channels for recording and playback, and the effects will be completely handled by the software.

Of course, the most appealing aspect of the Web Audio API is still support for spatial audio, commonly referred to as 3D audio. The 3D audio mentioned here needs to include the word "virtual". After all, there is still a big gap compared to Dolby Panorama, but if it is applied to current popular VR games or chats, this function of the Web Audio API is still Not bad. The Web Audio API fully supports virtual 3D audio functions such as Equal-power, HRTF, distance attenuation, Doppler shift, and sound source/listener model. With these APIs, tens of lines of code can achieve more realistic 3D effects.

The following describes the HRTF, the head related transfer function. In the popular sense, it is an attempt to simulate the human ear to acquire the sound model, and use this model to virtualize the feeling of the human ear caused by any sound source. Equal-power is also a method of 3D audio, but the effect is not as good as HRTF. The main parameters of the HRTF are the Interaural Time Difference (ITD) binaural time difference, the Interaural Level Difference (ILD) binaural intensity difference, and the binaural correlation coefficient (Interaural Coherence). Scientists have verified through extensive experiments that these parameters are closely related to the mechanism of the human ear processing sound, but there is no accurate theoretical derivation. So the HRTF needs to measure a large number of human ear data and use it to build an acoustic model of the black box, which is somewhat similar to machine learning.

From this point of view, the HRTF provided by the Web Audio API is still a standard model and still requires scientific research institutions and professional companies to continue research and development. Of course, open source is also a very important driving force. For example, Omnitone, which has been open sourced by Google, can be used with Web pages to experience the immersive sound effects of VR/AR. The domestic SoundAI team also has an open source HRTF database and supporting engine plan.

However, if there is no audio hardware support, these functions are also mostly furnishings and cannot produce the desired results. Especially before the iPhone 7's dual-speaker configuration, most users listened through the mobile phone still in a mono experience. If you want to experience virtual 3D effects, you must wear headphones. This is very inconvenient in the VR field. VR glasses wear headphones again. It is an obstacle. Not to mention the headphones presented by the manufacturers are basically the scourge of the user's ears. These supporting deficiencies are huge troubles for the users and are also an important reason for the slow development of 3D audio. Fortunately, most users are returning to the craving for sound quality. The tens of dollars of headphones and low-priced speakers are no longer sufficient. This point also has some clues from the side, for example, Vivo's selling point is just to emphasize the sound quality, and Apple directly on the iPhone 7 at the same time enhance the configuration of speakers and headphones. In fact, carefully think about it, since the phone comes standard with a dual camera, naturally also need to standard dual speakers, and even more than Mike these audio hardware, the user is nothing more than pay attention to the immersive effects of sound and video.

Since iPhone 7 is equipped with dual speakers, AirPods is also an independent wireless dual channel. It is believed that many mobile phone manufacturers in China will soon follow. This brings unlimited space for 3D audio applications, and even immersive 3D audio experience based on WeChat. Not far away. Since the difficulty of development is reduced a lot, what scene can stick to users? Games and music are naturally important scenes, but these two scenes are always cliches and users' expectations are basically consumed. Of course, this is a particularly important position for the Penguin Empire. The 6 million developers of the Penguin open platform will certainly launch various WeChat audio applications.

Product managers always like to dig out space and products that exceed user expectations, and the recent hot live broadcast and VR may be an important scene for 3D audio applications. Simple sound enhancement may be a surprise to users. In fact, Tencent cannot ignore live broadcasting and VR. In addition, the biggest advantage of WeChat lies in the large social groups of users. As a social platform, WeChat will not launch VR social networking, VR live streaming, or even 3D voice chat? This is also a direction we are looking forward to, although we will continue to use more music and games to support 3D audio first in WeChat applications, but at least when we chat with WeChat, we should perceive that the other party is right next to us.