Technical challenges and competition of 3D graphics technology in mobile phone applications

Technical challenges and competition of 3D graphics technology in mobile phone applications

The growing enthusiasm for graphics-oriented video games in the Asian market is spurring a huge demand for the next generation of mobile phones with complex 3D graphics capabilities. This article introduces the technical challenges of 3D graphics technology in mobile phones, 3D graphics API standards, and the technical competition that developers will face, helping engineers understand the current status of 3D graphics technology applications, and analyze existing technical challenges and possible technical competition.

Only two years ago, mobile phones only provided basic 2D bitmap graphics games as additional functions, just as mobile phones that could send text messages appeared only a few years ago. Mobile game functions have become popular in Asian countries, and consumers are easily attracted to the next generation of color screen mobile phones. Users soon began to hope to play higher-performance games on their phones. Mobile phone manufacturers have also introduced mobile phones that take advantage of advanced graphics features (16-bit color per pixel and QVGA resolution color screen with limited graphics acceleration) and more complex and interactive games. With the improvement of performance, the development speed of graphic-oriented mobile phones in Asia is faster than that of similar mobile phones anywhere in the world.

Now there is a 3D graphics function, the first stage is to introduce 3D graphics function through the software. This requires greatly improving processor performance, but certain operations still have significant performance limitations, and highly complex interactive features are not yet feasible. Last year, especially in the second half of last year, mobile devices with hardware-based, true 3D graphics capabilities began to appear on the market. However, in many other parts of the world, people still have doubts: why do we need a color screen for a common voice communication device, and also add 2D / 3D graphics to it? Playing advanced video games on mobile phones may soon become popular in Europe and in the United States in the next two years. In Western countries, high-end users interested in connecting to the Internet, wireless Internet access, GPS services, and applications such as built-in cameras, multimedia messaging, mobile video, MP3 playback, and games are increasing.

But an understandable concept of caution has begun to spread in the mobile phone industry: Although games rely on 3D graphics, or actually 3D graphics, to make higher-performance mobile games possible and drive future mobile phone replacement markets, this is not absolute After all, the game is only an additional function of the phone as a voice communication device. At present, there are several first-generation mobile phones with various architectures and 3D graphics functions on the market, some of which claim to have lower power consumption, while others promote better visual effects or specific functional characteristics. But overall, they do not differ much in performance or function.

The technical challenges facing creating the demand for advanced features is only one aspect of the challenges facing mobile device manufacturers. It is technically not easy to use more powerful processing capabilities to process and operate still and moving image data tasks, plus additional storage resources without increasing the size and weight of the device. If coupled with at least maintaining or ideally increasing the battery's standby time, the technical challenges will become extremely severe.

At this time, the graphics technology in the dedicated video game market has also greatly developed. For such devices, especially console-based devices, there is not much pressure to put powerful processing capabilities and dedicated game engines into narrow spaces. Power consumption is still a problem for portable gaming devices, although they often have enough space to accommodate other hardware, such as complex 3D graphics chipsets and accelerators. But dual screens, touch screens, interactivity, and the ability to handle a large number of pixels will make mobile phones a new threat facing game console companies such as Nintendo and Sony.

To integrate anything close to these performances into a mobile phone requires a completely new consideration of the mobile phone's architecture. About 6 hours of battery life is at least acceptable for game consoles, but for mobile phones, even this double time is not enough. In view of this, most mobile phone designers adopt solutions that can increase the processing power and work independently of the core CPU to reduce the power consumption of the device.

However, mobile phones are also under pressure from space, and other functions must be added with minimal silicon (hardware) overhead. Even with the first type of color screen, the task of processing and transmitting pixels still requires some coprocessors or dedicated DMA engines that understand the pixel space. Today, although quite a few DSP-based chips and cores, graphics accelerators, dedicated coprocessors, graphics engines, and software solutions are available, how to achieve them is still critical. Decisions such as architecture, hardware and software division, and system-on-chip (SoC) or chipset need to be made early in the design process and are critical. The manufacturing costs can only be allocated by mass production. Later design changes greatly increase the cost, and project delays caused by the changes will risk losing major markets and profit opportunities.

The graphics function is not a later added function, but an integral part of all operations related to the display. It takes a lot of work to integrate and verify this complex electronic circuit. The emergence of a dedicated platform is of great significance to mobile phone developers, because it can facilitate design reuse, while saving a lot of design time and verification work. The 3D graphics function is one of the most successful application subsystem implementations, especially it adopts a special accelerator that can perform some special functions (which is not easy to achieve through a general processing platform).

3D graphics API standard--Open GL-ES

The mobile phone industry needs the support of game developers to ensure affordable games for users to choose from. One of the potential challenges is the need for application programming interfaces (APIs) to communicate between processing hardware, software, and graphics engines. The mobile phone industry has foresightedly avoided the early detours of the PC 3D gaming market, when various proprietary APIs caused unnecessary competition and conflict. Major mobile phone manufacturers, chip suppliers, graphics engine and software providers, game developers, and infrastructure development companies, etc., quickly identified this potential problem and conducted extensive cooperation / collaboration to jointly ensure Use a consistent API from the beginning.

The emergence of Open GL-ES as the mobile 3D graphics industry standard avoids unnecessary market competition caused by mobile phone manufacturers due to mutually incompatible game software formats. Open GL-ES originated from the desktop market, and is particularly easy to scale, and has stripped some of the original unnecessary functions, so it can be realized with a smaller board area.

Open GL-ES has gained broad industry support, not only from mobile phone manufacturers and device and kernel providers, but also from graphics engines, game developers and operating system vendors. More importantly, although Open GL-ES is a low-level API, it can fully complement the higher-level JSR 184 API for J2ME applications in the Java game environment.

Now, industry experts are developing Open GL-ES 2.0, they are those who understand the market and hardware limitations and can correctly assess the resources required to develop software and hardware for future applications. The second-generation mobile phone may enter the design stage this year, and a new product will be launched a year later-although this largely depends on the user's acceptance of the first-generation product. At that time, these phones will be launched in large quantities, and it is expected that Open GL-ES 2.0 will be released in preparation for the development of the third generation of mobile phones with 3D graphics capabilities.

Faced with technological competition The real "technical competition" will start with the second generation of mobile phones with 3D graphics capabilities. Manufacturers will unprecedented competition in pure technical performance, especially once the API standard is fully established, there will be little disagreement. Some people believe that mobile phone manufacturers should avoid expanding their platforms too quickly and create some of their proprietary extensions. The industry must ensure that the Open GL-ES standard API evolves with the market.

In fact, the Open GL-ES roadmap has been established, and people have begun to develop Open GL-ES 2.0. Although the current API is based on the state machine, for the third generation mobile phone, it must be developed into a standard based on Shader (shading engine). Now, the API is based on a fixed function pipeline. This fixed function pipeline can enable or disable certain functions according to the current rendering conditions. It enables manufacturers to manufacture different mobile phones based on throughput, number of pixels, and similar functions.

With Open GL-ES 2.0, people can program certain elements in the graphics pipeline, allowing content developers to precisely define how to deal with vertices or pixels. This can use procedural algorithms to make code performance dependent on implementation, so it can not only provide suppliers with a larger feature set and performance innovation space and greater differentiation (especially in terms of visual quality and effects), but also for The developer keeps a public platform.

Compared with the desktop or console game console market with larger display screens, graphics subsystem developers must revisit the mobile phone market. The pixel density of the 2.2-inch display of a mobile phone is much higher than that of a laptop screen. The new 2.5-inch display is likely to have a density of up to 400 pixels per inch, which is twice the current pixel density. Therefore, the traditional method of using a large number of polygons and fast polygon speed to obtain high-resolution graphics is almost completely unsuitable. Smart designers will turn their attention to more advanced (but based on rougher models) technology for lighting and display. The difference is the pixel quality and the amount of high-end processing available for these pixels (the indicators are operations per second per vertex and operations per second per pixel). The developer plans to use this method to not only provide the picture quality that could only be achieved through millions of polygons, but also not to highlight the processor performance, memory capacity and power consumption indicators.

The planned Open GL-ES standard development not only provides a clear road map and stable growth for the entire mobile 3D graphics market, but also provides a healthy and competitive market for manufacturers. It is expected that in the next few years, with the development of open standards, mobile 3D graphics technology will expand to many other platforms and markets, including automobiles and avionics.

Compact 3D graphics solution Today, one of the most popular 3D graphics solutions for mobile device platforms is the PowerVR MBX graphics accelerator series developed by ARM. The product is jointly developed by ARM and ImaginaTIon Technologies. Its core is used by application processors to provide 3D graphics with the picture quality of PC and game consoles in mobile phones, wireless games and other embedded devices.

The graphics accelerator has two versions (MBX RS and MBX HR-S) that can provide QVGA and SVGA graphics performance respectively. The MBX RS version has a smaller die size and is mainly used for small wireless devices. The MBX HR-S version has higher performance. Both cores have the option of adding a vertex geometry processor, which can free the main CPU from geometry processing tasks to achieve advanced lighting features.

The PowerVR MBX solution, which supports full 2D and 3D feature sets and is compatible with Open GL-ES, uses block-based rendering (TIle-based rendering) to achieve full bandwidth-intensive pixel processing on the chip. By eliminating the Z buffer between the 3D core and external memory and ensuring that only visible pixels are written to the frame buffer, memory bandwidth requirements can be reduced. Block rendering not only enables high-bit precision rendering and synthesis (even on systems with 16-bit frame buffers), but also implements delayed textures that can cancel all redundant texture operations. This kind of scheme is easy to adopt a unified memory architecture. In this architecture, the graphics subsystem and the CPU share the system SDRAM together, so that it can be realized by a SoC that saves cost and space.

The MBX core is easy to integrate with the ARM926EJ-S and ARM1136J-S and the ARM PrimeXsys platform, so it can shorten the time to market. The core architecture is compact, saving space and silicon cost. By using mature power management technology (including module and register-level clock gating, etc.), the power consumption is minimized.

Through the use of ARM's AMBA AHB (Advanced High-Performance Bus) interface and various ARM PrimeCell peripheral devices, other integration support can also be provided. These pre-validated and mature hardware soft IP macrocells can be easily integrated with 3D graphics acceleration solutions through PrimeCell peripheral multi-port memory controller interfaces because these interfaces have been optimized for MBX RS and HR-S.

Evolving OpenGL-ES

The Khronos Working Group is an alliance of multiple member units that focuses on developing license-free and cross-platform open standard graphics APIs to create and play dynamic media on various platforms and devices. All Khronos members can participate in the development of the Khronos API specification, vote at various stages before public deployment, and accelerate the introduction of advanced 3D platforms and applications through early access to the draft specification and compliance testing.

Khronos API specifications include: OpenML for capturing, transmitting, processing, displaying, and synchronizing digital media (including 2D / 3D and audio and video streams); OpenVG, which can provide low-level hardware acceleration interfaces for vector graphics libraries such as Flash and SVG; OpenMAX, which provides access to media processing standards previously widely used in graphics, audio and picture libraries, and video codecs such as MPEG-4; and Open GL-ES.

Open GL-ES is a license-free, cross-platform API for full-featured 2D and 3D graphics on embedded systems (including handheld devices, instruments, and vehicles). It is a set of well-defined desktop OpenGL that can create a flexible and powerful low-level interface between software and graphics acceleration. OpenGL ES 1.0 includes Common profile (common class) and Common-Lite profile (shared simplified class) for floating-point and fixed-point systems, and the EGL specification for hand-held connection with local window systems. OpenGL-ES 2.0 is currently under development.

In addition, a "safety-critical" working group was established to help port the standard to other mobile platforms that use 3D graphics, especially automotive and avionics.