Strategies to Accelerate 5G NR Device Acceptance Test

This article introduces the following three strategies to accelerate device acceptance test:

• Leverage conformance test toolsets to expand use case testing
• Perform stress testing device battery consumption under different use case scenarios
• Use virtual drive test to find issues before field drive tests

Content Summary

Device Acceptance Test Is a Key Part of the Device Workflow
Three Strategies to Accelerate Device Acceptance Test
1. Leverage Conformance Test Toolsets to Expand Use Case Testing
2. Stress Test Device Battery Consumption Under Different Use Case Scenarios
3. Use Virtual Drive Test to Find Issues Before Field Drive Tests
Looking Ahead

Acceptance tests take several forms in different industry sectors. Generally, the acceptance test is a tool to ensure a product, component, or service meets an expected level of functionality or performance before customer acceptance.

In wireless communications, the acceptance test serves to ensure a device will meet the key performance indicators (KPIs) on a specific mobile network. Consumer expectations are high with 5G New Radio (NR) and consumers typically blame the mobile network operator (MNO) for any issues they experience. If a device causes dropped calls or crash a network, the operator gets the blame. Therefore, network operators stress test devices on their network to ensure that they will live up to consumer expectations.

The operators serve as gatekeepers to what their networks support. They create performance and quality metrics as entry qualifications to their network. An operator acceptance test verifies whether the device has adequate performance and security functions for the specific network. The acceptance test also helps to identify and resolve issues before a device gets into the hands of a customer.

The 5G NR vision of widespread connectivity, ultra-fast downloads, low-latency high-reliability connections, as well as the addition of millimeter-wave (mmWave) operating bands, requires testing a large matrix of use cases. Whether you go through a formal device acceptance program or complement your current test plan, you can use the strategies in this paper to shorten your device acceptance phase and achieve better quality-of-experience (QoE) and performance.

Device Acceptance Test Is a Key Part of the Device Workflow

Players in the wireless communications ecosystem – chipset vendors, device makers, and MNOs work closely together to ensure all the pieces of a 5G communications system coalesce to deliver a high QoE to consumers. Chipset vendors develop semiconductors to integrate into device makers’ products. Operators combine the devices, radio access, and network equipment into operational networks. Network operators have significant influence over the products they use to provide communications services to their customers and therefore, play a major role in defining the specifications in the standards bodies.

Radio access network (RAN) working groups in the 3rd Generation Partnership Project (3GPP) define minimum specifications and conformance test requirements for base stations and devices. These tests ensure a minimum level of performance and interworking of 5G NR and 4G E-UTRA in non-standalone mode.

The overall workflow of a device is shown in Figure 1. Devices go through a design phase that includes simulation and characterization of their physical properties to ensure that the design meets the requirements. Verification and regression testing evaluate a wider set of parameters early on to ensure margins and quality. Due to the expense of formal conformance testing, most device makers perform some type of pre-conformance testing to ensure a “Pass” before spending time and expense on official conformance testing.

Conformance testing represents a major milestone in the device workflow. In this process, independent third-party certification bodies, such as the Global Certification Forum (GCF) and PCS Type Certification Review Board (PTCRB), provide validation to ensure commercial 5G devices comply with the requirements specified in the 3GPP conformance documents. Regulatory tests are also part of the conformance test suite and ensure a device does not have excessive emissions or pose any health or safety threats. Although not mandatory, most major operators require conformance tests.

Device acceptance testing marks another major step in the device workflow. Figure 1 shows how the operator acceptance test fits into the overall workflow. All major operators use device acceptance testing to vet a product before it is released on their networks.

Ultimately, a device acceptance program enables operators to improve customer experience and differentiate themselves in the market. As shown in Figure 1, network operators in step 1 need to ensure that the devices they accept on their networks run on different infrastructure vendor equipment used in their network. This is known as a network vendor interoperability test.

In step 2, field trials deploy a device into the field and check the device’s operation in different locations with various usage scenarios. Field trials use to walk or drive tests through different geographic locations while performing various tasks.

Step 3 represents the conformance testing described earlier in this paper. In step 4, network simulation tests a device’s performance and interoperability using network simulators to mimic the operator’s network behavior and test scenarios that are not included in the conformance tests, or that are not easily performed on the actual network infrastructure. These tests consist of RF, protocol signaling, application, and performance-related test scenarios.

At a minimum, all operators perform steps 1 and 2 and many, but not all, require step 3. Typically, only the largest operators perform step 4, but this step is very useful and offers the greatest assurance of robust device and network performance. Employing all four steps enables the best chance to achieve the highest QoE and performance on a network.

Three Strategies to Accelerate Device Acceptance Test

It was nearly 10 years ago that we saw the first 4G LTE introduction which represented a significant improvement over 3G. Now, 5G requires technical advances in the physical layer to provide greater flexibility and scalability in support of many new use cases. Also, massive multiple-input, multiple-output (MIMO), and mmWave beam steering represent major changes in how 5G NR devices connect across sub-6 GHz and mmWave operating bands. It is critical to validate device QoE and performance on the network given these changes.

Operators create test plans that include performance and functional tests. These tests cover more than the requirements for conformance testing. The tests focus on specific features/functions for the operator’s network. For example, an operator might market its network as the most reliable, or as having the broadest coverage. The operator may also introduce capabilities in mmWave operating bands using higher channel bandwidth. To ensure customer satisfaction, the operator increases its focus on acceptance tests for the aspects that are essential to its business. Devices and networks are stress-tested for the many different use case scenarios expected on the network. It is important to understand operators’ priorities so you can focus on the aspects that matter most to them and avoid wasting time on features that receive little attention.

Due to the complexity and possible confidentiality of operators’ acceptance tests, device vendors may find it difficult to perform these tests ahead of time. However, there are strategies device vendors can use to test for the most likely use case scenarios to increase the likelihood of a “Pass”. Using these strategies, you can accelerate device acceptance and deliver better QoE and performance on an operator’s network.

1. Leverage Conformance Test Toolsets to Expand Use Case Testing

Device conformance tests consist of RF characteristics, performance, radio resource management (RRM), and protocol. Generic test suites represent the most likely network configurations. These test configurations are sufficient for some use cases, but not all. Network operators have specific configurations or use cases to test that are important for their network. For example, for LTE, a conformance test configuration used 10 MHz channel bandwidth, but some operators deployed 20 MHz channel bandwidth. In this case, operators performed additional testing to ensure quality and performance with 20 MHz bandwidth.

Network emulation allows you to test devices under many different use scenarios in a lab environment. A flexible conformance test solution provides a network emulation platform along with the test cases specified in the 3GPP standards. You may use conformance test solutions for operator acceptance tests if the conformance toolset allows customization of test cases beyond the certification requirements with additional test equipment. For example, adding a power analyzer enables testing battery performance, or extending the test equipment frequency range enables testing of different frequency band combinations or testing for other potential interfering signals.

There are years of expertise built into conformance toolsets that will make acceptance testing easier. At Keysight, for example, we work with leading chipset and device manufacturers to test and ensure their designs perform to the new standards and meet consumer expectations. Much of this work happens before standards completion and is updated as the standards evolve. By working from the same platform in early R&D through design validation and into conformance and device acceptance testing, continuity and expertise are embedded into the platform and across the device, workflow accelerating time to market.

It is important to select a test platform that incorporates regular software updates that include newly validated test cases to ensure testing against the latest standards. It is also beneficial to choose a platform that can cover 4G and 5G, as well as range from sub-6 GHz to mmWave frequencies, resulting in a more flexible solution that can cover the necessary test scenarios.

2. Stress Test Device Battery Consumption Under Different Use Case Scenarios

Manufacturers continue to add more features and functions to smartphones that consume more battery life. Also, with the use of mmWave operating bands and 4G LTE/5G NR dual connectivity, device power is critical to establish and maintain a quality communications link with 4G and 5G NR base stations. 5G NR use cases vary widely which impacts battery requirements and performance. Enhanced mobile broadband (eMBB), for example, focuses on high data throughput, requiring higher processing power that quickly drains the device battery. By contrast, massive machine-type communications (mMTC), require the support of small data packets transmitting over very long timeframes.

Battery life is a key influence on the consumer’s perception of product quality. A device manufacturer that can produce a device that operates longer between charges and under many different usage scenarios can gain a significant competitive advantage. Battery consumption tests require testing when connected to a network, and under the many different usage scenarios expected in 5G.

5G devices need to handle multiple, complex tasks. Battery performance tests need to evaluate battery life in real-world situations under different usage scenarios. Network operators develop their tests and focus on specific customer profiles based on the services and features they plan to offer on their network. Figure 2 shows an example of battery consumption under different usage scenarios. Battery performance under different usage scenarios of data consumption, talk time, or with the use of location-based services is a common test. As a device manufacturer, you can select key representative use cases that you anticipate on the mobile network and stress the device to ensure battery life with many different permutations and combinations of activities.

With a network emulator and power analyzer, you can test many different battery scenarios by simulating 4G/5G radio access networks and evaluating power consumption under real-world conditions.

Figure 2. Battery consumption under different usage scenarios

3. Use Virtual Drive Test to Find Issues Before Field Drive Tests

LTE relied heavily on the existing 2G/3G infrastructure to provide coverage until 4G was fully developed. This meant that devices required testing to ensure seamless handovers from one LTE cell to another, as well as handovers from 4G LTE to 2G/3G cells. Also, devices needed to switch between data and voice without losing the connection. While it was important for 4G to test for handovers between different radio access technologies (RATs) under different channel conditions, it is even more critical for 5G NR because of new initial access procedures.

In 5G NR non-standalone mode (NSA), 5G devices operate alongside and with the 4G infrastructure. While the 5G NR conformance requirements test for handovers, switching and fallback to 4G cell sites, these tests are not sufficient to determine the user’s experience during different field conditions or for covering all operator-specific handover scenarios.

Therefore, it is necessary to evaluate real-world handovers between radio networks. These types of tests are typically performed in the field using a driving test. Field drive tests, however, can be very expensive when you consider the number of hours required to test the many different scenarios in all geographic locations. Imagine sending resources out on the road to conduct these tests in all parts of the U.S, Europe, or China. The field tests would take many months, if not years to complete.

While testing a device under simulated conditions in a lab enables testing of the basic operation, operators also need to understand how the devices will behave with different signal propagation issues like path loss, multi-path fading, and delay spread. They also need to understand how the device reacts if something goes wrong.

Using virtual drive testing, device makers test the real-world handover performance of their device before deployment in a live network. Virtual drive test uses data captured in the field to build tests that replay drive or indoor test routes by emulating real-world RF network conditions, including network signaling, cell settings, and RF in a controlled laboratory environment. A real network infrastructure or a simulated network using a network emulator can replay the network conditions. The virtual drive test shown in Figure 3 enables real-world mobility scenario testing in a lab environment. You can do fault analysis and performance optimization by applying different test case and channel scenarios to identify potential issues. This will deliver a better QoE and performance before the completion of an official drive test.

Figure 3. Use virtual drive test to assess true performance and interoperability of networks in the laboratory

A virtual drive test is a proven test technique. Mobile network operators used this technique to assess the performance of 4G devices in network vendor interoperability labs (NV IoT) and for testing device performance verification in high-speed train scenarios.

Looking Ahead

Testing 5G NR introduces many new challenges. While 3GPP standard tests remain in development, conformance test solutions are a good starting point for your device acceptance test plan. By incorporating network simulation and expanding your test plans to include the most likely operator use scenarios, you can stress test your device and battery consumption against operators’ goals. Using virtual drive test also provides a means to validate quality-of-experience and performance with changing channel conditions before going to expensive, resource-intensive drive tests. Use these strategies to ensure your designs hold up to consumer expectations and get one step closer to a successful device acceptance test.

Source: Keysight Technologies