By Azimuth Systems
To meet strict enterprise requirements for carrying mission-critical applications and data, the performance and stability of Wi-Fi equipment and systems must be properly and thoroughly tested. However, testing Wi-Fi is much more challenging than testing traditional Ethernet networks, due to the complexity of the 802.11 protocol, the inherent mobility of the wireless devices and the prevalence of RF interference.
In the face of these challenges, what is the definition of “proper and thorough” testing? Individual companies rely on different testing methods and metrics. That’s why the IEEE is developing a test specification that will provide objective and standardized guidelines for test methodology to help businesses evaluate 802.11 products.
The IEEE formed the 802.11T Task Group in July 2004. 802.11T is a "recommended practice," not a standard—hence the capital T instead of the usual lowercase letter. The Task Group is developing a test specification document, known as “Recommended Practice for the Evaluation of 802.11 Wireless Performance,” for expected completion in January 2008. The efforts of 802.11T are being driven by the equipment manufacturers such as Dell and Microsoft, chipset manufacturers such as Intel and Broadcom, and test system manufacturers such as Azimuth Systems.
This article provides a brief review of the difficulties of testing Wi-Fi and an overview of the draft 802.11T test specification document, including the latest information on test environments and metrics.
Difficulties in 802.11 Testing
Because of the inherent mobility and erratic transmission environment of wireless networks, the 802.11 standard requires new protocols that do not exist in 802.3, the Ethernet standard. Mobility protocols have significantly increased the number of test metrics for qualifying product performance and behavior. Wireless metrics outnumber traditional Ethernet metrics by roughly five to one.
Testing of 802.11 products is complicated by RF interference in open air testing environments and by mobility issues such as roaming and rate adaptation that require the device under test to be in motion. Dealing with interference and mobility are the primary issues that the 802.11T Task Group has worked with since its inception.
802.11T Overview
The goal of 802.11T is to provide a reliable and objective set of performance metrics, measurement methodologies, and test conditions to enable manufacturers, test labs, service providers and users to measure the performance of 802.11 Wi-Fi devices and networks at the component and application level.
The committee is modeling much of its work on test specifications in Ethernet networking, such as the IETF RFCs 2285, 2544, and 2889 , which specify metrics and methods for evaluating the performance of Ethernet switches. Measurements such as throughput, packet loss, delay and jitter can be based on these RFCs, but 802.11T will cover much new ground in defining new wireless-specific metrics.
802.11T guidelines define test metrics in the context of use cases. The task group has identified three principal use cases and associated test metrics:
- Data
- Latency-sensitive
- Streaming media
Data. Data applications—such as web downloads, file transfers, file sharing, and e-mail—do not typically impose critical requirements on the network. Data traffic is usually transmitted using low priority. Performance test metrics important for data use case include:
- Throughput vs. range
- Access point capacity
- Access point throughput per client
Latency-sensitive. Voice over Wi-Fi is a latency-sensitive application. QoS requirements for such time-critical applications include limits on:
- Voice quality (latency, jitter, packet loss) vs. range
- Voice quality vs. network load
- Voice quality vs. call load
- BSS transition (roaming) time
Streaming media. Streaming media applications include real-time audio/video streaming, stored content streaming and multicast high-definition television streaming that require the most stringent quality of service including bandwidth and latency guarantees. Performance metrics include:
- Video quality (throughput, latency, jitter) vs. range
- Video quality vs. network load
802.11T classifies these metrics as primary and secondary. Primary metrics such as voice quality directly impact the user experience. Secondary metrics—such as latency, jitter and packet loss—impact voice quality, a primary metric.
Figure 1 summarizes the bandwidth and time criticality of use cases covered by 802.11T.
Figure 1: Use cases covered by the IEEE802.11T test metrics and methods specification
Test Environments
Both conducted and over-the-air test environments are defined by the initial draft of 802.11T. To emulate motion and ensure repeatability of measurements, most of the tests currently in the draft require a conducted environment, in which each device in the test setup is placed in a shielded chamber for isolation.
As shown in Figure 2, which illustrates such a test setup, RF cables connect the antenna ports of each device to other devices through programmable attenuators that emulate distance by controlling the path loss among devices in the test setup. Shielding and filtering are employed to protect the test setup from outside interference and to achieve device to device isolation.
Figure 2: Devices connected in a conducted test setup through programmable RF attenuators. RF cables connect to the 50 ohm antenna ports of the devices with antennas removed. Programmable attenuators emulate physical distance among devices and motion.
Device isolation eliminates signal paths other than through the attenuators. Isolation between any two devices in a test setup must be greater than 110 dB due to the wide dynamic range of 802.11. Such isolation is difficult to achieve, particularly in the 5 GHz band.
Test Metrics
The following table summarizes the work that the 802.11T Task Group has completed so far in defining appropriate 802.11 test metrics and their accompanying test descriptions and environments.
| Throughput vs. path loss/conducted |
Measurement performed on clients, APs or systems of clients and APs to determine throughput of the DUT/SUT as a function of path loss |
conducted |
| Throughput vs. path loss and range/over the air |
Measurement performed on clients, APs or systems of clients and APs to determine throughput of the DUT/SUT as a function of path loss or distance |
over the air |
| BSS transition |
Measurement performed on clients, APs or systems of clients and APs to determine roaming time as client moves from one AP to another |
conducted |
| Fast BSS transition |
Measurement performed on clients, APs or systems of clients and APs to determine roaming time as client moves from one AP to another and roams according to the emerging 802.11r fast BSS transition specification |
conducted |
| Antenna diversity |
Measurement performed on clients or APs to determine the performance of the antenna diversity algorithm used to optimize throughput |
conducted |
| Adjacent channel interference |
Measurement performed on clients, APs or a system of clients and APs to determine the throughput performance in the presence of adjacent channel interference |
conducted |
| Receiver sensitivity |
Measurement performed on clients or APs to determine frame error rate vs. signal level |
conducted |
| AP capacity |
Measurement performed on APs to determine the maximum number of clients an AP can associate |
conducted |
| AP association performance |
Measurement performed on APs to determine the maximum rate of association/authentication operations |
conducted |
Table 1: Metrics currently specified in the 802.11T draft document
Conclusion
Without proper testing, wireless equipment and networks cannot be relied on to deliver the required throughput, client capacity, security or fault tolerance for acceptable enterprise network performance. 802.11T will provide the wireless networking industry with a solid set of common and accepted benchmarks for testing Wi-Fi—standardized metrics, measurement methods and test conditions.
Testing in accordance with the 802.11T specifications will provide end users with an objective means of evaluating functionality and performance of 802.11 products. 802.11T will help ensure that 802.11 products meet the challenges and demands of enterprise networks, and will help enterprise IT managers choose faster and more robust 802.11 products that are more resilient to adverse network conditions.
This article has appeared in the following publications:
