These days, there's a lot of buzz in the industry touting mobile carriers' performance in various tests from a myriad of mobile testing outfits. However, the testing methods used by these analytics companies can vary greatly.
From when we first began assessing mobile network performance in 2008 to today, at RootMetrics by IHS Markit it's been our belief that a scientific, independent, and consumer-focused approach to testing yields the most accurate and actionable results.
That's why we want to help cut through the noise and clearly state our approach to evaluating network performance, especially with 5G networks rolling out.
Simple premise, sophisticated methodology
Independent with our methodology, comprehensive in our testing, and accurate with our results, we provide real-world insights that can help improve consumers' daily mobile experience. Our transparent, consumer-focused approach to evaluating network performance has changed the game—so much so that our data and insights are trusted by major mobile network operators, leading technology manufacturers and retailers, government agencies, and many others in the technology landscape across the globe.
To ensure that our results offer an accurate characterization of the consumer mobile experience, our testing methodology is guided by two fundamental standards:
- Testing mobile network performance from a consumer's point of view
- Providing objective, accurate, and unbiased assessments of performance in helpful and easy-to-understand terms
This may sound simple in theory, but there are numerous components below the surface that bring these standards to life in our testing and reporting.
Characterizing the consumer mobile experience
Our testing shows the impact of everything that can affect the consumer mobile experience end-to-end, from the network itself to the impact of device problems, routing or gateway issues, or problems encountered as networks switch from call to data service, among other things.
To truly characterize the consumer experience, we:
- Test with the same smartphones that consumers use
- Test the same mobile activities that consumers use their smartphones for on a daily basis
- Test at the same times that consumers use their smartphones
- Test in the same places where consumers use their smartphones
How and what we test
To accurately reflect a typical consumer's mobile experience, we use only off-the-shelf smartphones. We measure network reliability and speed performance across the activities for which consumers use their smartphones every day, like checking email, browsing webpages, using apps, making calls, and sending texts.
Our methodology is designed to ensure that our tests measure performance through a wide range of real-world situations. For example: we collect samples during periods of high and low congestion; we measure performance across variations in speed, from standing still to driving on the highway; and we perform tests whether coverage is poor or excellent or somewhere in between. We test each network head-to-head in these situations to make comparisons easy and to ensure that all networks are measured on a level playing field.
Test categories
Overall performance testing | A combination of reliability and speed results from data, call, and text testing |
Network reliability testing | A holistic look at reliability performance across data, call, and text testing |
Network speed testing | A holistic look at speed performance across data testing |
Data testing | The testing of reliability and speed during file uploads and downloads, when downloading email, and while downloading files that represent the loading of typical webpages and apps |
Call testing | The testing of how reliably each network can place and maintain calls, with call tests conducted by placing a call from a mobile phone to another mobile phone |
Text testing | The testing of how reliably and quickly consumers can send and receive text messages |
Where and when we test
To reflect consumers' varying mobile lives, we measure performance indoors as well as while driving and walking across multiple spaces.
When possible, the boundaries of the areas we test are defined by governments and official agencies—not by us. When official boundaries are not available—for example, in transit stations or event venues—we measure network performance in densely populated areas where consumers often use their smartphones.
We conduct tests nearly every day of the year to provide the most current view of mobile network performance. To ensure our test results are as fresh as possible, we test locations every six months.
We measure performance during the day and at night, and our sample collection periods are weighted more heavily toward typical consumer usage hours. Tests are performed each day of the week to establish a comprehensive, temporal view of performance. We also ensure that tests are performed in each location for long-enough periods to make sure that important, but rare, events are captured to accurately characterize a consumer's mobile experience.
In addition to our rigorous scientific testing, we take into consideration crowdsourced data collected from our CoverageMap app for the most complete picture of mobile performance available anywhere.
All about accuracy
We have a dedicated analytics department that ensures we've collected enough test samples to accurately characterize an area with statistical confidence. To prevent bias and to further ensure that our test results provide a statistically rigorous and accurate characterization of network performance, we also use randomized spatial sampling techniques to collect data.
Random sampling removes bias from a sample collection, which allows us to make objective, statistically accurate inferences about a larger area based on a smaller set of test results—whether we're testing in a metro area, an event venue, or a nation as whole. The random selection process also means that areas of each test location are tested at varying times. For example, by the time we've finished testing a metro area, we have a statistically valid, spatiotemporal view of network performance across the entire area.
Once the accuracy of the collected data is verified by our analytics team, they then transform the data into insights that help the industry improve while giving consumers an accurate picture of performance.
Continual improvement
To account for shifts in consumer behavior, technology, and the mobile industry, we regularly assess and review all our tests and methodologies with a high level of scrutiny and evolve them to provide an accurate and current measurement of the consumer mobile experience, from nation to neighborhood.
5G is here
5G networks will be rolled out in South Korea in May, with others soon to follow in the US, Switzerland, and the UK. And late summer through fall 2019, China, Japan, Australia, and India will undergo 5G deployments. We will be testing these 5G mobile networks and offering our assessments, along with market intelligence, in the 5G First Look subscription service. Feel free to contact us for more information.
And for more on what 5G is and how it is defined by IHS Markit, check out our white paper: The promise and potential of 5G: Evolution or revolution?
Kevin
Hasley is executive director for performance
benchmarking at IHS Markit
Posted 7 May 2019