Measurement methodology

Overview

The speed test algorithm has been developed and improved over time so that it provides accurate and consistent results across all devices, platforms and operating systems. It is important to state that the same algorithm is used in all cases but with different implementations as required by each platform.

The process involves:


Ping Measurement

Connections are made to at least 3 of our servers closest to the device and preliminary tests are made to select the most suitable Measurement Server to give the most accurate results. The choice of server is dependent on a number of factors including response times, geo-location and network type of the device.
The geo-location and network type is provided directly from the device’s operating system.
The process for choosing the correct Measurement Server is managed by the Speedtest controller and the result is passed to the application. An accurate ping test is also used to provide the user with ping response data.

How We Measure

Our Speedchecker uses the HTTP protocol to determine the ping time. This has the following advantages:

See “Why do we use HTTP instead of ICMP” (below) for a technical explanation of these advantages and "Differences in the Results" for how the results are affected.
We use XHR ‘ping’ which requires no special server technology and produces slower results than ICMP because we are measuring times to transfer larger amounts of data including opening a socket before measuring latency. This resembles the normal behaviour of a web browser, while using ICMP will produce faster results.
Our method provides response times that are relevant to practical web applications and does not require special server technology.


Alternatively, we offer a different method for calculating latency that delivers lower values closer to the ones measured using other methods. It consists of stripping down the overhead generated from handshaking while opening a TCP connection and only measuring the time taken to send the HTTP request and receiving the requested test object. Since this method calculates latency concentrating on the already open TCP connection, it delivers a good approximation to results obtained with ICMP.


Why do we use HTTP instead of ICMP?

Ping is the name of the tool measuring latency using the ICMP protocol, which can be useful for network diagnostics and many latency related issues. It has its own protocol and its header structure is solely oriented as a diagnostic tool. Using it has many advantages especially when diagnosing network connections. It is classified as a layer 3 protocol and is excellent in measuring and diagnosing at that level. It can be somewhat useful for measuring applications, but one has to take into account that it measures in the network layer. Another disadvantage is that it gets typically blocked in servers because of its abuse potential and also disabling it doesn't sacrifice web services for example.
A very good argument for using HTTP in our measurements is that, unlike ICMP, it's the same protocol used in web applications and therefore we are measuring the whole protocol stack, from layer 7 including everything else to the bottom of the stack. For real world applications it is much more useful because it will reflect the latency experienced by exactly the same process that will affect web development decisions. For example, HTTP is subject to problems in TCP connections and ICMP isn't. In other words, a server can ping very well, but because of other problems in higher layers, the application can still have problems that happen somewhere else and you can't see this just by pinging.


Differences in the Results

A ping test result (using ICMP) may show faster times than HTTP. These differences can be significant and are often a cause for concern and confusion when comparing results between different speed tests. Our SpeedTest and other testing products (such as CloudPerf) are focused on measuring the performance of the Internet as perceived by the user. Using HTTP for ping tests ensures that the results are what the user will experience in his actual browser rather than at the low-level network communication layer.


Selecting between Websocket and HTTP Fallback

Websocket is not supported on all devices or servers. By default, we maintain an up-to-date list of all servers capabilities and if the best server for a given user does not support Websocket, we perform testing on HTTP fallback. In some cases, e.g. CDN POPs do not offer websocket connections and therefore HTTP fallback is the only way to test against particular CDN.
All the modern browsers support Websockets, as you can see on this compatibility chart: https://caniuse.com/#feat=websockets
If the user browser is out of date and does not support Websockets, the test will run on HTTP fallback.


Websocket measurement

Websocket protocol is widely supported by most of the modern browsers and allows us to test more accurately by leveraging full duplex communication between server and client. Unlike in HTTP protocol Websocket server is able to push data to the client without a prior request from the client. This is especially useful in upload measurement. Moreover, data can be sent on open Websocket (and therefore TCP connection) to lower the overhead of establishing connections and introducing more noise to bandwidth measurements.
Our websocket measurement (both for download and upload) uses by default port 8088 (since port 80 is busy with HTTP fallback) and establishes several TCP connections between client (user) and server (best server established during Ping measurement phase)
Client controls the measurement phase by selecting optimal number of websocket connections. Once connections are established (default 4), it sends initial chunks of data (size between 4kB and 1MB) to establish approximate client connection speed. If connection speed is not saturated fully, client will increase number of websocket connections (up to 8) to increase the saturation and correctly determine maximum possible connection speed. The test runs for variable period of time (default 20 seconds).
To avoid short bursts influencing final measured speed, our algorithm removes top/bottom of the samples to ensure that the most representative samples are included in average speed calculation. Websocket is used both for download and upload measurement with only difference of sending data in different directions.


HTTP fallback testing

A few small files are downloaded from the chosen Measurement Server (established in the Ping phase) to quickly estimate the speed of the connection. The results of this test are used to estimate the optimum file size for the final speed test file.
Based on extensive lab testing we realized that splitting the download into 4 equal parts gives the most accurate real-life results. Using just one large file for testing does not accurately reflect real life results but splitting the file into too many parts is equally inaccurate. The download files are already stored in all the possible sizes on each Measurement Server (in 4 equal parts) ready for download.
These 4-part files are requested by our application via 4 separate threads. The download requests and the servicing of these downloads are carried out using the device’s own HTTP Protocol. Our application monitors each of these downloads via the 4 threads (see diagram) and uses the progress to generate download profiles. The file needs to be big enough to take more than 2 seconds to download to ensure the accurate results
It is during the downloading that the algorithm is busy refining the estimate to ensure the most accurate results. This refinement includes, but is not limited to:


Upload Speed Test

The algorithm works the same way for testing Upload speeds as it does for download tests with the following exceptions: