Measuring connection speed in browsers

I’ve said it before and I’ll say it again: browsers, especially mobile ones, should give information about the speed of the connection they’re on, so that we know whether to send the high-res images or lowsource ones.

Currently no browser gives any sort of information about connection speed — they don’t even have access to it. Still ... well, let’s say that it’s possible that this will change in the future. But we need to figure out exactly how the system should work.

One clarification: I’m not talking about connection type but about connection speed. Wifi can be slow in a busy coffeeshop to the extent that it’s actually faster to download web sites over 3G. I don’t care if someone is on wifi or LTE or 3G, I want the browser to tell me the actual speed.

Output

Units

For units we should stick to the standard: Mbps, Kbps, Bps, where the B is bits, and not bytes, as always with connection speed. So the HTTP header, JS property, and media query would contain something like 8Mbps.

More suggestions for units? Seems like a no-brainer to me, but maybe I’m overlooking something.

Calculation

Far and away the most complicated question is how we should calculate the connection speed. Simply reading out the connection speed at the moment the HTTP request is made, or the JavaScript property or media query is touched, is not the best way to go. After all, just in that millisecond the connection speed may be unexpectedly high or low, leading to a wrong reading and the sending of the wrong batch of images.

So all in all I favour something like taking the average connection speed over the last five minutes or so. Although better than a snapshot, it’s not perfect, either. What if the user just started browsing, and there is no reading for the speed in the previous five minutes?

Consistency

Suppose a user is in a train, and goes to a website on her smartphone. The current connection is excellent. The HTTP header reports this fact to the server, which decides on high-res images.

Then the train enters a tunnel with little connectivity. The connection speed drops, and loading slows to a crawl. When the style sheet finally arrives it contains a bandwidth media query to decide between high-res and lowsource background images.

If it does it gives false information. If it doesn’t you end up with high-res images but lowsource background images, which could be confusing.

How should we solve this conundrum? Or should we ignore it and assume the user understands that something went wrong due to the slow connection?

How much can we expect the user to understand of the whole connection problem? I mean, a bad connection is universally understood. Isn’t it?

Comments

1 Posted by Masklinn on 23 October 2012 | Permalink

Generally speaking, especially on mobile, I'd say I don't really care about bandwidth and would be much more interested in latency hints.

In any case, such a scheme would also need to decide how to handle link switches: what happens if you're on EDGE and manage to get a 3G or LTE cell? What if you reach your home or work and hop on WiFi? Would sites have to poll bandwidth/latency properties to be updated? How would the system even know to update them?

> How much can we expect the user to understand of the whole connection problem? I mean, a bad connection is universally understood. Isn’t it?

I guess, at least in terms of "damn it's slow again, stupid phone, you're only going to space and back" and then spam the reload button or whatever.

2 Posted by Orde Saunders on 23 October 2012 | Permalink

I've been experimenting with performance.timing and navigator.connection in JavaScript to estimate the connection speed, coupled with sessionStorage to try and smooth out over a number of loads.

As part of this I've done some research into connection timing: http://blog.decadecity.net/2012/09/15/how-long-does-an-http-request-take/

The intention is to tie this into a JS image replacement script with a number of other signals such as screen width and devicePixelRatio to try and pick a suitable replacement.

Having this information in headers sent to the server to get it right in the first instance would be much more useful than having to do it all client side and having the device work out the recent history would be even better.

3 Posted by Artemy Tregubenko on 23 October 2012 | Permalink

> Mbps, Kbps, Bps, where the B is bits, and not bytes

To be consistent with International System of Units it should be "bps" not "Bps" and "kbps", not "Kbps". Uppercase "B" is for bytes, lowercase "b" is for bits. "Kilo" multiplier is lowercase. So the units would be Mbps, kbps, bps

4 Posted by Andreas Gohr on 23 October 2012 | Permalink

I don't think the header should contain any units at all but should aways use the same unit. Eg. bits per second.

I also wonder how a webbrowers would know the connection speed. Connection to what? To the website it is requesting? In that case the speed can only be determined *after* the first request. For reliable speed measurement a certain amount of data would needed to be downloaded first. If you use a different website/server to determine (eg. a browser vendor server) it might differ highly from the actual connection speed.

5 Posted by Brant Gurganus on 23 October 2012 | Permalink

Units should use the binary prefixes: MiBps. They are a bit less common, but the decimal prefixes have long been conflated with the binary meaning. With the binary prefixes, there's no uncertainty whether the 'kilo' is 1000 or 1024 since 'kibi' is definitely 1024.

6 Posted by Dominykas on 23 October 2012 | Permalink

I am not sure this "bandwidth" media query/hinting is a good solution.

I'm using a portable 3G WiFi hotspot and I will need to last 2 weeks on 6Gb. So, connection type goes right out the window - as has been said. Bandwidth is alright, but I honestly don't want the website to decide to serve me the large fancy graphics.

Ideally, we should have some sort of size hinting (yes, yes, extra HTTP requests as it stands right now), and let the user agent (and more importantly - the user) decide which asset to download. This probably is just a complication on the "Calculation" part of the problem - how should the cutoff points be defined?

I guess what I'm saying is that "connection speed" is a bad metric - as is also evident by the "train in the tunnel" example. I'd much more prefer the UA to download the simple version, and then, only if enabled AND capable, continue to download the remaining/better pixels (for 2x dpi), rather than block the rendering.

7 Posted by David Paul Ellenwood on 23 October 2012 | Permalink

While this is a lovely ideal, I think auto-sensing such information (at least initially) will be virtually impossible, for the reasons already mentioned and possibly others we haven't touched on.

That being said, it would be realitively easy to follow the example of YouTube and other video hosts where users can select the version that best work for them (480, 720, 1080, etc.) Maybe a browser setting to adjust the HTTP header value? This allows us a place to start (a default) an also allows users the flexibility to wait for a higher-quality version on a slower connection and vise-versa as they wish.

Piggy backing on a default setting like this, the browser could then fine-tune the the header value as more of a specific page/site is downloaded over the course of a browsing session (if the browser setting allowed it). Again, much like the way YouTube will start videos at a lower-quality and then bump them up as more quality is downloaded.

I agree with other who have said it would be wise to avoid using a unit as part of the header value. Just stick to a universal unit (bits) and leave the value as an integer.

8 Posted by Sérgio Lopes on 23 October 2012 | Permalink

I'm against bandwidth media queries, http headers etc.

If you want to switch to low-res images on slow networks, you're assuming your user values speed more than visual quality. And if you think your user values speed more than visual, you should never use retina graphics, whatever the network speed is.

My point is that network speed says nothing about your user and his preferences. I don't think you should reconfigure your site based on network speed. We never did that before, including 10 years ago when many people had dial-up connections. I don't feel like we need this now...

9 Posted by Andy Davies on 23 October 2012 | Permalink

Oh here's a ball of fun... but first the easy bits...

Ilya Grigorik has proposed something similar in browser-hints - https://docs.google.com/document/d/1xCtGvPbvVLacg45MWdAlLBnuWa7sJM1cEk1lI6nv--c/edit#

And the W3C net-info API may well be able to provide the values for bandwidth.

The real challenge is how is the available bandwidth calculated

Throughput is a mixture of available TCP connection time, bandwidth,
latency and packet loss, etc.

In theory you could measure it from the browser but there are a number
of issues, here are some examples:

A typical webpage is made up of many components, which are retrieved via short, bursty conversations between the browser and server.

The initial TCP connections go though a 'slow-start' phase while the client and server determine what's the optimal number of packets that
can be sent without being acknowledged. The number of packets inflight and the latency effective set a cap on the throughput. So measuring the resources that are downloaded first would probably under report the available throughput

Multiple (sub-) domains confuse things further...

Assume you can effectively measure throughput from the first resources
to be downloaded (HTML, CSS etc), what happens if the images are on a
different domain e.g. a CDN? The throughput that's just been measured isn't applicable to the CDN's domain.

Caching further complicates things as you can't use anything that's in the cache to measure throughput (or can you?)

I know lots of issues, but can't quite see the answer right now...

10 Posted by Michael. on 23 October 2012 | Permalink

@Brant
Except that bits per second is never binary, and so using the binary prefixes is just wrong.

Personally, I would suggest that the device could tell the browser the speed of the connection, the browser could then send it. And, if in doubt, use a slower speed.

Oh, and don't forget about bandwidth caps, which I was just reading about today.

So how about instead of (or additionally) sending the speed, send a max amount allowed. Starting at 1 for really low image sizes please (I don't know what that would be), and going up infinitely.

And use SVG and CSS instead of bitmaps.

(See http://dbushell.com/2012/10/23/the-real-cost-of-retina/ for where some of the ideas in this comment came from.)

11 Posted by karl dubost on 23 October 2012 | Permalink

In case you had not seen it, related to your blog post.

http://dvcs.w3.org/hg/dap/raw-file/tip/network-api/Overview.html

12 Posted by Lee Kowalkowski on 24 October 2012 | Permalink

Why connection speed? That's not necessarily the limiting factor. A device could be crawling internally, it could have very few CPU cycles available or be low on memory - but still with a fantastic connection speed.

Or conversely, the server itself could be overloaded, and concurrent high-bandwidth connections, not helping.

You should only be interested in response times i.e. the bottom line. Unfortunately no contributing factor to response time is ever constant. So this does not allow you to use connection speed as the only variable to improve response time.

In fact, if you detect a higher connection speed and (ab)use it, you could cripple that device instead of helping it, placing your website out of reach to devices that would otherwise be fine.

If you want to be responsive to device performance, then you need to consider the performance as a whole, and not focus on any individual aspect.

13 Posted by Jos Shepherd on 24 October 2012 | Permalink

There are three reasons that I (personally) might prefer not to get low-res images when browsing a site:

a) I might be using a low-res device and would see no benefit from high-res graphics.
b) I might be on a low bandwidth connection and prefer speed to
c) I might be on a connection where bandwidth is expensive

3G and 4G networks with good connection speeds might be able to load graphic sufficiently quickly but you would still want to conserve your usage.

I think it is unrealistic to expect web-developers to weigh these factors against each other in order to decide which resources to deliver. I think that the browser should send a header specifying low-bandwidth, or full-fat. The browser can either give the user a choice, or try and be clever, setting it automatically.

14 Posted by don bowman on 25 October 2012 | Permalink

There is no real meaning to the 'last hop nominal connection speed', only for the current unidirectional end to end path speed.

I have a PC @ home with a 10G connection to a switch with a 1G connection to a modem with a 100Mbps connection to a cable network that is 30Mbps down, 1Mbps up... that goes through a CMTS that is 10GE connected etc.

This does not mean my bandwidth to any site is related to these, they may not be the choke point. And the choke point changes as a function of time (since the network is packet-based not circuit-based).

A better way to achieve your goal is to be adaptive, in the way that Netflix is. The TCP stack in each machine maintains a node-pair bandwidth-latency product... Expose that to the web server / web browser using an ioctl, and let them decide based on it. This has been in modern stacks for a while. It adjusts over time based on its congestion model and provides a decently accurate means.

So in the web server, in the code on a web page, make that available via PHP or whatever.

Then my laptop connected to my MiFi doesn't report 54Mbps for the 802.11g connection rather than the real throughput.

15 Posted by Owen Estherton on 26 October 2012 | Permalink

Perhaps updating the browsers dynamically update the User Agent string with info about the connection? (e.g. "3G", "LTE", "WiFi", "Ethernet", etc)

16 Posted by jive on 26 October 2012 | Permalink

How about a setting for lower bandwidth content in the browser, so if you have a plan that doesn't allow much it sets the X-Connection-Speed it sends to the server. Otherwise it would default to send what your current connection speed.

Also we need a No-JavaScript-Enabled media query and a No-Images-Enabled media query, I have JS and images disabled on my mobile phone, because its way too slow with all that turned on.