February 15, 2014

Benchmark Internet speed in a moving train

Mobile networks are wildly different from fixed line networks. No mobile network is perfect. Marketing talk and coverage maps just create an optimistic illusion around mediocre products. Only field benchmarks can accurately describe every network. Benchmarking mobile networks is quite tricky though.

Mobile network performance depends on location. Where do we measure it? Benchmarking at the point of sale is silly, of course. No network operator is so stupid to open their contact point inside of a blind spot of their network.

People usually travel over the same route regularly. Not only it makes sense to benchmark network speed along that route. It makes sense to avoid benchmarks everywhere else since those would be just an irrelevant distraction. The benchmarks are specific to your situation and unless lots of other people travel along the same route, it's up to you to perform the benchmarks.

So how do we perform the benchmarks? We need to perform lots of individual benchmarks, becuase network performance varies wildly as you move from one radio mast to another. We need something automated. I've looked around and found testmy.net.

It hosts several types of benchmarks. The one that's interesting for us is the auto benchmark, which will take measurements at regular intervals. It requires login, but that's worth the trouble. We will set it to short intervals, say 5 minutes. We also need to turn off the "express" mode, because that one is designed for stable fixed line networks.

The free version is limited to 25 consecutive tests, but we can easily workaround that by restarting the test after 2 hours. The test will eat some bandwidth, easily over 100 MB. That's acceptable considering that we are going to do the benchmark only once.

The website will be collecting the tests in the background. It is worth checking it time to time though since it can get stuck if connectivity to the mobile network is lost entirely, which happens all the time when travelling.

Once the benchmarks are collected, they can be viewed in tabular and chart form and they can be exported in CSV format for offline analysis.

One caveat though. The method this site uses optically improves the results. They are probing the network for 15-30 seconds and then measuring speed for 7 seconds. The trick here is that the slower the network, the longer the site spends probing the network. In effect, it ends up waiting for better network conditions.

I have used the software to measure speed of mobile networks in Slovakia, particularly the free WiFi in trains and Orange mobile internet. I've also benchmarked Slovak O2, a benchmark that merely proved that O2 3G network doesn't exist outside of cities.

No comments:

Post a Comment