...or why unrepresentative data is not reliable.

If you don't have a TomTom device in your vehicle then you may be surprised that if you live in one of the places recently researched by Atkins for the their Evaluation of 20mph limits then it doesn't matter how much you keep within the speed limit your considerate driving was completely ignored. And you wouldn't be alone. The report estimated that the actual number of drivers who were also ignored were in the 97% of vehicles who's speed they never measured.

Atkins and DfT decided that in order to gather information on vehicle speeds in 20mph streets in 12 case study 20mph areas and compare them with 3 other 30mph areas then they would use data captured from high end TomTom SatNavs or vehicles with in-built TomTom in-car devices. This is termed Floating Car Data (FCD).

You may therefore wonder just how accurate and representative their statistics and research were if it only included 3% of vehicles. We did as well, so we did some of our own research.

The report actually pointed out some problems which they had identified. These included only measuring an estimated 3% of vehicles but also the following on page 24 of the Technical Report:-

The first issue considers the bias which may have been introduced because only those with TomTom GPS in their car were measured. Could this subset of drivers have characteristics in their driving which meant they were unrepresentative of all the drivers using the roads? Could this in turn have skewed the results if a small percent of drivers with particular driving habits were predominantly the ones used. The question to ask is what is the profile of drivers who would have high-end TomToms or in-car devices. And let's remember that the data used goes back to 24 months before implementation of 20mph limits in the case studies and comparator areas. That is 2008 for the earliest schemes. Who was using high end TomToms 10 years ago? 

We suggest that the primary users would be :-

• Business users for trips to unfamiliar places
• Drivers who welcomed the dynamic identification and notification of speed cameras
• Delivery drivers
• Luxury cars

We make no judgement as to whether the above drivers would be more or less likely to comply with a 20mph speed limit, especially if they thought there was no enforcement and were being warned repeatedly of enforcement on 30mph roads. However, you may wish to make a judgement.

Even if there were the more usual cross-section of leisure, commuting, school run drivers also contributing to the FCD, we would suggest that for most local trips the the SatNav would be switched off as not required.

This is echoed by TomTom themselves. In a presentation in 2012 Nick Cohn and Hugo Bischoff of TomTom presented a paper on "Floating car data for transportation planning" which included the slide :-

We are unsure of what the acronym MON 2006 means, but the slide with the red annotations (on the original slide) do show that TomTom is over represented in drivers below 55, males, couples without children and working drivers. You may well consider whether the driving habits of these are different from the under-represented drivers. 

In addition, the presentation also went on to consider the frequency of usage of TomTom devices for different purposes.

For commuter traffic and shopping trips TomToms are only used regularly on about 10% of trips. It is for other trips that TomToms are used more regularly with business trips being one of the highest regular percentages. Once again the red arrows were on the original slide to show the bias.

From this it, we believe that it is highly likely that :-

• The data gathered is biased towards content from business, luxury cars, delivery vehicles and perhaps those seeking reassurance of the location of speed cameras. Such vehicles may be unrepresentative and the data be biased to faster drivers.
• The data gathered is biased toward content from drivers who are  under 55, male , married couples without children and drivers at work. If there is a tendency for such drivers to be in more of a hurry then the data will be biased to faster drivers.
• The data gathered is biased towards content from business drivers who may well have the need to reduce journey times as much as possible. The data will be biased to faster drivers.

We note that the final slide of the presentation says "When delivering to businesses and governments, be clear about potential bias in end results".

Whilst we cannot say with any quantitative certainty that the results were biased, we would suggest that the high leverage of the 3% segment samples indicates that their can be little reliance on the data unless the potential bias is eliminated from the results. We note that there has been no indication of this elimination being conducted in the review.

We also note that a decision has been made to use median speeds rather than the mean speeds which have been recognised since 2006 by DfT as the appropriate speed measurement to use when setting speed limits. Median speeds are not a recommended statistic to use. We note that the Technical Report on page 25 states :-

We find it concerning that a review that was to identify whether vehicles had slowed down after the implementation of a 20mph speed limit chooses to use a non-standard statistic that "helps to dampen the impact of slow-moving vehicles".

The combination of the low segment samples, the inherent bias in the use of Floating Car Data and the switch to non-standard measurement methods that dampen the impact of slower vehicles leads us to believe that  this review used data that was available and relatively cheap to capture rather than what was useful.

Whilst we do not claim any intent to mislead in the above by the authors of the Evaluation Review, we do not believe that this represents good practice in either the selection of data used, its low segment samples or the absence of any quantitative appraisal of the possibility of bias in the results.

Regardless of how well the data may have been analysed, a fact of data acquisition and analysis is that gathering data that is too small, too unrepresentative of the total population and subject to bias is neither comprehensive or sophisticated. Unless these issues are resolved and the questions answered then we cannot be confident of the 20mph Research Study's conclusions on speed.

And if you were one of those drivers in the case study or other areas in the review who observe speed limits and don't have or don't always switch your TomTom on, then we are sorry that your driving was never recorded.