Whether your web site gets 10 or 10 million visitors per day, we have a web solution for you.
Please note: www.MyForecast.com is owned and operated by CustomWeather, Inc.
Results of Large-Scale Weather Forecast Accuracy Study of Major Internet Weather Forecast Providers
By Eric Floehr
This study was initiated due to the lack of data concerning the accuracy of weather forecasts.The purpose of the study was to determine how accurate weather forecasts really are, and how accuracy differs between weather forecast providers. The author also personally wanted to know who provided the most accurate forecasts. In speaking with weather forecast providers and curious individuals, it became apparent that there has been no comprehensive, on-going, large-scale accuracy analysis of Internet weather forecast providers. This study aims to be the largest on-going accuracy analysis of Internet weather forecast providers. This report details the results of the first six-months of this study.
Comparison Criteria Selection
High temperature accuracy was chosen as the basis for this study because of its numerical precision and consistent definition between providers. The only aspects of a general forecast that are expressed as numbers, which are easy for computers to work with, are the high and low temperature, precipitation probability, and precipitation amount. All other predictions, from cloud cover to wind levels, are usually expressed in human terms.
Precipitation probability and amount were removed from consideration as not all internet weather forecast providers supply that information in their summary forecasts as a numeric value. Low temperature was removed after careful consideration because it was found after the study began that different providers have different meanings for low temperature. Some providers consider a low for a given day to be the low that occurred that morning and previous night. Others consider the low for a given day to the overnight low for the coming evening. Finally, some National Weather Service climatological summary reports document the low that occurred in the 24-hour period from midnight to midnight. For these reasons, the high temperature forecast was determined to be the most accurate, unambiguous measure of a forecast’s accuracy.
The top twenty most populous metropolitan statistical areas (MSA) according to the 2000 census comprise a wide-range of climatological environments. Also, as centers of population, a large number of the United States population would be interested in forecasts for these areas. Thus, high temperatures from the major National Weather Service observationstation in each of the top twenty MSAs were compared against forecasts for the station or the zip code containing the tation. The city, station location, station identifier, and zip code for the twenty MSAs selected are shown in table 1.
|City||Station Location||Station Call Signal||Station Zip Code|
|New York||La Guardia||KLGA||11371|
|Los Angeles||Los Angeles International||KLAX||90045|
|Washington, D.C.||Reagan National||KDCA||20041|
|San Francisco||San Francisco International||KSFO||94128|
|Dallas/Ft. Worth||Dallas-Ft. Worth International||KDFW||75261|
|Phoenix||Sky Harbor International||KPHX||85034|
|Minneapolis||Minneapolis-St. Paul International||KMSP||55450|
|San Diego||Lindbergh Field||KSAN||92101|
|St. Louis||Lambert Field||KSTL||63145|
|Table 1: Cities selected for forecast accuracy study.|
Weather forecast providers were selected based on the following criteria. They must provide forecasts for all locations specified.They must provide that service free of charge. They must be able to be queried for the location’s forecast using either the station identifier or the zip code. They must present the high and low temperature forecast as an unambiguous value. And finally, the web page on which the forecast resides must be able to be easily parsed.
Not meeting this criteria were CNN (provided by Accuweather), Weather Underground, and Wx.com (Meteorologix). Accuweather, Intellicast, MyForecast, Unisys, Weather.com, and WeatherForYou met all criteria and were included in the study.
|Provider||Forecasted Highs Retrieved||Percent of Possible||Forecasted Highs Scored||Percent of Possible|
|Table 2: Forecasted highs retrieved and scored versus possible.|
Data Collection Methodology
Each night starting at 10 p.m. Eastern from January through June, forecasts were collected from the six major internet weather forecast providers that met all selection criteria. Intellicast and MyForecast use the station call sign as the forecast identifier. For the others, the zip code from the table above was used to retrieve the forecast. The next day, two-day-out, and three-day-out forecasts were collected. The daily climatological summary from the National Weather Service Climate Prediction Center’s Climate Operations Branch was used as the official observational record of each day.
Table 2 lists the number of forecasts retrieved for each provider from the period of January 1, 2003, to June 30, 2003. The table also lists the number of forecasts compared to actual observations. There were 181 days in the time period. There were 20 cities in the study, each with a 1-, 2-, and 3-day-out forecast for each day in the study. Therefore the maximum number of forecasts that could be retrieved for each internet weather forecast provider is 181 days times 3 forecasts per day, times 20 cities, or 10,860 individual forecasts.
There were a total of 181 days times 20 cities, or 3,620 observation records possible. Of that, 3,414 observations were actually collected or 94.31% of the total possible. Possible reasons why a forecast or observation could not be retrieved were network problems, site unavailability, or other technical problems. Unisys in particular was challenging as not all forecasts were filled in with the high and low temperatures. Even with all those challenges, on average over 85% of possible forecasts were collected and scored against actual observations, a total of 55,754 individual forecasts. This makes the study the largest accuracy study of internet weather forecast providers ever.
The high temperature forecast error was calculated by subtracting the high temperature forecast from the observed high temperature, and squaring the result.This was done for the one-, two-, and three-day-out forecasts.Each calculated error was then averaged to derive a single error for each provider.This result is called the root-mean-squared error, or RMS error. RMS error was used because it is an indicator of standard deviation as well as difference.With this approach, wide variation is penalized more than consistency with a few large errors. This approach has its root in the customer experience: a customer of a forecast would rather see forecasts that were mostly right most of the time than right-on sometimes and dead-wrong others.With RMS error, a lower number indicates greater forecast accuracy.
Additionally, the number of high temperature forecasts that had an absolute error (the absolute value of the difference between the forecast and the observation) within three degrees was calculated.This number was then divided by the total number of forecasts for the provider to derive the percentage of forecasts within three degrees of actual. This is a measure of the number of forecasts that could be considered “correct”.This measure was included as a measure of correctness, as the “three-degree guarantee” is becoming popular amongst television meteorologists.
Graph 1 and Graph 2 show the results of the study.The value axis of each graph has been exaggerated to show the differences between forecast providers.
Graph 1 shows the calculated root-mean-squared error by provider.MyForecast had the lowest error at 24.68, while Accuweather followed in a very close second at 24.83. Weather.com and Intellicast, owned by the same parent, not surprisingly show very similar errors at 26.84 and 26.97 respectively, over two error points behind MyForecast and Accuweather. Unisys and WeatherForYou trail. WeatherForYou was somewhat of a control, as it uses the commercially available HAMWeather product to parse the National Weather Service zone forecasts. The zone forecasts cover a wide area and do not provide a precise temperature forecast for that very reason.The HAMWeather product compensates by parsing a text temperature forecast into a numeric one, changing low 80’s to 81, for example.
Graph 2 shows the results of calculating the number of one-, two-, and three-day-out high temperature forecasts that fall within three degrees of the observed value. Again, MyForecast leads with just over 60% of its high temperature forecasts within three degrees.Again, Accuweather comes in second place with just under 60%.Intellicast and Weather.com are third and fourth.WeatherForYou and Unisys again trail.What is surprising is that despite the inherent inaccuracy of parsing the zone forecasts into a numeric temperature, WeatherForYou, using HAMWeather, still managed to stay in the pack, and didn’t have the lowest percentage of forecasts within three degrees.
This study is ongoing, and will last an entire year, to encompass the full range of seasons and weather.Additionally, once information from all weather forecast providers is gathered regarding the interpretation of the low temperature forecast, low temperature accuracy will be calculated. Finally, precipitation and sky cover (the “icon” or “short” forecast) accuracy will be measured in a future study.
About the Author
Eric Floehr is the author of this study.He graduated from The Ohio State University with a degree in Computer and Information Science, and has had a distinguished career in the computer industry, specializing in large-scale data collection and analysis. Eric designed and built the search and organize component of the Blaze! Web Performance Pack, which queried and aggregated search results from over a dozen internet search engines. The June 1997 issue of PC Computing said Blaze’s built-in parallel-search engine was both faster and more accurate than [products by ForeFront Group, Quarterdeck, and Symantec]. Eric most recently led a team at MCI to architect and build a system that collects over 150 million performance measurements per day from over 15,000 internet VPN devices located throughout the world. He has always been interested in weather. Under the five-year-old entry in his baby book, Eric’s mom wrote “He is very interested in dinosaurs, space, and weather.”
Eric started his first business in middle-school, selling a Blackjack program for the Commodore 64. Eric is the founder of Intellovations, LLC.Eric can be reached at email@example.com.
508 Grace Drive
Marysville, OH 43040
About Intellovations, LLC
Intellovations, LLC is a full-service technology product and service company specializing in data-collection, analytics, and scientific applications.Intellovations was founded by Eric Floehr, and his wife Tessa Floehr, who holds a Masters Degree in Computers in Education.Intellovations has offices in Marysville, Ohio, and can be reached at firstname.lastname@example.org. Intellovations sponsored this study.