The Energy Information Administration (EIA) produces projections of energy supply and demand each year in the Annual Energy Outlook (AEO).  The projections in the AEO are not statements of what will happen but of what might happen, given the assumptions and methodologies used. The projections are business-as-usual trend projections, given known technology, technological and demographic trends, and current laws and regulations.  The potential impacts of pending or proposed legislation, regulations, and standards—or of sections of legislation that have been enacted but that require implementing regulations or appropriation of funds that are not provided or specified in the legislation itself—are not reflected in the projections, although there are a few exceptions.  It is assumed that current laws and regulations that have sunset dates, but which are regularly renewed, are extended for modeling purposes.  Thus, the AEO generally provides a policy-neutral reference case that can be used to analyze policy initiatives. While the analyses in the AEO focus primarily on a reference case, lower and higher economic growth cases, and lower and higher energy price cases; more than 30 alternative cases are generally included in the AEO. Readers are encouraged to review the full range of cases, which address many of the uncertainties inherent in long-term projections.

Each year since 1996, EIA’s Office of Integrated Analysis and Forecasting has produced a comparison between realized energy outcomes and the projections included in previous editions of the AEO. Each year, the comparison adds the projections from the most recent AEO and updates the historical data to the most recently available.  The comparison summarizes the relationship of the AEO reference case projections since 1982 to realized outcomes by calculating the average absolute percent differences for several of the major variables for AEO1982 through AEO2006.¹ The average absolute percent difference is the simple mean of the absolute values of the percentage difference between the reference case projection and the actual value. The historical data are typically taken from the Annual Energy Review (AER).² The last column of Table 2 provides a summary of the most recent average absolute percent differences for 17 of the most important projection components. The detailed calculation of these differences is shown in Tables 3 through 19. These tables also provide the average absolute difference, which is the simple mean of the absolute value of the difference between the reference case projection and the actual value.  The calculated absolute average differences can change from one year’s evaluation to the next as an additional year of data and projections are added to the evaluation series and also because of data revisions in the AER and the Monthly Energy Review (MER).

This evaluation departs somewhat from previous editions by changing the order of the tables to one that better reflects the linkages in the AEO modeling process.  Specifically, after the summary table, the economic growth and world oil price tables are presented first because these projections affect all the other projections.

Beginning with the AEO2004 evaluation,³ the basis of measure for comparing dollar denominated values, Gross Domestic Product (GDP), and energy prices was changed.  The GDP series is now evaluated on a nominal dollar basis using the contemporaneous deflator series from each AEO rather than a current deflator series (See box, Revisions to Gross Domestic Product and Implications for the Comparisons). Each AEO presents a “real” GDP measure and a matched price index.  Prior to the AEO2004 evaluation, the calculation of nominal GDP was done by applying the latest official historical deflator series to the “real” GDP projection to generate a nominal series for the evaluation. This process has been revised to use the values of the price index from each AEO document to inflate the “real” numbers to a nominal series. These revisions also change the energy intensity table, which is presented as energy consumption divided by nominal dollar GDP. These changes improve the consistency of comparing across AEOs.

The growth in GDP is a good measure of the growth of the aggregate economy over time.  However, other concepts like disposable income, industrial output, vehicle sales, expansion of commercial floorspace, to name a few, become the actual drivers to the energy projections. GDP is perhaps the single best summary metric to show how the aggregate economy grows; and the assessment of how projections of GDP compare to history is a good proxy for how projections of economic activity in general influence energy markets.

As indicated in Table 2, the reference case projections of energy consumption, energy production, and carbon dioxide emissions have been relatively close to realized outcomes, the projections of net energy imports have been moderately close to realized outcomes, and the reference case projections of energy prices have been the furthest from realized outcomes.  Both Table 2 and the individual tables show marked improvement in the absolute average differences for energy prices and net energy imports over time.

• The AEO1985 through AEO1990 projections of GDP overestimated GDP growth in the 1990s, whereas AEO1991 to AEO2000 underestimated GDP growth. The economic slowdown that began in 2001 led to an overestimate of GDP in AEO2001 through AEO2003. Because GDP is a good indicator of the economic activity, which drives energy consumption, the differences between projected energy consumption and actual consumption is often similar to the differences between the GDP projections and actual GDP (Table 3).

• Overestimation of world oil prices, particularly in publications prior to AEO1997 (Table 4), resulted in underestimation of petroleum consumption. The prices in the AEOs completed after 1997 tended to be underestimated, which led to overestimation of petroleum consumption (Table 5).

• The fuel with the largest difference between the projections and actual data has generally been natural gas. As regulatory reforms that increased the role of competitive markets were implemented in the mid-1980s, the behavior of natural gas was especially difficult to predict. The technological improvement expectations embedded in early AEOs proved conservative and advances that made petroleum and natural gas less costly to produce were missed. After natural gas curtailments that artificially constrained natural gas use were eased in the mid-1980s, natural gas was an increasingly attractive fuel source, particularly for electricity generation and industrial uses. Historically, natural gas price instability was strongly influenced by natural gas resource estimates, which steadily rose, and by the world oil price. More recently, the AEO reference case has overestimated natural gas consumption (Table 9) due to the use of natural gas wellhead price projections that proved to be significantly lower than what actually occurred (Table 8).

• Coal prices to the electric power sector were almost always overestimated prior to AEO1999 and underestimated thereafter (Table 12).  In general, the AEO coal projections produced prior to the use of NEMS (AEO1982 through AEO1993) did not explicitly model coal mining productivity.  From 1985 through 2000, coal mining productivity improved by an average of 6.4 percent per year, reducing the cost of production, and resulting in lower coal prices.  As a result, there was a tendency for pre-NEMS coal models to overestimate future coal prices.  An additional factor, contributing to the overestimation of delivered coal prices in earlier AEOs was a sharp decline in coal transportation rates that began in the mid-1980’s and continued through the 1990’s.  For the AEOs produced using NEMS (starting with AEO1994), coal mining labor productivity is explicitly modeled.  However, the rather sudden switch from steadily increasing coal mining productivity during the 1980’s and 1990’s to a flat to declining rate starting around 2000 and continuing though 2005 was not anticipated in most of the AEO projections generated using NEMS.  As a result, there has been a recent tendency to underestimate coal prices especially post-2002. 

• For 2001 and beyond there is an initial pattern of underestimation of coal consumption (Table 13) then a series of AEOs with overestimation.  This is consistent with the pattern of total electricity sales, the sector that accounts for the majority of U.S. coal consumption.

• Since AEO2001, coal production (Table 14) is overestimated in nearly all AEOs.  For AEO1991 through AEO2002, there was a tendency to overestimate coal exports and underestimate coal imports, both of which contributed to an overestimation of U.S. coal production.  From the AEO2003 through AEO2006, projections of U.S. coal production have aligned fairly well with actual production levels.  Another confounding factor regarding projections of U.S. coal production is the mostly unpredictable pattern of annual coal stock withdrawals and builds.  For example, a 38 million ton build-up of coal stockpiles in 2001 resulted in a higher production number than would have been projected in the AEO projections, contributing to an underestimation of coal production for 2001 in several AEOs. This result follows from the general AEO assumption that the supply and demand for all fuels will balance for all projection years not calibrated to EIA’s Short Term Energy Outlook projection.  Historically, other notable changes in coal stockpiles include stock drawdowns of 44 million tons and 41 million tons in 1993 (a strike year) and 2000, respectively.

• Electricity prices have generally been overestimated in the AEO projections through the 1990’s (Table 15). Electricity prices in the early AEOs assumed regulated, average cost pricing, where fuel costs make up roughly 40 to 50 percent of the total price. As discussed above, coal prices to electric generators were often overestimated in these AEOs, resulting in similar overestimation of electricity prices. In the more recent AEOs, electricity prices have been underestimated, again following the pattern in the coal price projections. Also, in more recent years, electric generators have become more dependent on natural gas and the price underestimation in recent AEOs coincides with natural gas price spikes that were not predicted by the AEO.

• The level of future electricity sales was underestimated for nearly all projection years for the AEO 1991 through AEO1997 (Table 16). Since about 90 percent of the demand for coal results from electricity generation, the underestimation of electricity sales contributes to underestimation of coal consumption in those years (Table 10). The underestimation of electricity sales was particularly large in AEO1994 through AEO1996 and coal demand in those AEOs exhibit similar patterns of underestimation.

• Over the last two decades, there have been changes in laws, policies, and regulations that were not assumed in the projections prior to their implementation because of EIA’s statutory requirement to be policy neutral. Many of these actions have had significant impacts on energy supply, demand, and prices. For example, the Powerplant and Industrial Fuel Act (FUA) of 1978 restricted the use of natural gas in power plants and industrial boilers. After FUA was repealed in 1987, use of natural gas for electric generation and industrial processing increased sharply. Consequently, those AEOs completed prior to or immediately after repeal of the FUA, e.g., AEO1986, AEO1987, and AEO1989, underestimated natural gas consumption in 2000 by considerably more than AEOs completed in more recent years.  AEOs completed after the year 2000 have tended to overestimate natural gas consumption.  The overestimation since 2000 can be attributed to underestimation of natural gas prices in those projections.

• Technological improvements in both the production and use of energy have had a significant impact on the price, supply, and consumption of energy. Earlier AEOs typically assumed much slower technology development than actually occurred. This trend was recognized, in part, by this type of evaluation exercise. Beginning with the AEO1994, the projections were produced using the NEMS, which was designed to represent technology in a more detailed fashion.  This has lead to an improvement in the representation of technological change in the AEO. As NEMS has evolved, additional studies on technological improvement have led to more optimistic assumptions in the more recent projections.  Further, the adoption of modeling innovations, such as learning-by-doing, have allowed the model to better reflect the impact on cost of experience with new technologies as they are adopted.

• External factors such as severe weather, economic cycles, and other supply disruptions have also had an impact on the accuracy of the projections, particularly in the short term.  These types of events are not anticipated in a mid- to long-term projection like the AEO.

Table 1 provides a summary of the percentage of overestimates as well as the absolute projection differences for the entire Annual Energy Outlook series as well as for just the NEMS AEOs by Table.  The percentage of overestimates is calculated as the number of overestimates relative to the total number of projections made (i.e., for each AEO and each year projected).  The absolute average differences are smaller in magnitude for the NEMS AEOs in all but 5 of the 17 tables.  The percentage of overestimates in the NEMS AEOs has improved (i.e., moved closer to 50%) for 10 of the 17 concepts shown in Tables 3 through 19.  As discussed in the box below, the Gross Domestic Product (GDP) concept has changed noticeably over time due to revisions and improvements by the Bureau of Economic Analysis.  These changes distort comparisons of realized GDP outcomes and projections.  Further, since the energy intensity measure uses GDP in the denominator, comparisons of actual energy intensity and projections are also distorted to an unknown degree.  Consequently, the comparisons for GDP and energy intensity do not track the actual data as well as other concepts.

* Formerly titled Annual Energy Outlook Evaluation.
Contact: Steven.Wade@eia.doe.gov .
1 Note the publication gap in the tables that follow, there was no publication of AEO1988.
2 GDP and the GDP price deflators come from the Bureau of Economic Analysis, while coal prices to electric generating plants are from the Monthly Energy Review (MER).
3 Annual Energy Outlook Forecast Evaluation 2004