Chapter 3.  End-Use Consumption Surveys:  Data Quality and Scope

EIA’s end-use consumption surveys were among the first fielded by EIA after its creation in 1977 and now cover energy end uses in three areas:  the Commercial Buildings Energy Consumption Survey (CBECS) covers the commercial buildings sector, the Residential Energy Consumption Survey (RECS) covers the occupied housing portion of the residential housing sector, and the Manufacturing Energy Consumption Survey (MECS) covers the manufacturing share of the industrial sector.³ These resource-intensive surveys are the only source of data for current estimates of energy end uses in homes and commercial buildings and for key industries within manufacturing.  Due to limited resources, they are not currently meeting their statutory requirements for frequency and scope⁴.

Two features limit the geographic level at which consumption data can be provided to the public:  1) sample sizes, which are driven by available resources and 2) data confidentiality laws, which require that EIA protect the identity of individual respondents and establishments.  In practice, where firms make up a large share of their industry class or building type, they could be identified at lower levels of aggregation.  However, EIA would be prohibited from publishing some statistics at the State level or would need to collapse disparate classes of data to protect its confidentiality.  The same legislation, however, does allow certified agents of EIA to make limited use of non-public files for statistical purposes only. 

Given current resources and confidentiality constraints, EIA is limited to publishing end-use data for the following geographic areas⁵:

• MECS:  National and Census Region
• CBECS:  National, Census Region, and Census Division
• RECS:  National, Census Region, Census Division, and the four most populous States (California, Florida, New York, and Texas)

Stakeholder Needs

Stakeholders expect EIA to take the lead in providing energy consumption data that meet the quality and scope necessary to monitor topics related to climate, the environment, and energy security and they often request data to help evaluate energy programs and policies that are often written, funded, and implemented at the State level.  They need more and new consumption data at lower levels of geography, more frequently, and with less lag time between the period of data collection and the release date.  A Federal statistical program that can assess the value of a dollar invested in a particular program, technology, or system is a much broader and more complex data operation than EIA has ever run.  Such a program would have profound resource implications for EIA.   

• EPA reports that it needs a larger sample size for CBECS to produce energy performance benchmarks for more building types.  EPA offers an online rating system called Portfolio Manager where commercial buildings can be rated for their energy consumption relative to similar buildings.  This tool has been used for more than 78,000 buildings; about 5,600 have achieved an EnergyStar rating.  Although this rating system has become the industry standard, EPA can only produce benchmarks for 10 broad building types.  An increase in the CBECS sample size would be needed to produce them for more diverse building types and principal activities.

• The American Society of Heating, Refrigerating and Air Conditioning Engineers (ASHRAE) states that the CBECS current sample size is too small to evaluate the impact of critical building labeling programs like EnergyStar by building type within climate zones.  They also believe that States and cities rely on CBECS information to develop rating systems for “comparable buildings.” Because data at that level are imprecise or absent, ASHRAE is concerned about the accuracy, utility, and impact of the ratings.

• The National Renewable Energy Laboratory (NREL) reports it needs much larger RECS and CBECS sample sizes to perform necessary multivariate analyses to understand the adoption rates and impact of the new technologies, building designs, and energy-efficient equipment they test and promote.

• The National Association of Home Builders (NAHB) indicates it needs larger samples to estimate how much consumption is explained by householder behavior versus that which the builder can control.  Although NAHB makes no direct request for State-level estimates, building codes are an important feature in that analysis because they are enacted at the State and local level.

• The American Council for an Energy-Efficient Economy (ACEEE) produces an annual State Energy Efficiency Scorecard, which ranks States according to their adoption and implementation of energy efficiency policies and programs.  Without sufficient State-level sample sizes in EIA’s consumption surveys, there is no accurate, direct link between State policies and consumer participation in these programs, which represent large, non-Federal program investments.

• The Department of Housing and Urban Development (HUD) and its local housing authorities use RECS data to calculate utility allowances for subsidized housing by States and localities.  State-level estimates would provide a direct source of data closer to what is needed for program management.

• Where utility costs are bundled with rent by landlords, the Bureau of Labor Statistics (BLS) uses RECS data to allocate portions of total rent to housing or utilities.  Rents and utility costs vary widely for smaller areas, so BLS would benefit from lower levels of data aggregation.

• The Department of Health and Human Services (HHS) uses RECS to support program needs of the Low Income Home Energy Assistance Program (LIHEAP).  Improved State-level estimates would provide marked improvement in allocating funds in this multi-billion dollar grant program to States.

• ACEEE notes that because EIA summarizes consumption data by Census Division, EIA necessarily aggregates statistics in the Mountain Division across a huge area—from Montana on the Canadian border to Arizona on the Mexican border.  Program evaluations are not served by combining data across such diverse States and climates.  Without additional sampling, EIA could increase breakouts in the Mountain Division only by decreasing the sample in the Pacific Division, which would reduce the accuracy of current estimates for California.

• Stakeholders want EIA to balance sample allocations between new and existing building stock for the RECS and CBECS to improve the accuracy of comparisons among and between buildings of different ages.  Although Federal or industry programs offer explicit guidance on energy-efficient building design and practices, program participation is often voluntary.  This situation is less true for State and local governments where codes can be tied to building permits.  Therefore, State-level data would provide more accurate analysis of the sources of differences in energy consumption for new construction.

• Numerous officials from a wide array of State program offices, consulting firms, utility companies, and equipment manufacturers contact EIA directly for data that are not available without large increases in survey sample sizes:  State-level energy intensities within consumption sectors including breakouts by industry, building type or activity, fuel, end use, and equipment type; and ‘typical’ intensities for specific buildings types, industry codes, or residential classes.

There is a recurring dilemma when trying to optimize sample allocations across such overlapping geographic variables as physical geography and climate zones.  A good sample can achieve sufficient coverage by States or by climate zones but not both without increasing the likelihood that some buildings or establishments could be individually identified.  For reasons of data confidentiality, EIA will probably need to continue to aggregate some data to mask the location of particular buildings.

Options for Improving Consumption Data Surveys

The consumption surveys draw samples from large, heterogeneous populations, and the survey methodologies necessary to produce accurate results mirror that complexity.  Therefore, for RECS and CBECS, EIA samples, lists, and enumerates units selected from area clusters at rates that have to balance sampling error across key characteristics:  geography, building size and type or activity, main fuel used for space heating, and so on.  Area-probability field studies are the most expensive but provide the best frame coverage for complex populations, the highest item and unit response rates, and the ability to collect key data in physical form (e.g., household square footage measurements and interviewer observations of buildings characteristics).  MECS is considerably less expensive per sampled unit because it is drawn from a known list of establishments with routine updates maintained by the Census Bureau and utilized for other economic data collections, and can be conducted by mail and the Internet. 

Options for improving the consumption survey data programs are described below.  The first options expand the current survey designs to improve data quality without State-level estimates, other than for the four most populous States in the RECS.  Programs with expanded survey designs would be a vast improvement over current programs because they would permit more complex analysis of key indicators of energy use, publication of more building types, and more accuracy for secondary uses of the data by other Federal agencies.  Short of a 50-State estimates program (or a 50-States and U.S. territories program), also described below, an intermediate option within each of these designs is to add some but not all States.  Proposals are also included that describe data collection needed to produce baseline measures for 50 States, although such a program would require considerably more resources.  Additional options focus on restoring and adding new end–use surveys, conducting feasibility studies to explore alternative methods to update end-use baselines, and decreasing the time between data collection and release. 

For each initiative or proposal, EIA estimates preliminary costs, which are presented in different ways depending on the initiative.  Costs are presented either in terms of increments to EIA’s 2009 budget request (for wholly new initiatives); increments to the projected budget for an upcoming survey cycle (in the case of initiatives that increase an ongoing survey’s sample size or data scope to improve data quality);  or relative per-cycle costs (for initiatives that address frequency of surveys).  More accurate budget numbers (both start-up costs and per-survey cycle costs) require knowing, inter alia, which States/Regions would be added, the homogeneity of key characteristics within them, and the level of accuracy desired.  

Enhance Data Quality by Adding Data Elements and Increasing Sample

There are many ways in which EIA could improve the value and accuracy of consumption surveys.  By far, the most important is to improve the accuracy of end-use estimates.  Managing the sample sizes and allocations among key characteristics is equally important as adding questions to capture new phenomena that effect consumption patterns and levels.  Changes in sample coverage below address two stakeholder needs:  1) produce sufficient sample size to be able to perform needed multivariate analyses (i.e., having the statistical power to understand the relative contribution of three or more characteristics) and 2) produce accurate State-level energy intensities and end-use estimates with which States can monitor and evaluate energy programs.

• Obtain More Geographic Detail on Fuel and Nonfuel Uses of Fossil Fuels for Manufacturing (MECS).  Doubling the quadrennial MECS sample size to 31,000 sample units would allow EIA to provide estimates for manufacturing energy consumption for industry groups by Census Region, improve the statistical accuracy of national analyses, estimate energy efficiency in this sector, and  calculate changes in carbon emissions over time that result from structural change.  This initiative would serve essential missions of many Federal, State and industry energy, environmental and commercial interests.  Initiative 3.1. Cost per 4-year Cycle: $2,800,000 (increment over EIA’s projected cost for 2010 MECS).

• Enhance the Quality of the RECS.  Increase the quadrennial RECS by 50 percent to 9,750 sample units and add questions to improve the accuracy of multivariate data analysis and end-use estimates.  For example, collect data on:  the share of remodeling that is done to incorporate energy efficient equipment, systems, and designs; the degree of compliance with building energy codes (and which version or source) for new construction; and building operation, because behavior can often explain more variation in consumption than can technology and equipment.  Conduct a periodic sub-metering study on a subsample of RECS households to measure actual energy use by refrigerators, hot water heaters, televisions, and computers and their peripherals.  Compare actual consumption by end use and behavior to estimates based on non-linear models developed by EIA staff.  Such a comparison would help EIA adjust for bias that might occur in an infrequent survey that lags in accounting for new technology, standards, and growing plug loads.  Initiative 3.2. Cost per 4-year Cycle:  $3,160,000 (increment over EIA’s projected cost for 2009 RECS).

• Enhance the Quality of the CBECS.  Increase the sample size for the quadrennial CBECS by 50 percent to 17,250 sample cases (which include buildings and establishments) and target specific building types that are big energy users, such as data centers, laboratories, convention centers, and arenas.  Add questions to improve the accuracy of multivariate data analysis and end-use estimates.  For example, EIA could collect new data on: the degree of compliance with building energy codes (and which version or source) for new construction; building operation, to isolate energy management practices from fixed factors; and building type, to explain more variation in energy consumption, e.g., linear shelf feet rather than number of refrigerators for groceries, size of the eating area for food service, volume of transactions/sales for retail, number of beds or rooms for dorms/hotels/hospitals, and number of service bays for auto repair shops.  EIA could conduct a quality study on a subsample of the CBECS buildings in detail and compare the end-use energy use of these buildings to the estimates developed by EIA non-linear models.  EIA could validate the information collected during the field interviews and also provide an independent estimate of the energy consumption by end use.  This information could lead to improvements in future CBECS questionnaires, as well as measuring the quality of current CBECS estimates.  Initiative 3.3. Cost per 4-year Cycle:  $6,880,000 (increment over EIA’s projected cost for 2011 CBECS). 

Restore and Add New Surveys to Fill Data Gaps

In addition to increasing the geographic scope of consumption surveys to provide a neutral source of data for Federal, State, and local energy policy, EIA could increase coverage across consumption sectors to improve forecasts of short- and long-term energy demand.  Data gaps are sometimes addressed by increasing sample sizes to permit publication of more subclasses, or by adding questionnaire items or surveys.  To increase the coverage and accuracy of the amount and sources of growing energy demand, particularly for electricity and petroleum-based fuels, EIA would need to restore and add data collections for transportation and agriculture.  The transportation sector is going through a dynamic era of technological, fuel, and industry change in response to energy prices and global economic and climate concerns.  Measuring transportation energy consumption poorly, infrequently, or not at all has implications across all sectors of the economy.  With the emergence of biofuels, energy and food policy are now more closely linked.

• Collect Data on End Uses of Energy by the Residential Transportation Sector.  Restore the Residential Transportation Energy Consumption Survey (RTECS), which was discontinued after the 1994 data year due to insufficient funds.  EIA could expand the current RECS to include a follow-on study of residential transportation end uses, which account for two-thirds of the entire transportation sector and most of motor gasoline consumption.  Many stakeholders value a reliable, policy-neutral source of data to understand on-road fuel economy, price elasticities, vehicle miles traveled, commuting behavior, and vehicle purchases relative to new energy policies and technologies.  Initiative 3.4. Start-up Cost:  $1,000,000; Cost per 4-year Cycle: $4,048,000 (increment over EIA’s 2009 budget request).

• Collect Data on the End Uses of Energy by the Non-Residential (Truck) Transportation Sector. To increase coverage of the transportation sector, EIA could sponsor, in part, the collection of data on the physical and operational characteristics of the Nation’s private and commercial truck populations.  Until 2002, the Census Bureau conducted a truck survey with its quinquennial economic census.  A similar truck survey, with additional questions about fuels used, energy end-uses, and costs would be beneficial.  The series would produce national and State-level estimates of the total number of trucks and their end uses.  Expanding the survey to the State level could require data coordination and cost-sharing with Federal, State, and local agencies.  These data could serve missions of Federal energy and transportation agencies by providing a comprehensive data set for assessing energy efficiency and the environmental impact of the Nation's truck fleet.  Cost estimates only assume EIA’s role adding fuel-related questions.  Initiative 3.5. $3,000,000 (increment over EIA’s 2009 budget request). 

• Collect Data on End Uses for the Agricultural Sector.  EIA currently collects data on industrial sector energy end uses only for the manufacturing portion.  Adding an agricultural survey would improve the industry coverage in EIA’s State Energy Profiles.  For example, fuel rates vary significantly by scale and type of operation, but EIA now assumes that commercial rates prevail.  An agricultural series would provide the only baseline to:  measure opportunities for new energy-efficient technologies and practices; allow a more uniform evaluation of the impact on production agriculture of Federal and State energy policies, such as fuel tax abatements, efficiency incentives, and alternative fuel use; and produce estimates of the share of greenhouse gases resulting from different enterprises, production practices, and technologies.  Two agencies would be served:  the Department of Agriculture (USDA) could measure the farm-level response to changes in energy prices and supply (food security), and the Department of Energy could study the flow of crops into biofuels compared with other uses (energy security).  Cost estimates only assume EIA’s role in a new agricultural survey.  Initiative 3.6. Start-up Cost:  $200,000; Cost per 4-year Cycle: $1,500,000 (increment over EIA’s 2009 budget request).

Produce End-Use Data for All 50 States

Developing and operating a 50-State consumption data program (or a 50-State and U.S. territories data program) would require significantly more resources than EIA’s current program and significantly more than any of the proposals discussed so far.  For example, a preliminary estimate for adding States to RECS would be $750,000 for each additional State.  For the CBECS, each additional State would add about $1,200,000 to the total survey budget.  State selection criteria would vary according to measurement goals, accuracy and confidentiality requirements, and costs relative to sampling efficiencies for other States.  For example, if States were selected according to population or building rank, coverage would quickly increase for one Division in the Midwest⁶ (Illinois, Ohio, and Michigan in the East North Central Division) and one in the South (Georgia and North Carolina in the South Atlantic Division).  Population or the number of buildings as the main selection criteria would yield no data improvements for the Mountain or West North Central Division States and might not meet the goals of States with aggressive energy policies, stakeholders such as the ACEEE, and many Federal agencies with little or no data to monitor programs.  The criteria for adding States would need to be carefully developed, with stakeholders’ input, and clearly communicated.

• Provide Residential Energy Consumption Data for All 50 States (maintaining current 4-year cycle).  Increasing the RECS sample to cover 50 States would ensure that new (often State) policies can be monitored for their impact on fuel type used, intensities, and end uses.  Accuracy would improve for national, regional, and division-level estimates so that analysts could isolate the effect of such factors as new efficiency standards, building technologies, and program participation from factors over which consumers have no control, such as weather.  EIA would have sufficient sample counts to produce estimates for small appliances and home electronics–a growing portion of residential consumption, a source of greenhouse gases, and an opportunity for technology change and innovation.  Expanding the program to include all States would serve essential missions of many Federal, State, and energy industry, environmental, and commercial interests.  Initiative 3.7. Start-up Cost: up to $8,000,000; Cost per 4-year Cycle: up to $26,460,000 (incremental cost over EIA’s projected cost for 2009 RECS). 

• Provide Residential Transportation Data for All 50 States (maintaining current 4-year cycle).  If residential transportation data were deemed necessary to collect via a follow-on survey to the 50-State RECS design, it would require additional funds beyond funds required for the 50-State RECS.  Initiative 3.8. Start-up Cost: $1,000,000; Cost per 4-year Cycle: $15,660,000 (incremental cost over EIA’s projected cost for 50-State RECS).  

• Use Current Population Survey to Collect State-Level Residential Transportation Data.  Some State-level residential transportation data could be collected by adding questions to the Current Population Survey (CPS), such as the vehicle identification number and the current odometer reading for all vehicles held by members of the sampled household.  The CPS, which is conducted by the Bureau of the Census for the Bureau of Labor Statistics, is a large monthly survey that would yield about 15,000 interviews per month.  Estimates would permit State-level estimates of fuel use by month for large States and 6- or 12-month averages for smaller States.  The data could potentially be combined with other social variables, such as household characteristics, employment, and income.  Initiative 3.9. Start-up Cost to EIA:  up to $4,000,000 (over 3 years); Cost per Annual Cycle: up to $900,000 (incremental cost relative to EIA’s 2009 budget request).  

• Provide CBECS Data for All 50 States (maintaining current 4-year cycle).  Increasing the CBECS sample would allow EIA to publish State estimates for some major building types and dramatically increase the number of types that could be published at the national, regional, and division level.  A larger sample would also allow EIA to undertake energy efficiency analysis; calculate changes in carbon emissions over time; and monitor the adoption of new building design, equipment technologies, and energy management tools and practices.  Initiative 3.10. Start-up Cost: up to $13,000,000; Cost per 4-year Cycle: up to $42,640,000 (incremental cost over EIA’s projected cost for 2011 CBECS).

Increase Frequency of Energy End-Use Data

• Collect End-Use Data More Frequently (maintaining current sample sizes).  Note that per-cycle costs are roughly the same regardless of whether the cycle is 2, 3 or 4 years but shorting the current cycle time would require additional resources on a per-year basis. 

Conduct the MECS biennially.  With the current 4-year cycle, EIA cannot accurately describe how energy consumption in manufacturing relates to changing energy market conditions, to the cost and availability of capital for investment in new technologies and energy management practices, and to structural shifts in demand for its products.  Because energy intensities and fuel-switching capacities vary considerably between industry classes, so would their response capacity.  Features such as the lag time between a market signal and demand response, whether it is temporary or permanent, and the relative impact by industry sector are not measureable or are missed by infrequent data collection.  Enormous changes have occurred in the structure of American industry, where labor and the supply and cost of energy are key factors in global competitiveness.  There is a significant value to energy policymakers, the Bureau of Economic Analysis, the Federal Reserve Board, and the Bureau of Labor Statistics in having a clearer understanding of the flow of energy, capital and labor.  MECS is conducted primarily via the Internet, so the marginal burden to respondents and cost to EIA of updating this series biennially would be minimized.  A more frequent MECS could integrate policy-relevant topics into the survey in a more timely manner.  Initiative 3.11. Cost per 2-year Cycle: $3,987,700.

Conduct CBECS, RECS, and RTECS triennially.   Adhering to a 3-year cycle would ensure that EIA could best account for factors that are important in forecasts of national consumption, and in identifying trends and structural shifts caused by changes in policy, technology, and behavior.  EIA would realize some efficiencies in survey management with more frequent data collection, more staff in more specialized roles, and new resources to improve data, management and release processes.  Initiative 3.12 (CBECS). Cost per 3-year Cycle:  $13,763,000.  Initiative 3.13 (RECS): Cost per 3-year Cycle: $9,551,000. Initiative 3.14 (RTECS): Cost per 3-year Cycle: $4,740,000. 

Increase Timeliness of End-Use Data Releases

• Improve Timeliness of Data Releases.  To increase the availability of end-use data, EIA would need to reduce the time lag between data collection and data release in part by making better use of the Internet for data release.  In the past 10 years, three major computing changes occurred while budget (in real terms) and staff resources declined:  EIA migrated from a centrally controlled mainframe to a distributed local-area network processing environment, data collection moved from paper-administered forms to computer-assisted interviews or Internet data collection instruments, and reports moved from physical publications to electronic media on the Web.  The processing environment requires more skilled coordination, management, and documentation. The data collection technology requires more pre-survey preparation, which backs into the previous survey cycle.  Web publications require ongoing, specialized support functions to meet user needs and expectations.  Although EIA has begun work to improve performance on all these fronts, additional resources could help attain consistency and efficiency across the consumption surveys.  Initiative 3.15.  Start-up Cost: $500,000; Cost per 2, 3 or 4-year Cycle: $800,000 (increment over EIA’s 2009 budget request).

Conduct Feasibility Studies for Alternative Sources of End-Use Data

• Feasibility Studies for Alternative Sources of Data for End-Use Estimates.  Evaluate alternative Federal, State, and commercial data sources for meeting the statutory requirements of EIA’s end-use program under budgetary constraints. Identify other surveys and administrative records that may prove valuable for benchmarking, modeling, or filling data gaps in the program.  Evaluate the methods used and quality produced relative to EIA’s needs.  Identify the potential for interagency data and cost sharing, collaborative data collections, and value-added analyses to meet the challenges of scarce Federal resources, economic disruptions from the global recession, climate change, and health or national security events.  Initiative 3.16. Annual Operating Cost: up to $500,000 (increment over EIA’s 2009 budget request).

Additional Challenges to Improving Energy Consumption Data

Where there are gaps in the scope, frequency, timing, and/or publication of EIA consumption data, users increasingly resort to ad hoc means to address them.  For example, EPA is assisting businesses, user groups, and trade associations in collecting their own consumption data to augment the CBECS program. In other situations, States are independently collecting data to produce more localized benchmarks.  These efforts are vulnerable to funding cuts, which would likely cause quality challenges similar to or worse than EIA’s.  Furthermore, EIA is subject to OMB’s statistical standards, whereas State data collections are not.  Although resources vary considerably, States are relying more on smaller data collection firms and less expensive, less accurate modes of data collection.

In another example, the Department of Housing and Urban Development produces a Utility Schedule Model using RECS, the basis for calculating utility allowances for various Federal programs.  Because RECS is conducted infrequently, users are left to make their own idiosyncratic or no adjustments to the model for non-RECS years.  Adjusting a model in rapidly changing energy markets is beyond the scope, resources and expertise of most of these users.  As a result, EIA analysts receive direct requests from metropolitan housing authorities (or their consultants) for inter-survey estimates for geographic areas smaller than in the current RECS.  Trade associations for commercial buildings and for manufacturing firms that are trying to work around Federal data gaps also request ad hoc advice and support.  Such requests suggest that the methods used to define and evaluate efficiency and other energy program targets are idiosyncratic and will diverge until EIA can provide more and better consumption data for smaller geographical divisions and analytic subclasses.

Stakeholders and other data users report that the demand for high-quality State-level data will continue to grow.  EIA terms, definitions, and survey methods then become the de facto basis for benchmarking and assessing energy policies led or enacted by States, as well as by other Federal statistical agencies.  Improving the consumption survey program would add considerable value and coherence to data that are central to policy and other decision makers.