AMERICAN STATISTICAL ASSOCIATION COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION Thursday, April 6, 2006 Washington, D.C. 2 1 AMERICAN STATISTICAL ASSOCIATION MEMBERS: 2 NICOLAS HENGARTNER, Ph.D., Chair 3 NAGARAJ K. NEERCHAL, Ph.D., Vice Chair 4 DEREK BINGHAM 5 JOHNNY BLAIR 6 MARK BURTON, Ph.D. 7 CUTLER CLEVELAND 8 JAE EDMONDS, Ph.D. 9 MOSHE FEDER, D.Sc. 10 BARBARA FORSYTH 11 WALTER HILL 12 NEHA KHANNA 13 THOMAS RUTHERFORD 14 SUSAN M. SEREIKA 15 DARIUS SINGPURWALLA 16 PARTICIPANTS: 17 BOB ADLER, Energy Information Administration 18 MARGOT ANDERSON, Energy Information Administration 19 JUSTINE BARDEN, Energy Information Administration 20 BETTY BARLOW, EI-13 21 JAKE BOURNAZIAN, Energy Information 22 Administration BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 3 1 PARTICIPANTS (Cont'd): 2 SUSAN BUCCI, Energy Information Administration 3 GUY CARUSO, Energy Information Administration 4 JOHN CONTI, Office of Integrated Analysis and Forecasting 5 AUDREY CORLEY, EI-45 6 JOHN PAUL DELEY, Energy Information 7 Administration 8 STAN FREEDMAN, Energy Information Administration 9 DWIGHT FRENCH, Energy Information Administration 10 WILL GIFFORD, Energy Information Administration 11 HOWARD GRUENSPECHT, Energy Information Administration 12 VICKI HAITOT, Energy Information Administration 13 BILL HELKIE, Energy Information Administration 14 SHANNON HOWELL, Energy Information Administration 15 ALETHIA JENNINGS, Energy Information 16 Administration 17 STAN KAPLAN 18 NANCY KIRKENDALL, Energy Information Administration 19 TOM LORENZ, Energy Information Administration 20 PAUL McARDLE, EI-01 21 TINA PEARSON, Energy Information Administration 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 4 1 PARTICIPANTS (Cont'd): 2 MARK SCHIPPER, Energy Information Administration 3 SCOTT SITZER, Energy Information Administration 4 BOB SCHNAPP 5 JOHN STAUB, Office of Integrated Analysis and Forecasting 6 PHILLIP TSENG, Ph.D., Energy Information 7 Administration 8 BILL WATSON, CNEAF 9 SHAWNA WAUGH, Energy Information Administration 10 PAULA WEIR, Energy Information Administration 11 WILLIAM WIENIG 12 JOHN WOOD, Energy Information Administration 13 14 * * * * * 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 5 1 C O N T E N T S 2 Open Meeting, Nicolas Hengartner Chair, 7 ASA Committee on Energy Statistics 3 Greeting and Remarks, Guy Caruso, 14 4 Administrator, Energy Information Administration, EIA 5 Updates for the Committee Since the Fall 29 6 2005 Meeting, Nancy Kirkendall, Director, Statistics and Methods Group (SMG), EIA 7 How Can Modeling Suggest Data Needs? Open 40 8 Discussion. Nancy Kirkendall, SMG Chair, Margot Anderson, and other invited EIA 9 senior management 10 2006 MECS: Looking at Past Performance 89 Statistics to motivate New Methods of 11 Collection. Bob Adler and Tom Lorenz, EMEU, EIA. 12 ASA discussant on 2006 MECS past 155 13 performance: M. Feder 14 ASA discussant on New Oil Production SAGE 162 Model: N. Khanna 15 Additional Committee Discussion 168 16 Invitation for Public Comments, Nicolas 184 17 Hengartner, Chair, ASA 18 Measuring Perceptions of Applying 185 Alternative Disclosure Limitation 19 Methods, Jake Bournazian, SMG, EIA. 20 Functional Requirements for EIA's Internet 235 Data Collection System, Stan Freedman, 21 SMG, EIA. 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 6 1 CONTENTS (Cont'd): 2 ASA Discussant on Functional Requirements 304 for EIA's Internet Data Collection System, 3 D. Singpurwalla 4 ASA Discussant on Improving the SAGE 314 Petroleum Refinery Model: T. Rutherford 5 Additional Committee Discussion 323 6 Making Adjustments to Survey Data 330 7 When the Collected Data Do Not Meet Expectations, Stan Kaplan, CNEAF, EIA 8 Invitation for Public Comments 388 9 Adjournment of Thursday meeting 388 10 11 12 * * * * * 13 14 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 7 1 P R O C E E D I N G S 2 (8:39 a.m.) 3 DR. HENGARTNER: Oyez, oyez, oyez. 4 Ladies and gentlemen, thank you very much for 5 your attention. Since we're already 10 6 minutes late, let's start. 7 I'd like to welcome you all to 8 another meeting of the American Statistical 9 Association Advisory Committee on Energy 10 Statistics, and again, I want to emphasize 11 this is an ASA meeting and not an EIA 12 committee, and we meet periodically for 13 providing input and advice to the EIA. 14 This is an open meeting, open to 15 the public, and public comments, of course, 16 are very welcome, either in form of oral 17 comments -- and time we'll set aside for this 18 at the end of today and at the end of 19 Friday's sessions -- as well as written 20 comments that can be sent either to the EIA 21 or to the ASA. 22 I'd like to remind that all BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 8 1 attendees should hopefully sign in at the 2 entrance -- and if you're here you've 3 probably done that -- and also provide e-mail 4 address for whatever reason. 5 Restrooms: Restrooms are most 6 likely on that side, way over there, and 7 probably on the other side, way over there. 8 There is an open line for phone. 9 If you want to have someone call in and leave 10 messages, let me give you the phone number. 11 The area code is (202) 586-0832; the other 12 phone number where they may call here and 13 leave messages during our stay at the meeting 14 is, again, (202) 586-0946. 15 So for the committee member, you 16 know Kathleen Wert and Donna Arrington, they 17 are from the ASA, they will help you with 18 travel reimbursement and other 19 technicalities, so please see them if you 20 have any questions. 21 I'd like to remind all the speakers 22 and all the people commenting to please speak BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 9 1 clearly and loudly into the microphone. This 2 will be highly appreciated by the gentleman 3 here who is making transcripts of our 4 proceedings. And finally, before we start, 5 I'd like to go around the table, and actually 6 the entire room, so that everybody can 7 present themselves, and we'll take it from 8 there afterwards. So let me start. My name 9 is Nick Hengartner; I'm from Los Alamos, and 10 I'm chair of the ASA Committee on Energy 11 Statistics. 12 MS. KIRKENDALL: I'm Nancy 13 Kirkendall, the director of the Statistics 14 and Methods Group in Energy Information 15 Administration. 16 MR. CARUSO: Guy Caruso, 17 administrator, Energy Information 18 Administration. 19 MR. RUTHERFORD: Tom Rutherford, 20 I'm a professional economist. 21 DR. BURTON: Mark Burton, economist 22 from the University of Tennessee. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 10 1 MR. SINGPURWALLA: Darius 2 Singpurwalla, Law, Economic, and Consulting 3 Group. 4 MS. MILLER: Renée Miller, EIA. 5 MR. BOURNAZIAN: Jake Bournazian, 6 SMG and EIA. 7 MR. RASMUSSEN: Erik Rasmussen for 8 the administrator, EIA. 9 MS. LEE: Audrey Lee at DOE. 10 DR. TSENG: Phillip Tseng, SMG, 11 EIA. 12 MS. JENNINGS: Alethea Jennings, 13 SMG, EIA. 14 MR. STAUB: John Staub, EIA. 15 MR. SWEETNAM: Glen Sweetnam, in 16 OIF in EIA. 17 MR. CONTI: John Conti, OIF. 18 MS. KHANNA: Neha Khanna, Economics 19 and Environmental Studies, Binghamton 20 University. 21 MR. CLEVELAND: Cutler Cleveland, 22 Center for Energy Environmental Studies, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 11 1 Boston University. 2 MR. BINGHAM: Derek Bingham, 3 Department of Statistics, Simon Fraser 4 University. 5 DR. NEERCHAL: Nagaraj Neerchal, 6 Department of Math and Statistics at UMBC. 7 Also the vice chair of it. 8 MR. HELKIE: Bill Helkie, EIA. 9 MR. WATSON: Bill Watson, EIA. 10 MR. LORENZ: Tom Lorenz, EMEU, EIA. 11 MS. HOWELL: Shannon Howell, EIA. 12 MR. FENNELL: Dean Fennell, CNEAF, 13 EIA. 14 MS. FRENCH: Carol French, Office 15 of Oil and Gas, EIA. 16 MR. DELEY: John Paul Deley, NEIC, 17 EIA. 18 MR. LU: Ruey-Pyng Lu, EIA. 19 MR. DISBROW: Jim Disbrow, OIT, 20 EIA. 21 MS. HOJJATI: Behjat Hojjati, EIA. 22 MR. FAWZI: Aloulou Fawzi, OIF, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 12 1 EIA. 2 MS. HOLT: Susan Holt, Office of 3 the Administrator, EIA. 4 MR. BRADSHER-FREDERICK: Howard 5 Bradsher-Frederick, EIA and SMG. 6 MR. STROUD: Ron Stroud with SMG, 7 EIA. 8 MR. WOOD: John Wood, NRPD, EIA. 9 MS. WAUGH: Shawna Waugh, 10 Statistics and Methods Group. 11 DR. HENGARTNER: Thank you very 12 much. Let me continue. First of all, I'd 13 like to remind you this is a DOE meeting, and 14 it's under the DOE terms; that means that 15 Nancy Kirkendall here is the designated 16 official, and she has the right to interrupt, 17 adjourn, or take over the meeting as she sees 18 fit. 19 MS. KIRKENDALL: Boy, talk about 20 power. 21 DR. HENGARTNER: The other thing 22 I'd like to tell you is I think this is going BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 13 1 to be a very interesting meeting. Before we 2 start the meeting, I'd like to welcome Derek 3 Bingham; he's from Simon Fraser University, 4 he's essentially their subs for Randy Sitter, 5 so that's big shoes to fill, but I know Derek 6 will do a great job at this. 7 The first session, we're going to 8 have the pleasure of hearing the EIA 9 director, Mr. Guy Caruso, followed by Nancy 10 Kirkendall's overview of the Statistics and 11 Methods Group, and there are going to be two 12 sessions in the morning, I believe, and then 13 lunch -- very important, never forget lunch. 14 And lunch is going to be in room -- the 15 first-floor room which we've been all -- if 16 you don't know where it is, follow the crowd, 17 that's where it is. And tonight, we're going 18 to have dinner at the Marrakesh, which is a 19 Moroccan restaurant; it's a seven-course 20 meal, I believe, and so I'd like a show of 21 hands to see who might be wanting to join us 22 there for tonight so that we can make BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 14 1 appropriate reservations. So -- 2 MS. KIRKENDALL: I have two. 3 DR. HENGARTNER: Two, so -- seven. 4 Are you coming for dinner, Barbara? 5 MS. FORSYTH: Not tonight. I can 6 barely make it for breakfast. 7 (Laughter) 8 DR. HENGARTNER: Do you want to 9 introduce yourself? 10 MS. FORSYTH: Sorry to be late. 11 I'm Barb Forsyth. 12 DR. HENGARTNER: So I think that's 13 all there is. So tomorrow, reminder, we're 14 going to resume at 8:30, but I remind you 15 about those technicalities at the end of the 16 day. So without further due process, it's a 17 pleasure to recognize Mr. Guy Caruso, the 18 administrator of the Energy Information 19 Agency. Thank you very much. 20 MR. CARUSO: Thank you, Nick, and 21 good morning everyone. I want to make, 22 again, a special thanks to all of the ASA BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 15 1 committee members, and we appreciate your 2 taking the time and having this meeting to 3 advise us twice a year and as I've said many 4 times in previous meetings how much we really 5 appreciate your advice and how useful it is. 6 And I know you see the results of it -- I 7 hope -- from meeting to meeting and on a very 8 practical way, whether it's frames of our 9 surveys, to the website improvements, to 10 methodological changes in modeling, all of 11 these have benefited by the counsel and 12 advice of this committee. So thank you very 13 much, and thank you, Nick, for chairing the 14 session, and I'd look forward to, again, 15 having a good two days of discussions. 16 As always, in that six months' 17 interval between meetings, the EIA has been 18 very busy with a large number of activities, 19 whether they be our outlooks or new surveys, 20 improvements in old surveys, et cetera, all 21 of which we'll talk about today. 22 The usual Annual Energy Outlook, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 16 1 going out to 2030 this year, was released in 2 its entirety in February, and it's available, 3 of course, on the website, and I think when 4 one looks at what's the main highlights of 5 the AEO '06, this year, clearly, the 6 reassessment of the oil price outlook was by 7 far the biggest change and has implications, 8 of course, for interfuel competition as well 9 as the oil market outlook, and -- 10 One of the things when we make 11 reassessments -- in this case, oil prices -- 12 and indeed, even natural gas prices are up 13 considerably in this outlook -- it stretches 14 the NEMS, the National Energy Modeling System 15 that many of you are familiar with. And 16 clearly, this is a case where, this year, 17 looking at much higher prices, both for oil 18 and natural gas, and the implications for not 19 only oil but the other sectors besides 20 transportation, such as electric power and 21 the implications for coal -- all brought to 22 bear some significant methodological BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 17 1 challenges, and I think some of them will be 2 discussed today, but I think, more 3 importantly, we'll be looking hard at those 4 of how the model responds to these new 5 assumptions, and, as I'll mention when I talk 6 about the budget, we're hoping to launch a 7 serious upgrade of our National Energy 8 Modeling System, beginning in the second half 9 of '06, but continuing, we think, if money is 10 appropriated, for the next several years. 11 So, again, that was the big change, 12 and you could see it here, about a 13 $20-per-barrel increase over the long-range 14 assumptions here through 2030, going from 15 roughly $30 a barrel to roughly $50 a barrel 16 on average. And part of it is the investment 17 side of things, particularly from OPEC 18 countries, but also the recognition that the 19 cost of doing business in the upstream in the 20 oil sector is increasing with the commodities 21 boom, and the cost of doing just about 22 everything from drilling through the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 18 1 downstream and refining and distribution. 2 All have contributed to that. We are still 3 not among the camp that says that world oil 4 is peaking and declining in the near term and 5 therefore that's what's driving prices. 6 That's not what we're saying; we're saying 7 that the issues driving prices are mainly 8 above the ground, not below the ground, and I 9 know John Wood is here, and he'll talk about 10 mainly the new survey for crude, but 11 certainly John and his people have been very 12 firm in their belief that there's still lots 13 more oil to be discovered and more productive 14 capacity to be added over the next 20 to 30 15 years. 16 Of course, short term tends to get 17 a lot more attention, and our latest 18 short-term issue, as we head to the driving 19 season this year, is the transition in 20 reformulated gasoline composition by the 21 decision being made by almost all refiners to 22 remove MTBE -- methyl tertiary butyl ether -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 19 1 from reformulated gasoline as the main 2 oxygenate and octane booster and replacing it 3 with ethanol for the most part. That 4 transition is underway, and our view is that 5 it could be a bit of a rocky road to make 6 this transition, and that could lead to some 7 supply dislocations in certain regions of the 8 U.S., and it's gotten some attention both in 9 Congress and in the administration. And of 10 course, we've all witnessed some run-up in 11 gasoline prices already this spring; only 12 part of that, we believe, is part of this 13 transition, but nevertheless, it's an issue 14 that EIA is deeply involved in and raised the 15 flag early on, in January, by a paper that 16 was done by Joanne Shore in our oil and 17 natural gas office. 18 Again, as I mentioned, we continue 19 to look for ways to improve the collection of 20 data, in this case, on the upstream crude oil 21 collection. As you know, we've started 22 collecting natural gas production data, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 20 1 heavily relying on the advice of this 2 committee in developing that form, and John 3 Wood's here to talk about the 914, the 4 expansion of the data there, but we are 5 considering expanding that in 914 form to 6 include the collection of crude oil 7 production and very much appreciate hearing 8 your advice on that. 9 One of the things that's really 10 improved the productivity of collecting data 11 -- and the quality, we believe -- is the 12 Internet data collection systems that you've 13 heard about previously from the various -- 14 particular from our coal and natural -- in 15 nuclear and electricity -- CNEAF office. 16 That's going along very well. Several IDCs 17 were launched in the last year or so, and 18 they're operating pretty well, we think, and 19 definitely have improved the speed, the 20 timeliness, and accuracy of the responses 21 because so much of the editing is done at the 22 respondent's side of things and therefore BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 21 1 there's less follow-up and it allows us to do 2 other things, to work on the quality and the 3 distribution of the data. So that's going 4 very well and appreciate very much all of 5 your help in giving us advice on that. 6 We've -- after a long process, the 7 voluntary reporting of greenhouse gas 8 programs -- guidelines have been revised, and 9 we're about to begin the redesign of that 10 program, and we expect that we'd start around 11 the first of June and would be fully 12 implemented during 2007. This has gotten a 13 lot of attention with respect to the issues 14 of climate change, and EIA's playing a very 15 important role in -- and will continue to 16 play an important role in the office that is 17 headed by John Conti, OIF. 18 As many of you knew from being 19 participants in the improvement of our 20 website, we did launch the new site in 21 November. We've gotten substantial positive 22 feedback on this redesign, and continue to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 22 1 work on improvements and certainly, again, 2 welcome any advice you would have on the new 3 website. It's a work continually in 4 progress, as you're aware, and we're 5 currently doing a customer survey on the 6 kids' page, which is one of the most popular 7 components of our website. And on the 8 personnel side, just this week, Monday, a new 9 Web champion, Gina Pearson, has come on 10 board. She'll be the director of our 11 National Energy Information Center, NEIC, 12 succeeding John Weiner, who many of you know 13 from his many years as heading up NEIC. 14 Among Gina's many talents is her experience 15 and success in redesigning websites. She's 16 most recently come from the Energy Research 17 Service, where she completely redesigned -- 18 MS. KIRKENDALL: Economic. 19 MR. CARUSO: Economic Service? 20 MR. KIRKENDALL: Economic Research 21 Service. 22 MR. CARUSO: Economic Research BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 23 1 Service. It's part of the Department of 2 Agriculture, right? Yeah. And she's done a 3 outstanding job there and at Department of 4 Education, and we're looking forward to even 5 greater things as our Web evolves. 6 I mentioned one component of our 7 budget request for 2007, which is currently 8 being discussed on Capitol Hill. That is the 9 request for an increase of several million 10 dollars, which partly will be used for this 11 upgrading of the National Energy Modeling 12 System, and we look forward to hopefully 13 getting that money appropriated and working 14 with you all when this gets more to where we 15 move from a planning to implementation phase 16 and getting your advice on that. 17 This year, we have a budget of a 18 bit more than $85 million; it's a small 19 increase from '05, and enables us to pay for 20 about 370 federal employees and a number of 21 contractors. We have, as I mentioned, asked 22 for about $4.5 million increase for '07, with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 24 1 a small increase in FTEs to 375, but more 2 importantly, the increase would go to, as I 3 said, upgrading the NEMS as well as some 4 other areas. 5 DOE is now moving toward a 6 five-year budget cycle, and we're certainly 7 participating in that, and, as I mentioned, 8 we think the NEMS upgrade will be a multiyear 9 task. It will require at least three years 10 to go from the planning through actual 11 utilization of that new revised NEMS. And 12 when the current system was designed and put 13 into use, it took about three years. I think 14 it started in '91 and it was '93 before the 15 Annual Energy Outlook was fully utilizing the 16 current NEMS. So clearly we look forward to 17 this five-year cycle that the department is 18 now engaged in, and our targets for 19 timeliness and quality continue to be quite 20 stringent. 21 And the other part of the 22 longer-term -- five years, if you will -- is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 25 1 our continued need for the succession 2 planning and workforce planning that we've 3 talked about before this group previously, 4 and that is with an organization that's now 5 approaching 30 years. Many of our most 6 experienced people are reaching the 7 retirement age, and we're having a fairly 8 substantial turnover. Last year, we lost 16 9 senior EIA employees, and we've been at least 10 successful so far in the recruitment and 11 succession to those 16, but we're looking at 12 probably 20 to 25 senior people leaving in 13 '06 and again in '07. So, again, as I've 14 said in the past, any assistance you can be 15 in terms of recruitment of your students or 16 your associates that you come into contact 17 with, we very much appreciate that because 18 you know the kind of work we do here and the 19 type of people that we need to keep up the 20 tradition of high quality. 21 About one year ago, in response to 22 OMB evaluation of our budget, one of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 26 1 things they said, and we talked to you about 2 and asked your advice, was no one's ever 3 really done an overview evaluation of EIA. 4 We've seen evaluations of specific surveys, 5 specific components of the program, but what 6 we'd like to see is an independent external 7 review of EIA in its entirety, and I was 8 fortunate to be able to convince Denny 9 Ellerman of MIT to chair a team to do exactly 10 that, look at what it is that EIA does and 11 are we doing the right thing, is the main 12 mission we asked of Denny and his team -- and 13 he went around and recruited four other 14 outstanding individuals who many of you know: 15 Paul Joskow of MIT; Kathy Cooper's a former 16 undersecretary of Commerce and former chief 17 economist of Exxon Mobil; Jay Hakes, my 18 immediate predecessor; and Phil Sharp, who 19 was then at the Kennedy School and now is the 20 director of Resources for the Future and 21 former congressman. They looked very hard at 22 what EIA's original legislation authorization BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 27 1 asked of us 30 years ago and how we've done 2 in meeting that mandate, and, even more 3 importantly, looking out in the future. Are 4 we preparing for what the future's going to 5 bring and the pressures and the requirements 6 on us? 7 They met twice last year, and we 8 expect that we'll receive their full report 9 -- we've gotten a preliminary briefing from 10 Denny last month -- and we expect to have the 11 full report, which we would make available, 12 of course, to you, and we'd hope to have a 13 full discussion of what they have found in 14 their review, and, more importantly, what 15 they are recommending for the future for EIA, 16 including areas where we may want to place 17 more emphasis and areas where we may want to 18 place less emphasis and indeed, even to maybe 19 even stop doing some things. So we found it 20 to be an outstanding opportunity to actually 21 almost do a self-evaluation in order to 22 prepare for the external evaluation. As you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 28 1 had so wisely recommended, we approached this 2 kind of like a visiting scholar's approach 3 that many of your universities use, and I 4 think that was a very wise advice on your 5 part because we found it to be very useful. 6 As I mentioned, we are approaching 7 our 30th year, having come into being as the 8 Energy Information Administration in 1977, 9 and so we're looking forward to having a 10 celebration some time. Probably it'll be a 11 rolling celebration throughout '07 but we'll 12 focus it mainly on the educational aspects 13 that we think we can play an important role 14 in the energy education of not only the 15 decision makers, but the public mainly 16 through our website. 17 So we'll be asking for your ideas 18 and input as well as we plan on this, so, 19 again, let me close by once again thanking 20 you, Nick, and all the members of the 21 committee for your past advice and especially 22 for the advice we'll be getting today, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 29 1 tomorrow, and in the future. So if there are 2 any questions, or anybody like to go into any 3 more detail on any things I've mentioned, I'd 4 be happy to do that now or any time during 5 the next couple days. 6 Okay, thanks. 7 DR. HENGARTNER: Thank you very 8 much. 9 (Applause) 10 DR. HENGARTNER: I'd like now to 11 welcome Nancy Kirkendall to give us a review 12 of what the Statistics and Methods Group has 13 been up to in the last six months. 14 MS. KIRKENDALL: What I'm actually 15 going to review is what we did in response to 16 some of the committee's suggestions from the 17 sessions we had last time. Sometimes that's 18 the same thing and sometimes it's not. 19 I have picked out a number of the 20 sessions that we had last time, and I thought 21 I'd give you a little update on what we've 22 done or what we haven't done in some cases. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 30 1 So let me start out -- this is the Short-Term 2 Energy Outlook. Last time, Margot Anderson 3 talked to you about her plans to produce a 4 report that would analyze the forecast errors 5 produced from the STEO, the Short-Term Energy 6 Outlook. You made a number of 7 recommendations. One of the things a lot of 8 people said -- and this was more than one of 9 the people in the room -- said that one of 10 the reasons for developing a model is to 11 identify new data needs. And when I went 12 through the suggestions, I thought, okay, 13 well, now what do we do. So we've put that 14 on the agenda to talk about in more detail, 15 and we'd like to get more of your ideas on 16 what we can do with that idea. 17 The other suggestion that I thought 18 was interesting was to separate the model 19 into two parts, or the modeling errors into 20 two parts: The error in the modeling and the 21 error in the input variables. And that's 22 something that I still think is really BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 31 1 interesting and useful. We haven't had a 2 chance to do that yet. However, Margot's 3 group has produced a draft of a forecast 4 error report. They've done analysis of 5 forecast errors for one-, six-, and 6 twelve-month forecast horizons, and that is 7 currently out for review, so hopefully we'll 8 have something to share with you next time to 9 get more input on it. 10 Vehicle energy use. Mark Schipper, 11 also from the Office of Energy Markets and 12 End Use, gave a talk about the work they've 13 been doing with the Department of 14 Transportation. This is basically to fill a 15 data gap caused by losing the residential 16 transportation energy consumption survey. 17 And so the committee suggested that they 18 liked the modeled data and you also 19 recommended that we continue collaborating 20 with the Department of Transportation, and of 21 course we're planning on doing that. And 22 since the last meeting we have offered a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 32 1 public-use version of a report from the 2 modeled data, so there's been further work on 3 that. 4 There were two sessions by Renée 5 Miller of the Statistics and Methods Group 6 last time. One of them talked about some 7 data quality initiatives, how to come up with 8 a nice little product that would illustrate, 9 perhaps, a different data series to try to 10 measure the same thing and use it as an 11 educational tool. The other one talked about 12 discrepant estimates. Any time you have a 13 survey to measure something, if you have two 14 surveys, you get two different numbers. 15 Sometimes this is useful and sometimes it's 16 mystifying. So we talked to you about that. 17 So this is a summary of your advice. One of 18 them -- and Mark Bernstein told us this all 19 regularly through his time on the committee 20 -- and that is that in all of our 21 presentations, we're supposed to state our 22 objectives clearly. Hopefully, this time, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 33 1 you'll see the objectives more clearly in the 2 presentations, but we'll see. But in terms 3 of advice on this particular issue, you 4 suggested that we need to focus on the 5 biggest data quality problems first, which 6 makes a lot of sense -- sometimes it's not 7 easy to figure out which those are -- to 8 measure data quality, do an internal report, 9 and then develop a Web product after that. 10 So the purpose is to evaluate data quality, 11 so the advice was to have a more systematic 12 approach that would lead to possibly a Web 13 publication, but maybe not in some cases. 14 The other thing that you have 15 strongly recommended that we do is to inform 16 users about data quality issues, and on that 17 one, we know we need to do that, but we're 18 not clear quite how to yet. The issue has 19 been raised with the interoffice issue group 20 that Renée Miller chairs and Howard 21 Gruenspecht is a member of that. So we'll 22 still be debating how to do that. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 34 1 We talked about -- we've looked a 2 little bit at looking at performance measures 3 to try to find some measures that would be an 4 indicator of data quality problems we should 5 look at. This is just ongoing work. Renée's 6 group is going to take on, in -- sometime in 7 the '80s or '90s there was an evaluation -- a 8 comparison of data from the Residential 9 Energy Consumption Survey and data that we 10 get on our supply surveys. And naturally 11 enough, they didn't match, and we tried to 12 explain why. So that report was done -- it 13 was a good report, but it was done a long 14 time ago, and they're going to update it. So 15 it's a good model to use. We need to look at 16 different data -- at the data sources and see 17 what we can learn from that. So that -- 18 we'll work on that beginning sometime this 19 spring or summer. 20 And then the last one I already 21 mentioned. The survey self-assessment 22 process -- this is Tom Broene's work -- and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 35 1 it was a big undertaking. Tom interviewed 2 all of the survey managers in EIA to try to 3 help fill out a little evaluation form, and 4 unfortunately it turns out to be a huge 5 burden to do that. And you obviously 6 recognized that and have suggested that we 7 need to be more flexible in how we develop it 8 and how we implement it because we need to 9 have a process we can actually do on a 10 regular basis. So that's sort of a summary 11 of what your advice was. 12 We are reconsidering the process, 13 and the only thing that we think we'll keep 14 from the old process is the -- we ask for 15 survey-specific targets for improvement, and 16 I think we still would like to do that. At 17 least it tries to focus each survey on what 18 they need to do -- or it could be survey 19 system -- flexibility idea. 20 This was an analysis between -- I 21 think they were trying -- it was Bill Trapman 22 and Leila from the Office of Oil and Gas -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 36 1 it was a modeling exercise trying to look for 2 predictive relationships between futures 3 prices and Henry Hub prices. And I think the 4 committee was supportive of the analysis and 5 didn't have particular recommendations for 6 what to do different, but they thought with 7 that particular analysis -- you did advise 8 trying to incorporate NYMEX prices into STEO 9 for predictive purposes. And that's sort of 10 interesting because I think that's one of the 11 things the external review team has also 12 talked about, is that EIA should make better 13 use of some of the public data, like the 14 NYMEX prices. So that's kind of interesting 15 when advice kind of comes together. Probably 16 means that we need to be serious about taking 17 it. 18 And then finally, last time we 19 reported to you on the frames evaluation that 20 was done between -- it was a cooperative 21 relationship between the Census Bureau and 22 EIA. It was a good effort. You commented BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 37 1 that we had a pretty high coverage on these 2 surveys -- 75 to 100 percent by volume -- and 3 that was especially true, given there were a 4 lot of problems in matching. The name and 5 address -- all we had to match was name and 6 address lists, and they were not exactly the 7 same. They weren't formatted the same and so 8 there was a level of detail that was missing. 9 You suggested collecting EIN numbers to help 10 in matching; I think we are considering that, 11 and we'll see how that plays out. And you 12 suggested that we should use a cost-benefit 13 approach in deciding whether to try to fix 14 some of these frames. I think that means 15 work on the ones that are important and don't 16 worry so much about the others. And so of 17 course we don't have as many resources as we 18 might want for this sort of thing, so we're 19 definitely doing that. Right now, CNEAF has 20 identified a different data gap related to 21 their frame, and that's what they're working 22 on. They've discovered that we're missing BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 38 1 some production of waste coal, and they're 2 trying to add that on the production side, 3 which is not the surveys that we're doing 4 here. 5 But another comment on this 6 evaluation, tomorrow after the break we're 7 going to be giving a little certificate of 8 appreciation to the people from the Census 9 Bureau who helped us on this project and who 10 gave presentations to you. So that was 11 really good. 12 Well, that's my summary from last 13 fall. If you have any questions, please ask. 14 Okay, well, I think we have an 15 interesting set of topics to talk to you 16 about today, and we look forward to your 17 advice. 18 DR. HENGARTNER: Thank you very 19 much, Nancy. 20 (Applause) 21 DR. HENGARTNER: Before we 22 continue, I'd like to invite the people who BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 39 1 trickled in after the first set of 2 introductions to introduce themselves to the 3 microphone so that you can be on the record 4 as being present. Moshe, do you want to 5 start? 6 DR. FEDER: My name is Moshe Feder, 7 I'm a statistician from the Research Triangle 8 Institute. 9 MR. HILL: Walter Hill, St. Mary's 10 College of Maryland, Department of Political 11 Science. 12 MR. FREEDMAN: Stan Freedman, EIA. 13 MR. ZHANG: Bin Zhang, EIA. 14 MS. BLUMBERG: Carol Blumberg of 15 Winona State University, and I'm actually 16 interviewing for a job here today. 17 (Laughter) 18 MR. BROENE: Tom Broene, EIA. 19 MS. ANDERSON: Margot Anderson, 20 EIA. 21 MS. BARDEN: Justine Barden, EIA. 22 MS. WEIR: Paula Weir, EIA. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 40 1 MR. WEINIG: Would you step forward 2 and use one of the microphones, please? 3 MR. BENNECHE: Joe Benneche, EIA. 4 DR. HENGARTNER: Okay, thank you 5 very much. I'd like to continue -- we're on 6 schedule, fortunately, with a -- I think it's 7 an open discussion, is that correct, by -- 8 how can we model suggested data needs, and 9 it's going to be essentially led by Margot 10 and Nancy and other invited EIA senior 11 management. 12 MS. KIRKENDALL: There is a summary 13 of the comments you made. I think it's under 14 item one in your notebooks, and all I did was 15 pull out some of the quotes that I thought 16 were appropriate for this discussion. EIA is 17 really fortunate that we are both users -- 18 producers of data and we use our own data, 19 and you would hope that we could do a good 20 job with a feedback relationship between the 21 users and the producers of the data. That 22 would help improve both sides of the issue. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 41 1 However, we haven't ever managed to figure 2 out how to do that, and so we would be 3 interested in your advice on how the modeling 4 and data and have a good feedback -- how do 5 you learn from your modeling what your data 6 needs are? Any ideas on what kind of a 7 feedback mechanism might work, if you've seen 8 anything like that in the other places. 9 MS. KHANNA: I have a very 10 preliminary question -- this could be because 11 I wasn't here the last time -- which without 12 language and models, how do you figure out 13 what your -----? How does the EIA even start 14 that process? 15 MS. ANDERSON: Well, we already -- 16 in most cases, we're not starting up with any 17 new models. We basically have two or three 18 models that we use for -- do I need -- 19 MR. WEINIG: I couldn't hear. I'm 20 not sure anyone here heard the opening 21 remarks. 22 MS. KHANNA: Sorry. Okay, I hope BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 42 1 you can hear this now. My basic question 2 was, without knowing a model, how do you even 3 begin to assess your data needs? And so for 4 me the starting point always is here is what 5 I'm trying to model, and that almost always 6 drives my data needs. So I'm a little 7 surprised by the question in the first place. 8 MS. ANDERSON: I think we were 9 looking at it from the aspect of having 10 existing models and how do you enhance 11 existing models and identify the data gap 12 that you might fill by collecting new data or 13 doing new surveys or collecting already 14 commercially available data. We don't have a 15 lot of models in EIA; we tend to have a 16 couple of forecasting models, a short-term 17 model, some long-term forecasting models, but 18 for the most cases, we're not building new 19 modeling frameworks. So sort of the question 20 that first comes out of is, we have these 21 data programs first and the modeling came 22 somewhat after having the data programs BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 43 1 intact. So the data programs were -- what 2 kind of data do we have that can support the 3 modeling that we would like to do, the 4 short-term and the longer-term forecasting. 5 And it's opposed to the other way around of 6 starting a model from fresh and then 7 determining what data. John might have some 8 ideas on if we go through this process, when 9 we go through this process, of redoing the 10 national energy model, and it certainly might 11 engender some new questions that we might 12 want to either collect some data or rejigger 13 some data that we're already collecting or 14 use a portion of data more frequently. But 15 right now we're not building any new models 16 that would require us to reevaluate the data 17 programs, we're trying to get a sense of how 18 do we use the models that we have to ask 19 ourselves what data might improve the 20 predictive capability of those models. I 21 think. That's the question I'm asking -- or 22 at least -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 44 1 MR. CONTI: Let me maybe expand on 2 that a little bit in getting back to the 3 question. First of all, I think once upon a 4 time, when people were trying to understand 5 the energy industry, it wasn't what data do 6 we need for the models, it was what analytic 7 information did we need to have in order to 8 understand the energy questions that were 9 being asked of us. And so EIA created data 10 programs to provide those answers. As the 11 questions became more complex, they developed 12 models to help them do the analytics to 13 answer the questions. 14 Now we ask the question, well, what 15 does the model tell you in terms of data 16 requirements. Well, the model often tells us 17 that we are missing key data that we'd like 18 to have. And we've identified areas in the 19 past where this data is missing. For 20 example, in the ethanol area, we sat down 21 with people, both in DOE and other parts of 22 EIA, and identified -- we'd like to have this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 45 1 data, and they said, "Well, thank you very 2 much, we'd like to have it, too. We can't 3 afford to give you that data." As a matter 4 of fact, just a couple of weeks ago, EIA 5 announced it was eliminating two data forms, 6 one of which helps us understand crude 7 quality and crude prices, something that's 8 probably of analytic interest in the future. 9 It's not the fact that you identify a need 10 with the model and then, okay, we just run 11 out and get the data, you still have to have 12 a budget to be able to get that data. So -- 13 that's where I'll stop. 14 I'm John Conti, director of the 15 Office of Integrated Analysis and 16 Forecasting. 17 MS. KIRKENDALL: The other comment 18 was a lot of our data series -- EIA was 19 organized and established in '78, and there 20 was a predecessor organization, too. So our 21 data collections have been around for a long 22 time, and we tinker with them. You add BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 46 1 products, you subtract products, you find a 2 few new surveys you need to do. But a lot of 3 the data programs have been around for a long 4 time. 5 MS. ANDERSON: And never intended 6 to be necessarily used in a modeling context. 7 They were almost ends of themselves because 8 people either gave a series or gave 9 information on series that weren't really 10 asked to do -- look at these relations -- 11 MS. KIRKENDALL: Well, except that 12 you were providing the data for people who 13 want to do that kind of analysis. That's 14 what our data program's for; it's supposed to 15 be -- doesn't have to be for us. And so it's 16 supposed to support analysis that's outside 17 the EIA, too. 18 MS. ANDERSON: Correct. 19 MS. KIRKENDALL: It's just that we 20 do have an opportunity to learn more about 21 what data are needed from our own internal 22 uses and modeling. So that's sort of -- how BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 47 1 do we do that a little better. And John 2 talked about the ethanol usage -- is there 3 another way of -- and that's going to be 4 coming around and we'll probably end up doing 5 something about that. 6 MR. CLEVELAND: Well, I'm glad to 7 hear somebody say that, but as far as I 8 understand it, I think key staff people have 9 identified data needs in the past then it 10 really doesn't get past that, identifying the 11 need. I don't think the problem is 12 identifying the need, I think the problem is 13 figuring out how could we meet that need 14 within existing budgets. 15 MR. KIRKENDALL: Or get that need 16 moved up to a higher-level discussion when 17 we're talking about allocation of scarce 18 resources. At least at that discussion point 19 we know what's on the table for data needs. 20 You might not be able to do anything, but at 21 least it gets into the discussion. 22 MR. CLEVELAND: I guess one of my BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 48 1 concerns is to what extent the questions that 2 you're even asking have been constrained by 3 the historical data collection and modeling 4 process, and I'm wondering -- and I'm sure 5 that your external review panel is asking if 6 we had to blow it up and start all over -- 7 which I'm not proposing to do -- what 8 questions would we be asking and what models 9 would we be trying to build to answer those 10 new important questions and, in turn, what 11 data -- what do we need to do with that, 12 rather than approaching it from the other 13 way. 14 And a secondary question is, to 15 what extent is the demand for data by people 16 outside of the EIA, like me, in terms of 17 determining what you do and don't collect, 18 and how do you balance that against your own 19 internal needs? 20 MS. KIRKENDALL: So a question is 21 what's a good way for getting input from you, 22 for example, on what you think the data needs BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 49 1 are -- and other external users. 2 MR. CLEVELAND: Yeah, except when I 3 hear the budget situation, it's hard to me to 4 even conceive that our needs would be that 5 high on the list. But it seems like 6 something you should be doing. 7 MS. ANDERSON: It kind of goes back 8 to this issue of defining your objectives. 9 If you don't know why you're doing something, 10 whether it's to create a structural model or 11 a forecasting model, you're not clear what 12 the questions are that you're trying to 13 answer, you're going to fail on trying to 14 determine what data must you collect and who 15 do you need to go ask about your data. 16 So it goes back to the advice that 17 we've been given and we try to take, but 18 you've got to define your objectives because 19 we can't do everything. And it's not such an 20 easy exercise to define your objectives in an 21 organization that has such a diverse client 22 base and feels that it is required to answer BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 50 1 or cover all aspects of the energy sector. 2 And we really do try to stay balanced; we 3 really do try to have universal coverage, and 4 regret always when something has to go away 5 and there are multiple objectives that 6 sometimes work at cross-purposes. 7 But I think it is really 8 identifying the key objectives, and it is 9 asking ourselves, if we could start all over 10 again, would we do what we have today -- even 11 to ask ourselves that in little microcosms. 12 If we didn't have a short term for our 13 forecasting program, would we create one or 14 would we do something else? And those are -- 15 we really created that form -- I think the 16 external study team has caused all of us to 17 take a long breath and kind of ask some of 18 those fundamental questions. But even if we 19 go through that exercise internally, you run 20 up with the same kind of constraints that 21 John's identifying. 22 How can you communicate that to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 51 1 folks that tend not to live and breathe this 2 and are much more -- adhere to the status 3 quo? We've been getting it all these years, 4 we've been getting this product all these 5 years, change is very jarring. 6 So I don't want to say it's 7 impossible, but you kind of have to tackle 8 these things on the margins a bit. But it 9 does go back to define your objectives: What 10 are we trying to do here at EIA? 11 MR. CONTI: Maybe getting back to 12 the first part of that question, Guy 13 mentioned earlier how we will be developing a 14 National Energy Modeling System, and as we 15 start down this path, one of the first things 16 we'll do is we'll do a requirements analysis. 17 And within this requirements analysis, we'll 18 be speaking to a bunch of our stakeholders to 19 try and understand the questions that they 20 want us to be able to answer in the mid to 21 long term. And then, assessing all of those 22 needs, we'll try and determine which ones are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 52 1 feasible to address -- and there'll probably 2 be some things we won't be able to address -- 3 and then the ones that we think that are 4 relevant to address and possible to address, 5 we'll do an assessment of, does the data 6 exist, and if the data doesn't exist, we'll 7 do an assessment of, well, how expensive is 8 it to get that data. So in the end, we might 9 not be doing it, but we will have gone 10 through a couple of steps to, does it need to 11 get done, how expensive is it to get done, 12 where can we get the data, and how do you get 13 that into a model. 14 MR. RUTHERFORD: One other comment 15 I'd just add to that is the -- Tom Rutherford 16 -- I think the discussion has been quite -- 17 my comment at the last thing was perhaps a 18 bit of a naïve question because clearly it's 19 kind of obvious, well, duh, that's what we 20 do, right, is we collect data, we do models, 21 we answer questions. So obviously this is 22 part of the process. The question is, to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 53 1 what extent can you formalize that process, 2 right. So it's obvious that you guys are 3 involved in -- I wasn't suggesting by my 4 comment that you weren't going about things 5 this way. 6 MR. CONTI: I think that's 7 important. How do we formalize the need for 8 specific data elements, since I'm the modeler 9 and not the data collector, and where does 10 that fit into the priority list on the data 11 collector's side? 12 MR. RUTHERFORD: Yeah. 13 MR. CONTI: Is it their number one 14 priority? They have a bunch of priority 15 things they want to do, and being able to 16 answer some question from someone they don't 17 know is not necessarily very high on their 18 priority list. 19 MR. RUTHERFORD: The second comment 20 I'd make is that I think the process -- if we 21 think about the cost-benefit -- I think your 22 portraying of the issue as a cost-benefit BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 54 1 tradeoff is right on the money -- but I'd say 2 it's more difficult than solving a linear 3 program to determine this cost-benefit 4 process because clearly the collection of 5 data involves discrete decisions about 6 setting up the apparatus for collecting data, 7 and then the process of just turning the 8 crank to get the data out is probably much 9 less costly. So it's really an integer 10 program you have to solve to determine what 11 to do. In other words, once you have a 12 series going, then the cost of keeping it 13 going is probably much less than the process 14 of deciding what to do. So I'd say it's sort 15 of a difficult issue, but I think that -- 16 just my question really harks back to this 17 idea: We have policy issues which can be 18 solved on the basis of back-of-the-envelope 19 calculations that are informed by data, and 20 there are other more subtle questions that 21 models help tease out which may identify data 22 that aren't obvious at the outset. And so BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 55 1 just being aware of that as part of the 2 process I think is helpful. 3 MS. KHANNA: There's another way of 4 framing the discussion, which is to say are 5 specific elements of the energy market 6 changing in any way fundamentally, and how 7 does that drive our modeling and therefore 8 data -- or data and therefore modeling -- 9 needs. Because I look at the EU and the huge 10 shift in oil prices, which makes a lot of 11 sense to me, but that's a whole different 12 band than we've seen historically. I know 13 you're talking about 30 year band and the $50 14 to $60 price range; what's driving that? 15 That's a fundamental change in the oil market 16 as far as I can tell. 17 MS. ANDERSON: The external study 18 team focused a lot on this, particularly in 19 the area of electricity and electric sector 20 restructuring and whether we really were 21 collecting the data that were consistent with 22 these huge changes in that part of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 56 1 market, and I think we'll have some 2 recommendations on what we might do to be 3 more up to date in that area. So they 4 certainly were cognizant of these issues, and 5 they also talked a lot about the 6 international factors driving markets, which 7 would certainly come to your point at looking 8 at oil markets in a different way. 9 MS. KHANNA: But it also goes back 10 to John Conti's point about taking the 11 discussion up to a different level. If you 12 can frame it in the sense, look, here is how 13 the market is changing, and we have a whole 14 different set of questions we want answered, 15 it might help you overcome the hurdle which 16 you are perceiving. 17 MS. ANDERSON: I keep harking back 18 to if there are different clients for data 19 series, they may be people like you who are 20 using the raw data into a model you're 21 creating that looks a whole lot different 22 than any models we're doing, and then the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 57 1 client base that is for model output, which 2 is a different sets of clients. And so our 3 data are not only for input for our models, 4 but serve their own purpose, their own end 5 result, and trying to find areas where what 6 we're collecting for that audience is the 7 same as what we're using internally, and 8 that's where you have a really great 9 situation. But there's an awful lot of data 10 we collect that doesn't go directly into a 11 short-term or a long-term model, but is used 12 purely as output for answering perhaps more 13 basic questions but not integrated questions 14 are maybe not used in a modern framework. 15 And is that client any less valuable to us or 16 valuable to EIA than the clients that use the 17 model base that ----- 18 MS. KHANNA: I'd be willing to 19 hazard that the same users that use your raw 20 data for their modeling would be upset if you 21 actually stopped providing some of the series 22 that you are because you're both users of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 58 1 your raw data as well as your model output 2 for the most part. 3 MS. ANDERSON: That's right. 4 DR. BURTON: I'm going to suggest 5 something that probably is close to heresy. 6 The discussion and the decision processes 7 that you're talking about take place in the 8 private sector all the time, but they're 9 guided by pricing. If you're constrained in 10 what you're able to do, you provide the 11 products that people are willing to pay the 12 most for and you observe that. You don't 13 have -- at least given the current structure 14 -- the ability to do that. You don't have 15 the price signals from the users, both 16 internal and external, that a private firm 17 would have, but that's what you're trying to 18 do, is essentially develop a decision process 19 to substitute for what a market would do in a 20 private setting. 21 DR. NAGARA: Could I ask a specific 22 question? Are there some examples where you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 59 1 felt that if you can run this survey instead 2 of once a year, twice a year, that would make 3 a tremendous impact on your model accuracy, 4 for example? I'm wondering whether there's, 5 based on all the models more or less here 6 today, I just wish I had more data on this 7 particular one? I think that in order to 8 answer Nancy's question, to really bring it 9 up to a higher level of discussion, if you 10 have a example of something ---- say, hey, 11 listen to me, you are doing pretty good here, 12 but you can to four times as well by spending 13 just a little bit more. 14 MS. ANDERSON: Well, maybe the best 15 example is on the consumption survey side, 16 that if the Short-Term Energy Outlook had 17 annual consumption data from the Residential 18 Energy Consumption Survey, we would be more 19 accurate in our ability to predict heating 20 expenditures or fuel usage across the year by 21 region. Yes, that's probably an area where 22 we would enhance the accuracy of our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 60 1 estimates. Again, we get back to -- it's 2 unlikely that we're -- I mean, we're going 3 the opposite direction. We're collecting it 4 in less-frequent intervals ----- some years. 5 Most of the other data series that drive the 6 Short-Term Energy Outlook are collected on a 7 monthly or weekly basis, and we're 8 constrained by the monthly basis for the 9 Short-Term Energy Outlook, and I don't think 10 we want a weekly Short-Term Energy Outlook. 11 Maybe we do, but I keep hoping we don't. 12 (Laughter) 13 DR. NAGARA: Please don't say that. 14 MS. ANDERSON: But certainly there 15 are some data that we've cut back on, 16 definitely on the consumption side, that I 17 think would help both of our modeling 18 efforts, the long-term and the short-term. 19 MS. KIRKENDALL: The only problem 20 with that is that the consumption surveys are 21 among the most expensive ones we run, and 22 that's why the frequency has been reduced. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 61 1 Not that they weren't useful. 2 MR. CLEVELAND: Responding to 3 Mark's comment, which I found interesting, 4 thank God our decisions aren't driven by 5 market considerations, and if they ever start 6 becoming that, you're in deep trouble. Look 7 at the kids' page, the hits on the kids' 8 page, how important -- you're in a really 9 difficult place because energy now, more so 10 than ever, is a huge business, and energy 11 data, billion-dollar decisions, daily, hinge 12 on your information. But energy also is 13 becoming more important in a broader social 14 context. The role of energy in education -- 15 look at the number of hits on the kids' page 16 -- is amazing. Information about climate 17 change and CO2, the increasing role of energy 18 in international markets, which aren't these 19 short-term financial issues, are also 20 increasingly important. And so you really -- 21 I would argue those issues need serious 22 attention as well as the stuff that people BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 62 1 are banging on your door for every day, and 2 that puts you in a -- in one sense, a good 3 position because energy is more important 4 throughout all aspects of society, but in a 5 budget-constrained environment it's a 6 difficult row to hoe. 7 MR. WOOD: I've certainly said in 8 front of this group -- and I was here before 9 the beginning, I think, so that's more than 10 30 years -- that the analytical needs of EIA, 11 as mandated -- those areas where you have a 12 responsibility for -- and anything we can 13 justify a reason for including -- ought to 14 drive the data collection. In practice, over 15 time, sometimes that doesn't -- since a 16 data-collection enterprise may be quite 17 expensive, to change them -- and they always 18 have constituencies, if only internal -- is 19 sometimes difficult. But nevertheless, the 20 analytical needs of the organization and the 21 public for energy data and analysis are what 22 ought to drive the data collection process, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 63 1 and that means periodically something new 2 might have to be added and something old 3 might have to be dropped, although I would 4 caution against dropping a lot of things just 5 because they went down in Web hits during one 6 quarter -- the proverbial drive my company so 7 that the market numbers are hit for this 8 quarter because long-term data series that 9 extend back 50 and 100 and 150 years are 10 really important if you think you're going to 11 be giving insight into questions which may 12 unfold over 25 or 50 or 100 years. 13 Secondly, I'm going to ask a 14 question tomorrow, so I might as well start 15 asking it today, on how in detail you 16 actually make a decision whether to add 17 something or cut something. And in 18 particular, in the 914 survey last year, in 19 large part, potentially to get through the 20 OMB process, we decided that we needed to get 21 natural gas production data for a couple of 22 major producing areas, and so did we want the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 64 1 number one area or did we want the top five 2 or the top ten, and top ten sounded like a 3 big number, so we ended up with six -- and 4 everybody else. Six is a nice number; it's 5 divisible by two and one and three, but the 6 compelling logic is slightly not totally 7 obvious. On the other hand, there are also 8 budget questions: The more states we collect 9 in detail, the larger perceived burden it is 10 on the respondents, and two, the larger 11 budget requirements for EIA. And so what we 12 did, and what drove it was, these are the top 13 six producing areas, so we're going to do 14 them individually because we ought to pay 15 attention to them. I think I'm going to 16 suggest we do more like the top ten when we 17 modify the 914, and there's something that 18 goes with some of the opening remarks. We 19 think there are going to be some major 20 increases in production from unconventional 21 oil resources. Those states right now do not 22 produce very high amounts of oil, but we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 65 1 assume one or two of them will, and we assume 2 -- or I assume -- that we ought to add them 3 to the list of the states we are formally 4 going to collect data from. And so that is 5 how to -- where do we cut it off? There are 6 37 states that produce oil and gas, it seems 7 like the top ten would be a useful group to 8 have detailed state-level information on, and 9 we're going to do that in the time period of 10 a month. The next question is, if we assume 11 that the future of oil production is going to 12 ride on unconventional resources, should we 13 add a couple of states right now that we 14 assume are going to change their production 15 levels in the immediate future or in the next 16 decade. 17 MR. BINGHAM: Can I ask a quick 18 question? I'm Derek Bingham, by the way. So 19 I'm just looking at the remarks from last 20 time so I -- this is my first time here, so 21 bear with me -- but I'm just kind of curious 22 -- so two people commented about prediction BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 66 1 errors, I guess, and I'm curious how much 2 time is spent kind of explaining the 3 prediction errors in some of these models. 4 And the reason I would say that, for instance 5 -- this may correlate with what Lee's saying 6 -- is that if you didn't include these areas 7 that may be quite important, then at least 8 you could go back after the fact say, well, 9 it turns out that these areas are now 10 becoming an important part of the oil supply 11 or whatever it is that they're producing. 12 They're producing whatever it is. And I 13 guess my question is for looking at and 14 explaining why you missed in a model and then 15 either building components into the model for 16 that reason because they're now important 17 where before they weren't. Are there large 18 efforts in that respect? 19 MS. ANDERSON: Well, I can talk 20 about what we're doing in the short-term 21 context, and John can talk about for the 22 longer term. This is the program that Nancy BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 67 1 alluded to in the beginning in her bullets 2 that we are -- her office is now and her team 3 is reviewing as a proposal we have on the 4 table for looking at forecast areas for about 5 40 key variables that come out of the 6 Short-Term Energy Outlook, precisely to 7 answer the questions you're asking, try to 8 understand why we're off, and if that's a 9 missing data problem, could we fix it by more 10 data, could we fix it by changing the model, 11 as well as just informing the public of how 12 accurate are we and explaining that in some 13 cases you can't get any more accurate than 14 what we're doing because these are often due 15 to weather variations and things that we're 16 not going to really enhance our predictive 17 capability of. 18 So, in fact, we're trying to do a 19 better job of explaining to our readership 20 what our errors are, as well as to ease that 21 information back to our modelers to try and 22 either fix the models or improve data, data BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 68 1 accuracy -- whether that's a problem with the 2 accuracy of the actual data, the timeliness 3 of the data, or, in fact, missing data, we 4 don't know yet, but we would certainly want 5 to move the program in that direction. John 6 already does some of that in the long-term 7 model. 8 MR. CONTI: We've had a forecast 9 evaluation report, I believe, that EIA's 10 published for about a decade and a half now, 11 and it doesn't go into as many variables as 12 Margot does or will; we have more aggregate 13 variables. But we've done it for every AEO 14 since 1982, and so some of those earlier 15 AEOs, we actually have, where our 15- or 16 20-year predictions are actually data values 17 now, and we go back and we assess why we 18 didn't predict -- well, if we notice big 19 discrepancies, we try and go back and assess 20 why we didn't predict it correctly, and we 21 see if we still have that same faulty logic 22 and models we use today. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 69 1 We do tend to look at the most 2 recent years to see if we're trending 3 differently from the actual numbers and to 4 try and understand that. But we don't sort 5 of have -- it's just a matter of resources 6 and how much resources you allocate to each 7 of these efforts, but we do do that. 8 Actually, one question that I was going to 9 ask the committee somewhere in the next 10 couple of days was in this forecast 11 evaluation report we look at the sum of the 12 absolute errors to give an indication of how 13 far we're off. Well, that's a good idea of 14 how far you're off, but I'm always interested 15 also in bias. I'm not so worried if we're 16 equally dispersed high and low, but I'm more 17 worried if we have a bias to our forecast. 18 So it was a question I was going to -- what's 19 a good measure of bias? I was thinking of 20 maybe putting out graphs where people could 21 visualize it. To me, that's the most 22 instructive rather than some other BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 70 1 statistical single number measurement. 2 MS. ANDERSON: And we'll be using a 3 bias measure in our estimates. It's one that 4 we're proposing. 5 DR. HENGARTNER: One of the nice 6 things about the model is that it's not too 7 picky of what kind of data it needs. I find 8 that the many applications that -- Rolling 9 Stones are right, you don't always get what 10 you want, but you get what you need. 11 In a way the question I'm asking 12 is, we are very focused on continuing 13 historical series, which is a good thing, but 14 in terms of improving our models and 15 predictions, there are other sources of data 16 that may be available out there essentially 17 for free -- maybe not exactly free, but close 18 to free -- which are not exactly what we 19 want, but they're close enough that they 20 could be incorporated into the modeling 21 effort. That's what models are for, right? 22 To borrow strength of things that kind of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 71 1 inform us -- and yes, it's not a perfect 2 match, but it has value. 3 And so while I don't have any 4 specific examples -- I see John nodding his 5 head -- I'd like you to tell me, yes, 6 actually, that's something I've been thinking 7 about. 8 MR. CONTI: I'm sure there are a 9 lot of analysts in the audience here, but I 10 know there are variables or data we use that 11 EIA does not collect, that we purchase from 12 outside sources, and only because it's just a 13 lot cheaper to do. It's used commercially, 14 so somebody picks it up and sells it to a lot 15 of people, and we just buy it and procure it. 16 So I don't know if that was the direction 17 you're going in, but for example, in 18 commercial floor space, we tend to buy those 19 projections. We buy a lot of our 20 macroeconomic information. And we use other 21 sources from other agencies. So we purchase 22 data where it's cost effective, but if it's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 72 1 just raw energy data that no one else 2 collects, and we sort of have responsibility 3 for more or less -- if it were to be 4 collected, we would be the ones to collect it 5 -- and it gets back to prioritizing -- how do 6 you prioritize which data you collect? 7 For example, John mentioned it 8 became very relevant for us to collect this 9 weekly natural gas storage information. I 10 don't need that in my long-term models, 11 Margot doesn't probably need weekly in her 12 monthly models, but I --- most of us would be 13 lynched if we said we shouldn't collect it 14 weekly because, no, we don't use it. No, 15 there are a lot of people outside of EIA that 16 would kill us if we stopped collecting it on 17 a weekly basis. 18 DR. HENGARTNER: But I'm trying to 19 answer more the question, what data do we 20 need or what does our model inform us about 21 data needs. And as I said, there is this 22 dichotomy of two things, and you again go BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 73 1 back, prioritize, what are the questions you 2 want to answer -- 3 MR. CONTI: Right. 4 DR. HENGARTNER: -- what are our 5 clients' need in terms of raw series, but if 6 you're going to go and really answer the 7 question, what does the model tell us about 8 our data needs -- 9 MR. CONTI: Right. 10 DR. HENGARTNER: -- then you are 11 entitled, and I would highly encourage you, 12 to think broadly and in new ways because 13 that's what models are for. And raw data is 14 something else, and I don't want to discount 15 your need of raw datas -- and I'm not 16 advocating that either, so -- 17 MR. CONTI: Right. 18 DR. HENGARTNER: -- don't get me 19 wrong here -- it's just -- I'm trying to 20 answer very pointedly the question that I 21 think was set out in the beginning. 22 MR. CONTI: Okay. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 74 1 DR. HENGARTNER: Yes? 2 MR. WATSON: Bill Watson with 3 CNEAF. I'd like to come back to the issue of 4 use of more data by external parties, 5 especially people that are trying to do their 6 own modeling. We do have at EIA -- at least 7 from my perspective -- a fairly systematic 8 way to understand what the data needs are of 9 people that like our AEO modeling effort, 10 come to our AEO outlook meeting, and we get a 11 fair amount of feedback from them. John has 12 already talked about an assessment that would 13 look at the data and the data needs for any 14 enhancements or changes in the modeling 15 efforts in that area. I'm not aware -- and 16 this is partly a question, partly a statement 17 -- that we have anything like that on the 18 side of the users of our survey data. We 19 don't have an annual meeting, or we don't 20 have a systematic way to communicate with 21 people that just use our data series. Many 22 of them would be modelers, and they would BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 75 1 have some good opinions probably and good 2 recommendations about whether we're missing 3 some data gaps there from their perspective. 4 So I guess maybe the question I would ask to 5 the committee is, do we need to have some 6 sort of balance here between the internal use 7 of the data for modeling and the use of the 8 data by external modelers and some way to 9 assess on the external side what the data 10 gaps may or may not be. 11 Just to give an example, in the 12 coal area, especially in the last 12 to 24 13 months, we get a continuing series of calls 14 from people asking about coal prices: What 15 data do you have on coal prices; what data do 16 you have on coal transportation costs; what 17 data do you have, other than your annual data 18 that's with a two-year lag, on coal stocks. 19 And so we're hearing this from the outside 20 community, but it's not good, I think, to 21 base recommendations on ad hoc information 22 such as telephone calls; you need to do it BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 76 1 systematically. 2 MR. CLEVELAND: Well, that raises 3 the question of how well do you know who your 4 client base is. I mean, there's Web hits and 5 downloads, but that's pretty limited kind of 6 information. And so I agree the Annual 7 Energy Outlook meeting, which I have 8 participated in, is a good way to get 9 feedback from at least some slice of your 10 users for that, but it seems to me that some 11 other type of form -- we don't necessarily 12 have to do it every year -- whereby you try 13 to identify who these other users are of the 14 various types of data you collect, bringing 15 them together and getting some of that kind 16 of feedback. But the question is how do you 17 think you know who your users are? We know 18 the squeaky wheels are the ones that -- we 19 know who the big squeaky wheels are, but 20 that's clearly not who we want to be making 21 all our decisions around. 22 DR. BURTON: The question that you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 77 1 asked, I think is where I thought this 2 discussion was probably going to go to begin 3 with. And one thing that I find is fortunate 4 -- and may be a way to sort of sell modeling 5 efforts -- is by doing modeling internally, 6 you're likely to run into a lot of the same 7 problems that we as external users encounter, 8 and the fact that you do do modeling 9 internally is a way to inform the process 10 about what data is most useful. 11 MR. CLEVELAND: So you do do 12 customer surveys? What kind -- what are 13 those like? 14 MS. KIRKENDALL: We've done -- 15 let's see, when are we done? Where's 16 Colleen? I want to -- we've participated in 17 the American Customer Satisfaction Index, and 18 that's a survey that's associated with the 19 Web so it's a random selection and people are 20 invited to participate in the American 21 Customer Satisfaction Index. That's one way 22 we've done surveys of people who are on our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 78 1 Listservs, thinking that people who sign up 2 to be on a Listserv are keenly interested in 3 those products they've signed up for. And of 4 course, it's not complete, but -- 5 MR. CARUSO: And we do the ones 6 where -- the pop-up surveys for users. 7 MS. KIRKENDALL: We do pop-up 8 surveys for users. 9 MR. FREEDMAN: We've done surveys 10 of the NEMS conference -----. 11 MS. KIRKENDALL: Yeah, we do 12 surveys of the NEMS conference. They do that 13 regularly to monitor how well the conference 14 runs. We've also done surveys -- it's a 15 sample from the Listserv for the AEO and for 16 the STEO -- to get some input from them on -- 17 DR. BURTON: Do you get a good 18 response rate from them? 19 MS. KIRKENDALL: Well, no. 20 MS. ANDERSON: No. 21 MS. KIRKENDALL: About 25 percent. 22 MR. ANDERSON: Nor is there often BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 79 1 space to get really at some of the detailed 2 questions you want to get at that maybe if 3 you brought people together in a focus-type 4 setting, you might get -- and you -- 5 MS. KIRKENDALL: Yeah. 6 MR. CONTI: I was going to say, we 7 do that with -- we have a follow-up on the 8 NEMS conference where we get sort of really 9 heavy users of our models and our forecasts, 10 and we do in-depth surveys with them and try 11 and understand their needs and faults they 12 have or problems they've identified. And so 13 we do what we do at the conference for 14 everybody, and we get a fair response rate, 15 and then we go into detail with some people 16 from a broad array of areas that do energy 17 modeling. 18 MR. SINGPURWALLA: Do the people 19 outside the Energy Department, are their 20 issues and needs different than the ones 21 inside? 22 MR. CONTI: Sometimes they're more BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 80 1 specific. Yeah, they have a more specific 2 interest and they want to go into greater 3 detail than we do. But a lot of times, we 4 tend to focus on people that are trying to do 5 similar things. Sort of broad national-level 6 forecasting. 7 MR. SINGPURWALLA: Right. 8 MS. ANDERSON: But have we ever 9 done anything that might get the users of the 10 consumption surveys together in a room or in 11 a conference and ask specifically about the 12 coverage and the use of the data that we are 13 putting out, whether it's -- how we put it 14 out or how frequently we put it out or the 15 depth of coverage, and I don't think we've 16 done that on all of our surveys, we've 17 probably done it on some and asked more 18 Web-based questions. But we probably haven't 19 gone into as full detail on some of our 20 surveys; it might be beneficial to do that. 21 MR. CLEVELAND: Yeah. I don't 22 think the issue is NEMS so much -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 81 1 MS. ANDERSON: Yeah. 2 MR. CLEVELAND: -- because I think 3 you could do a pretty good job of getting 4 feedback on that, it's the other broader 5 array of data products whose constituencies 6 are much flatter and broader and less 7 well-identified than I think is the issue 8 here. 9 MR. CONTI: Some of those issues do 10 come up. We typically have workshops -- and 11 we meet a couple times a year -- in the 12 modeling area, and some of those go back to 13 -- like the industrial workshop tends to go 14 back to the Manufacturing Energy Consumption 15 Survey, and we get people from EPA and 16 sometimes outside the Alliance to Save 17 Energy, and there'll be discussions about the 18 MECS at that point in time -- and Tom's been 19 there and people from your office have been 20 there to talk about the MECS. So it's not 21 focused on the MECS, but it's focused on 22 industrial modeling and some of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 82 1 discussions deal with the MECS. 2 MS. KIRKENDALL: We do -- when 3 we're considering redesigning surveys -- it 4 used to be at least -- but when electric 5 power was going to redesign their surveys, 6 and I think they did this in natural gas, too 7 -- they had a series of focus groups with 8 users, or different kinds of users. So you 9 might have our own users and then other 10 federal users and other groups of users that 11 you could identify to come in and talk about 12 data needs. So those kinds of meetings would 13 be held at the time you want to redesign the 14 forms, so it's not an ongoing effort. 15 MS. FORSYTH: Those same people 16 might also be used, not just to talk about 17 data needs, but to talk about other data 18 users that might be beyond the scope of the 19 current data collection so that they'd be a 20 source of perhaps exploratory data in that 21 kind of question that you're asking. There 22 are some that will listen. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 83 1 DR. HENGARTNER: We're 10 minutes 2 early. Any more comments? 3 MS. KIRKENDALL: Well, if you think 4 of anything, let us know. 5 MR. CLEVELAND: I do have one other 6 one, actually. And I'm wondering if the 7 external review panel -- this seemed to be 8 one of the fundamental questions that they 9 must have talked about or -- did they collect 10 any other information in their review about 11 whether or not -- how we go about collecting 12 data and identifying who the users are and so 13 on and so forth? 14 MS. KIRKENDALL: They looked at all 15 of our -- we put together a lot of 16 information from our customer feedback, all 17 the vehicles that we have, so they got 18 summaries of all of that. 19 MS. ANDERSON: They certainly made 20 some suggestions on things we ought to take a 21 second look at and really ask ourselves why 22 are you collecting this, is it still relevant BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 84 1 to collect this information, and for us to 2 really ask ourselves that question we're 3 going to have to do precisely what you asked: 4 Make sure we fully understand who's using it 5 and what the costs are of scaling back or 6 eliminating entirely. They did feel that due 7 to change in mission or change in the sector, 8 there are some things that we're collecting 9 that are really not relevant to the evolution 10 of the energy sector, or could be augmented 11 by private sector data, and it may, in fact, 12 be a net savings to EIA. So I think that's 13 an interesting discussion. 14 MS. KIRKENDALL: Yeah, but they 15 really didn't go into detail on our core 16 program; they sort of assumed that the core 17 program and the data that we need to inform 18 the energy users was, if not wonderful, at 19 least it was necessary, and they didn't get 20 into the details because it wasn't a long 21 enough -- 22 MR. CLEVELAND: Sure. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 85 1 MS. KIRKENDALL: -- time period to 2 do that, really. So that there were a few 3 things they identified as low-hanging fruit 4 that they weren't sure were relevant. 5 MS. KHANNA: I just had one 6 cautionary plea as you consider using data 7 collected by other sources, especially 8 private sector, is to focus a lot of data on 9 the accuracy and robustness of the data 10 collection process as well as the final data 11 that you get. And a great advantage of EIA 12 collecting its own data is you know exactly 13 how it is collected and what the pitfalls and 14 strengths of it are. If you buy -- as a data 15 user, I mean, I'm always worried about that. 16 MR. RUTHERFORD: We have five 17 minutes to go until the planned time, so I 18 thought I'd just add a comment. I thought 19 Bill's comment about thinking creatively 20 about ways of soliciting information about 21 how data is used is an intriguing one. I'm 22 thinking there are two ways that you could BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 86 1 leverage this: One would be to sort of 2 organize a special issue in the Energy 3 Journal or something about a specific topic 4 that uses, say, natural gas data from DOE 5 would be the theme to try to leverage input 6 from academics who have worked in the area. 7 Also, if you're thinking about sort of model 8 revisions at DOE, having a chance to sort of 9 get a survey up close of what's been done 10 with your data would be useful for thinking 11 about how to move forward. And the other 12 thing is perhaps prizes. I don't know, you 13 could have some sort of -- anyway, there's -- 14 I just -- I think it was just -- this is sort 15 of off the top of my head; it just seemed 16 like an interesting idea. 17 MR. CLEVELAND: Yeah, I like the 18 idea of this special issue. An analogous 19 thing on the modeling front might be to get 20 Hill Huntington and the Stanford Energy 21 Modeling Forum folks together and maybe have 22 one year -- one cycle of the modeling forum BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 87 1 to essentially give you guys feedback on what 2 you guys are doing. I think that would be a 3 very interesting use of some pretty 4 high-powered talent. 5 MR. CARUSO: Yeah, we've worked 6 very closely with Hill and the EMF, so that 7 would be good. I like both of those 8 suggestions. 9 DR. HENGARTNER: Okay, seems this 10 discussion has ran its course. We're 11 fortunate to be ahead of schedule, and this 12 is a good thing. I'd like to take a break 13 and remind you that we're going to continue 14 afterwards in break-out sessions in Room 7-E 15 -- that's here; Derek Bingham, Johnny Blair, 16 Moshe Feder, Barbara Forsyth, myself, Walter 17 Hill, and Darius Singpurwalla should remain 18 here. And Mark Burton, Mr. Cleveland, Jae 19 Edmonds, Neha Khanna, Nagaraj Neerchal, and 20 Tom Rutherford will go downstairs to Room 21 5-E. And we will reconvene in this room 22 afterwards for discussion from these BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 88 1 break-out sessions at 11:35. So we have 2 about 15 minutes break, and before we all 3 leave this room, those who trickled in, one 4 last time, please identify yourselves so we 5 can have you on the record. Thank you very 6 much. 7 MR. GRUENSPECHT: I'm Howard 8 Gruenspecht with EIA. 9 (Recess) 10 DR. HENGARTNER: This is the next 11 break-out session. It's going to be the 2006 12 MECS. By the way, I was told that those 13 microphones are picking up just fine if you 14 project towards them. I'm just relaying the 15 message I heard from the master. So this is 16 the 2006 MECS looking at past performance 17 statistics to motivate new method of 18 collection, and it's Bob Adler and Tom Lorenz 19 who are going to be leading our discussion -- 20 at least giving us an introduction -- and 21 hopefully we're going to all chime in and 22 argue and discuss some more after this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 89 1 presentation. So, welcome Tom. Thank you. 2 MR. ADLER: Okay, I'm Bob Adler, 3 and that's Tom Lorenz. First of all, can 4 everybody hear me? We will be talking about 5 how the 2002 MECS performance statistics kind 6 of justify and motivate the 2006 MECS and our 7 big redesign. We want to thank Rick Hough, 8 Vicki Haitot, and Stacey Cole, and since 9 Susan's here, I'll thank her too, Susan 10 Bucci -- 11