AMERICAN STATISTICAL ASSOCIATION (ASA) MEETING OF THE ASA COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION (EIA) OF THE UNITED STATES DEPARTMENT OF ENERGY Washington, D.C. Thursday, October 20, 2005 2 1 COMMITTEE MEMBERS: 2 NICOLAS HENGARTNER, Chair Los Alamos National Laboratory 3 MARK BERNSTEIN 4 RAND Corporation 5 JOHNNY BLAIR Abt Associates 6 MARK BURTON 7 University of Tennessee 8 JAE EDMONDS University of Maryland 9 MOSHE FEDER 10 Research Triangle Institute 11 BARBARA FORSYTH Westat 12 WALTER HILL 13 St. Mary's College of Maryland 14 NAGRAJ NEERCHAL University of Maryland 15 DARIUS SINGPURWALLA 16 LECG 17 RANDY SITTER Simon Fraser University 18 EIA PERSONNEL: 19 BOB ADLER 20 MARGOT ANDERSON 21 COLLEEN BLESSING 22 TOM BROENE BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 3 1 EIA PERSONNEL (CONT'D): 2 GUY CARUSO 3 JOHN PAUL DELEY 4 ALOULOU FAWZI 5 HOWARD BRADSHER-FREDRICK 6 STAN FREEDMAN 7 CAROL FRENCH 8 DWIGHT FRENCH 9 BILL GIFFORD 10 BEHJAT HOJJATI 11 SUSAN HOLTE 12 ALETHEA JENNINGS 13 RAY KASS 14 ROBERT KING 15 NANCY KIRKENDALL 16 TOM LORENZ 17 RUEY-PYNG LU 18 PRESTON McDOWNEY 19 RENEE MILLER 20 MICHAEL MORRIS 21 ERIK RASMUSSEN 22 MARK RODEKOHR BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 4 1 EIA PERSONNEL (CONT'D): 2 TOM RUTHERFORD 3 BHIMA SASTRI 4 MARK SCHIPPER 5 JOHN STAUB 6 LAWRENCE STROUD 7 AMY SWEENEY 8 PHILLIP TSENG 9 KEN VAGTS 10 ANGELA VEITCH 11 SHAWNA WAUGH 12 PAULA WEIR 13 ALSO PRESENT: 14 SUSAN LISS Federal Highway Administration 15 NANCY McGUCKIN 16 United States Department of Transportation 17 WILLIAM WEINIG ASA 18 KATHLEEN WERT 19 ASA 20 21 22 * * * * * BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 5 1 C O N T E N T S 2 AGENDA SESSION: PAGE 3 Administrator's Remarks 13 4 Committee Updates: 5 Since Spring 2005 Meeting 31 6 Results of Simulation Study 39 for the EIA-914 7 EIA's Regional Short-Term 46 8 Energy Outlook 9 Short-Term Forecasting Performance 71 Measures and Accuracy Evaluation 10 Preserving EIA Trustworthy 117 11 Datasets, Model Documentation and Contextual History 12 Learning from the Past: 187 13 Updating Data Quality Efforts 14 Can Discrepant Estimates 232 Be a Good Thing? 15 Survey Self-Assessments 277 16 17 18 * * * * * 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 6 1 P R O C E E D I N G S 2 (8:32 a.m.) 3 DR. HENGARTNER: Ladies and 4 gentlemen, oyez, oyez, oyez. Good morning, 5 everybody, welcome to this meeting. This is 6 an ASA, not an EIA, meeting. This committee 7 periodically provides advice to the EIA. 8 This meeting is open to the public and public 9 comments are welcome. Time will be set aside 10 for comments at the end of each morning and 11 afternoon sessions. Written comments are 12 welcome as well and may be sent to either the 13 ASA or the EIA. 14 All attendees including guests and 15 EIA employees should sign in at the register 16 in the hall and include their e-mail 17 addresses. So there's a little bit more of 18 the formalities. The rest rooms are located 19 either on one end of the hall at this end and 20 the other end of the hall. 21 The telephone in this room shares a 22 line number. Let me give you that. It's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 7 1 202-586-3071 and you can also get messages 2 through the following number if you need, 3 586-6273 and 586-6202 and probably it might 4 be a nice gesture to turn off cell phones. 5 Kathleen Wert with the ASA Meeting 6 Department is here and available for the 7 committee members' questions on expense 8 reimbursements, and I'm sure she will be 9 happy to help us, she has in the past been 10 very helpful. In commenting each committee 11 member is asked to speak toward a microphone. 12 The microphones are here if you don't speak 13 clearly and loudly we will be reminded. The 14 transcriber will appreciate that. 15 The speakers from the audience are 16 asked to use the microphone that's over 17 there. We did have a portable one but 18 because of the echo I think that might be 19 better. Also, if you need help Bill Weinig 20 will help you with this. 21 Okay, the announcements. First I'd 22 like everybody to introduce themselves in the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 8 1 room. I will start with myself. My name is 2 Nick Hengartner and I'm the chair of the 3 committee. 4 MS. KIRKENDALL: Nancy Kirkendall, 5 Director of the Statistics and Methods Group, 6 EIA. 7 MR. CARUSO: Guy Caruso, 8 Administrator at EIA. 9 DR. FEDER: Moshe Feder, RTI, 10 committee member. 11 MR. BERNSTEIN: Mark Bernstein, 12 member of the committee. 13 DR. BURTON: Mark Burton, 14 University of Tennessee. 15 DR. EDMONDS: I'm Jae Edmonds at 16 the Pacific Northwest National Laboratory. 17 MS. FORSYTH: I'm Barb Forsyth. 18 I'm a member of the committee. 19 DR. SITTER: Randy Sitter, Simon 20 Fraser University. 21 MR. McDOWNEY: Preston McDowney, 22 Statistics and Methods Group, EIA. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 9 1 MR. TSENG: Phillip Tseng, SMG, 2 EIA. 3 MS. BLESSING: Colleen Blessing, 4 National Energy Information Center, EIA. 5 MS. MILLER: Renee Miller, 6 Statistics and Methods Group, EIA. 7 MS. FRENCH: Carol French, Office 8 of Oil and Gas, EIA. 9 MS. VEITCH: Angela Veitch, 10 Petroleum Division, EIA. 11 MR. SASTRI: I'm Bhima Sastri. I'm 12 with the Office of Integrated Analysis and 13 Forecasting, EIA. 14 MR. STAUB: John Staub, EIA. 15 MS. HOJJATI: Behjat Hojjati, EIA. 16 MS. JENNINGS: Alethea Jennings, 17 Statistics and Methods Group, EIA. 18 MR. FAWZI: Aloulou, Office of 19 Integrated Analysis and Forecasting, EIA. 20 MS. WEIR: Paula Weir, EIA. 21 MR. LORENZ: Tom Lorenz, EIA. 22 MR. BRADSHER-FREDRICK: Howard BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 10 1 Bradsher-Fredrick, EIA. 2 MR. VAGTS: Ken Vagts, EIA. 3 MS. HOLTE: Susan Holte, EIA. 4 MS. SWEENEY: Amy Sweeney, EIA. 5 DR. HENGARTNER: Thank you very 6 much. For your information, under the terms 7 of the DoE grant to the ASA Nancy Kirkendall 8 here may chair but must attend each meeting. 9 And she's authorized to adjourn the meeting 10 if she determines this to be in the public 11 interest. She must approve all meetings of 12 the advisory committee and every agenda. 13 Also she may designate a substitute in her 14 absence. 15 I think that we have a good agenda. 16 We've all seen a little bit of what's going 17 to be on the program. I'm looking forward to 18 it. There are going to be four breakout 19 sessions and a few presentations here. 20 We have also one new committee 21 member I hope to be able to introduce to you 22 shortly and with both sadness and pleasure BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 11 1 I'd like to recognize the participation of 2 two of our former members. It's going to be 3 their last session. One is Mark Bernstein 4 here and the other Randy Sitter. Both of 5 them have been with us for six years and 6 their contributions have been valuable to the 7 committee and we'd like to recognize this. 8 MR. CARUSO: Thank you, Nick. Let 9 me start with Mark since he's the closest 10 one. There's couple of small tokens of 11 appreciation for you, Mark. There's one from 12 Nancy and me and this one's from Secretary 13 Bodman. So, Mark, thank you. 14 MR. BERNSTEIN: Thanks so much. 15 MR. CARUSO: We'll miss you here. 16 MR. WEINIG: Could you and Mark 17 come up towards the front so we can get a 18 couple of additional pictures? Thank you. 19 DR. HENGARTNER: We're thinking of 20 putting Mark's picture on the wall here. 21 The second member who's going to 22 depart us and whose last meeting it is as BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 12 1 well is Randy Sitter and I'd like also to ask 2 us to acknowledge him. 3 DR. SITTER: Thank you very much. 4 DR. HENGARTNER: So just a note on 5 the program. The first address this morning 6 will be done by Guy Caruso on the EIA general 7 state of affairs. Then Nancy Kirkendall as 8 the past director of the EIA Statistics and 9 Methods Group will tell us about what's going 10 on. And other summary presentation will be 11 from Preston McDowney, Margot Anderson, and 12 that's it for the locals. Good. Now, the 13 important thing, lunch. 14 DR. SITTER: Oh, already? 15 DR. HENGARTNER: Lunch will be as 16 usual served on the first floor at 12:25. 17 It's in the same corridor. Just follow me. 18 I'll find it. And also there's going to be 19 dinner tonight in honor of Mark. We are 20 going to go to a pub and I know that Bill 21 Weinig has sampled them and has chosen us the 22 best pub in DC. To make life easy for our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 13 1 organization can I have a show of hands of 2 who will be attending? 3 Thank you very much. Tomorrow 4 morning there's going to be breakfast at 7:45 5 and we will resume the meetings here at 8:30. 6 Finally, just a reminder, I am the Chair and 7 my responsibility is to keep us on time and 8 therefore we're 20 minutes late. Don't 9 worry, we'll catch up, so thank you very much 10 and I'd like to ask Guy to give us the state 11 of affairs. 12 MR. CARUSO: Thank you, Nick, and 13 good morning to the members of the committee 14 and everyone else joining us this morning. 15 It's a pleasure always to welcome the ASA 16 Energy Statistics Committee and, as I have 17 said many times, we find the work and advice 18 of your committee extremely useful and that 19 will show up in my remarks this morning as 20 well as presentations throughout the next two 21 days. It's been put to some very practical 22 use just in recent days in our analysis of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 14 1 the hurricanes and in our short- term outlook 2 and winter fuels outlook particularly. It's 3 always sad to see members of the committee 4 leave but Mark and Randy have been with us 5 for six years and they had extremely useful 6 comments and then some of the other 7 activities with Randy, some of Randy's 8 students and other activities, so we wish 9 them well and we'll be seeing you in other 10 venues. 11 Let's very quickly go over what's 12 been a very busy six months since we've last 13 met. Most recently, of course, our services, 14 our analysis, our data have been under close 15 scrutiny as the analysis in the aftermath of 16 both hurricanes Katrina and Rita have had 17 such a devastating effect on our energy 18 infrastructure in addition to the more 19 important personal tragedies of the 20 individuals involved but we've had a large 21 number of request for our services, for our 22 analysis, for our testimony, and just last BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 15 1 week released our winter fuels outlook, which 2 we'll talk a little bit more about later, and 3 one of the things we've put to very good use 4 is your advice on regional short-term energy 5 outlook. I'll present a couple of examples 6 to that but Margot Anderson, whose office is 7 responsible for producing that analysis, will 8 go into more detail later. 9 And I'll also say a little bit 10 about the redo of our website which we're 11 very excited about and again your comments on 12 that last meeting were taken into account and 13 you'll see that hopefully by the end of 14 November when the new website is up and 15 running. 16 And couple of comments about the 17 budget, the last time we mentioned that we 18 now have an external review team and they are 19 about to meet for the second time and we'll 20 talk a little bit about that and I think 21 that's the highlights. 22 Increased activity due to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 16 1 hurricanes, of course, has been the request 2 we've had on a regular basis for our data and 3 for our analysis and clearly, as always, it 4 moves financial markets, it does shape 5 policy. It was very instrumental in the 6 decisions made by Secretary Bodman and 7 recommendations by Secretary Bodman to the 8 President about utilization of the SPR and 9 the activities of the International Energy 10 Agency. 11 We've had nine testimonies since 12 Labor Day before Congress and numerous 13 briefings and there's been enormous activity 14 on the website, particularly on the situation 15 report that Margot Anderson's office 16 publishes every day, so just when we thought 17 things couldn't get any busier I think again 18 we're being asked to provide the latest 19 information and we're tracking as shown 20 something like this which we released last 21 week what's been happening to US crude oil 22 production in the Gulf of Mexico since the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 17 1 onset of Katrina and Rita. And we, of 2 course, project that out through the winter 3 and it's instrumental in our analysis of 4 where we think crude oil prices are going but 5 perhaps even more critical this winter will 6 be natural gas, which has suffered, I think, 7 even a larger blow to the infrastructure than 8 oil. 9 It's still about 10 percent of our 10 gas productions off line as we stand here 11 today and the restoration is coming perhaps 12 slower than I think many of us thought it 13 would be and it's partly because the on- 14 shore infrastructure for processing the gas 15 and distributing that gas has had such 16 extensive damage. 17 And then, of course, the refinery 18 situation probably is improving a bit quicker 19 than the production of oil or natural gas. 20 Nevertheless, as we've all witnessed as we 21 pull up to the pump, the high price of 22 gasoline but probably more important for this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 18 1 time of the year, heating oil prices are even 2 higher than gasoline, which is relatively 3 unusual for heating oil to be selling at a 4 premium to gasoline but as you can see by 5 this chart that the upper, the darker, line 6 there is heating oil and diesel fuel. Prices 7 will stay above gasoline for the foreseeable 8 future certainly through the winter and 9 that's largely because of the refineries. 10 Ten percent of our refineries still remain 11 offline. 12 So we're expecting very tight 13 markets for crude products of particularly 14 heating oil and jet fuel but gasoline as 15 well. And natural gas the average consumer 16 is going to be paying about $16/1,000 cubic 17 feet in the US and that's at least a 50 18 percent increase over what you as a consumer 19 would have been paying a year ago. 20 So it's a huge political issue and 21 I think there's a lot of frustration in 22 Congress and elsewhere about what to do about BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 19 1 it because there are very few levers in the 2 short term especially for natural gas because 3 there's no world market for natural gas 4 compared with crude oil products. So prices 5 have attracted an enormous increase in the 6 imports of gasoline but in the case of 7 natural gas there are no available spare 8 supplies on the world market. 9 I mentioned that the work that we 10 did with you and the advice you gave us on 11 the regional short-term energy outlook was 12 invaluable and our August STEO included the 13 regional projections and had been very useful 14 in our ability to make specific briefings to 15 individual congressmen or policy makers about 16 the different regions. And so once again the 17 work of this committee has translated itself 18 into a better product for EIA and I think a 19 more useful product in policy analysis and 20 we've had independent expert reviews looking 21 at these models and Margot will go into that 22 in much more detail in a few moments. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 20 1 This is an example of how we've 2 been able to utilize that and we now break 3 down the heating bills, in this case natural 4 gas, by four regions. The model itself, we 5 used nine census regions, as you may recall, 6 but for purposes of presentation we are using 7 four regions but the STEO itself goes into 8 much more detail. So the richness of the 9 data and the analysis has really been useful 10 and, of course, as you all know, every time 11 you give the customer a little more detail 12 they want more and it raises more questions. 13 Yesterday when I presented this to the House 14 Energy Committee those who lived in different 15 regions wanted to know why are we paying 55 16 percent more this year when other regions are 17 only 29 percent so it generates more 18 questions and more work for Margot and her 19 team and she'll talk about that. 20 In a continuing effort to improve 21 the longer term outlook we've been working to 22 improve the National Energy Modeling System, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 21 1 NEMS. We're extending in this year's outlook 2 our projection period to 2030 for the first 3 time. That will be out, if all goes well, 4 about the end of November with a reference 5 case and I think it's going to be pretty 6 interesting because our price assumptions 7 obviously are going to be much higher this 8 year than they were last year and there are 9 some pretty interesting results with respect 10 to the long-term impact of higher prices of 11 oil, of all energy but oil and natural gas in 12 particular, what it means for the mix of 13 generation for electricity, what it means for 14 imports and the demand for products and the 15 economic effect of those prices on what we'd 16 call nonconventional sources of both oil and 17 natural gas such as the oil sands of Canada. 18 Even coal to liquids is breaking some new 19 modeling ground and we'll be adding also the 20 latest Manufacturing and Energy Consumption 21 Survey, MECS. 22 I didn't want to put anything on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 22 1 this chart because it's still very tentative 2 but we've asked the Secretary and now OMB but 3 it's still in the very preliminary stages. 4 We've asked for an substantial increment of 5 money over the next three years to completely 6 redo NEMS. As many of you know, and 7 certainly Jay was part of that, this is a 8 modeling system now about 15 years old and 9 we've been trying to patch it up and you've 10 seen some of those patches in some of the 11 sessions we've had with you. Clearly it 12 needs a real top to bottom look and if we get 13 that funding I think we can make a lot of 14 improvements and incorporate some of the 15 structural changes that have occurred in the 16 industry and a number of other things that 17 we're gleaning from the data and so hopefully 18 I'll be able to report to you in the spring 19 meeting that we now have that approval from 20 OMB and that's the budget for '07 that would 21 be presented by the President in January. 22 I just wanted to give you a little BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 23 1 heads up about that because we'll certainly 2 be asking for your advice and assistance as 3 we move forward on that. It's not just a 4 question of if but when and if we don't get 5 as much money as we've asked for we're still 6 going to try to move, do something, because 7 it's clear that a number of the components of 8 NEMS need updating. 9 The other topic which you've been 10 directly involved in is the National Natural 11 Gas Production Survey. I think it was one of 12 the first items that we discussed when I got 13 here three and a half years ago and it's 14 finally come to fruition. We now have the 15 914 data on the website and we'll be running 16 that in parallel with the existing system for 17 a while to do our data testing and quality 18 control and Ken Botts and Nancy's teams have 19 both worked very hard on this and Preston's 20 going to talk a bit more about natural gas in 21 his presentation. And we're also changing a 22 little bit on the data for the Texas Natural BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 24 1 Gas and we'll talk about that in more detail. 2 I mentioned about the '07 budget. 3 Well, '06 budget here we are what, three 4 weeks into the fiscal year and this budget 5 hasn't been approved yet. As many of you 6 know, it's being debated and we're on a 7 continuing resolution. What was reported out 8 of our appropriations committee was about $86 9 million for '06. It's up $2 million more 10 than '05 and keeping our number of FTEs at 11 369. 12 The '06 budget obviously is going 13 to be subject to pretty intense debate 14 because of the need to pay for the federal 15 costs in restoration and recovery in the Gulf 16 Coast so most people are expecting there will 17 be some impact on everyone's budget so we'll 18 be working closely with our congressional 19 appropriation staffers to encourage them to 20 think of EIA as part of the solution and not 21 part of the problem. So maybe we won't 22 suffer too badly but stay tuned on that one. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 25 1 I mentioned the website being 2 redone. The website hits continue to exceed 3 our expectations and clearly during September 4 and thus far in October has been as much 5 activity as I think we've ever seen and 6 Margot's office's products of the daily 7 situation report and this winter fuels 8 Outlook, STEO, getting a lot of attention. 9 And I mentioned the external review 10 Team, I think, last meeting. We may have 11 mentioned it but I'm not sure. Did we, 12 Nancy? 13 MS. KIRKENDALL: We mentioned it. 14 I think we gave them Denny Ellerman's name as 15 Chair but I'm not sure we told them about the 16 other members yet because I don't think that 17 was fixed yet. 18 MR. CARUSO: This external review 19 Team we're getting, we're anticipating an 20 extremely useful report coming from Denny 21 Ellerman's team and we have outstanding 22 members, Kathy Cooper, who is a former BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 26 1 Undersecretary of Commerce now back in Dallas 2 teaching, Jay Hakes, my predecessor as 3 administrator, Paul Joskow, whom you all know 4 is the head of the MIT Energy and Environment 5 Policy Center, and Phil Sharp, now the 6 Director of Resources for the Future. 7 We met in July and had a very good 8 discussion about their views and where we 9 think that they could be useful in mapping 10 out where EIA might be going over the next 11 five years, what our mandate to them was, and 12 Denny, as he always does, has been very 13 thorough and even while he's spending a year 14 abroad on sabbatical getting lots of e-mails 15 and requests and they're going to meet in 16 about a month and that will be the critical 17 bringing together of the team and then after 18 that will prepare a report for us and it will 19 be very useful as we look out in our 20 strategic plan and particularly as we respond 21 to OMB's request that we look at the whole 22 program. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 27 1 They criticized our part submission 2 because we had submitted individual reviews 3 of individual components of what EIA does but 4 we had never done a complete review of what 5 EIA's mission is, what it should be, are we 6 doing the right things, so that's what this 7 is all about and I think it's going to be 8 extremely useful and with that team that you 9 see there I can't help but think that it will 10 be very important to EIA's future planning. 11 As I mentioned, we're going to have 12 a new home page in our website and we're 13 hoping it will be available by the end of 14 November. We had invaluable comments from 15 this group at the last meeting. It's now 16 gone through internal testing and review and 17 I've been looking at it myself over the last 18 few days and I think it's going to be 19 extremely popular, even make EIA's website 20 even more popular. 21 With that, Nick and members of the 22 committee, I once again want to thank you for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 28 1 everything you've done for EIA and I know 2 you'll see from the further sessions how 3 important your advice has been and how we put 4 it to use and why we know that we've still 5 got plenty of room to improve and we look 6 forward to working with you, Nick, and the 7 rest of the committee. Thanks. 8 DR. HENGARTNER: Thank you very 9 much. 10 MR. CARUSO: I'll be happy to 11 answer any questions. 12 DR. HENGARTNER: The strategy is if 13 someone in the committee has a question we 14 have a tent and we lift it. Questions, 15 anyone? I had actually a few questions 16 concerned with the response to natural 17 disasters and, more generally, disruption to 18 the energy infrastructure. And the first 19 question is are we actually well equipped to 20 report in a timely manner such disruptions? 21 Secondly, are we gathering the right data to 22 be able to do that? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 29 1 MR. CARUSO: I think EIA itself is 2 pretty well equipped because we do our weekly 3 surveys particularly for petroleum and 4 natural gas. We have a ready-made network 5 that we can reach out to in an emergency. As 6 you could imagine, in an emergency like this 7 a number of our respondents were either 8 evacuated or offline so it was more difficult 9 but I think we've played a pretty important 10 role. The actual emergency response and 11 operational side of that are the 12 responsibility of another office within DoE 13 called Office of Energy Reliability but we've 14 been feeding our information through them in 15 some instances and then we have our own 16 product in the case of the daily situation 17 report on our website. 18 But clearly you find out in 19 situations like this there are gaps and 20 particularly on the natural gas side much 21 more difficult because there are 30,000 22 operators as opposed to the oil side where BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 30 1 you have a more limited universe. 2 DR. HENGARTNER: Yes, Jae. 3 DR. EDMONDS: As we go through 4 further sessions will we be hearing more 5 about plans for NEMS and the approach that 6 you're going to be taking as you rethink that 7 architecture? 8 MR. CARUSO: Yes, I would hope, 9 Jae, by April that the next meeting of this 10 group that that would be a fairly significant 11 part of the agenda although you may recall 12 the last time we did it it took three years. 13 And that's what we've requested, a three-year 14 budgetary plan. The first year we would 15 actually start using some of our own money in 16 '06 and try to develop the architecture and 17 the last time we did it by taking I think it 18 was between a handful and 10 people out of 19 their day to day jobs, creating a task force 20 and designing the NEMS with the assistance of 21 contractors as you remember. 22 And we even had a national academy BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 31 1 review of that and I think it was '91 or '92 2 that Jim Sweeney, I think, chaired. So the 3 answer is yes, but not this meeting, partly 4 because other than the in-house agreement 5 that this needs to be done we now need to get 6 OMB on board. I'm hopeful. 7 DR. EDMONDS: Right, well, I think 8 it's a very exciting prospect. 9 DR. HENGARTNER: Thank you very 10 much. 11 MR. CARUSO: Thank you, Nick. 12 DR. HENGARTNER: Next we will hear 13 from Nancy Kirkendall, who is going to talk 14 to us about what's going on in the Statistics 15 and Methods Group at EIA. 16 MS. KIRKENDALL: Well, actually 17 what I'll talk about is some of the 18 suggestions that you gave us before and what 19 we've done as a result of them. A lot of 20 that is what we're doing in SMG2, though, 21 since that seems to be how I entertain my 22 troops. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 32 1 We've talked to you a number of 2 times about our frames. We had an overall 3 team evaluation of all of our frames last 4 year and we talked to you about it in the 5 spring. As you know we've also had the 6 Census Bureau that's been evaluating five of 7 our frames and on Friday you'll be hearing 8 the last of their reports. They've completed 9 their study or it's almost complete at least. 10 So one of the recommendations you 11 made after the last time was that we might 12 not want to evaluate every frame on a 13 periodic basis every three to five years. 14 Even you thought that perhaps we should use a 15 more targeted approach, pick the frames that 16 are in most need of help and look into them 17 in more detail and come up with actual 18 recommendations for improvement. 19 So this is an ongoing effort. We 20 are going to think about your recommendations 21 and I've asked the frames team, which Howard 22 Bradsher-Fredrick is chairing, a group of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 33 1 people from throughout EIA, to think about 2 what we should be doing in the long term in 3 terms of frames evaluation. So we will 4 contemplate that and come back and talk to 5 you about it in the spring. 6 For one thing the Census Bureau's 7 report is new and we haven't had a chance to 8 look at it and try to figure out what we 9 should do about that too so you will have a 10 chance to give us some input on that 11 evaluation and what we should do going 12 forward. 13 As Guy mentioned, the web usability 14 testing was very useful. We've taken most of 15 your suggestions into account. These are 16 some of the specifics. They've done the 17 prototype and, as Guy mentioned, we are 18 actually testing it within EIA. This is the 19 redesigned system and what we're supposed to 20 be doing, we being EIA staff, is finding bugs 21 in it. So we're trying to debug it before it 22 goes live and hopefully the new system will BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 34 1 go live in November or December so we're 2 aiming to have something soon. 3 The external review team, in fact 4 we came to you I guess was it in the fall 5 last year when we first asked you what we 6 should do in terms of getting an external 7 review? So this whole effort came from some 8 of your recommendations. In the spring you 9 particularly suggested that we should do a 10 self-study before we have the external review 11 team come in. 12 And we did think about it, we came 13 up with a plan, we thought perhaps we'd have 14 senior managers in the EIA answer a subset of 15 the questions we were asking the committee to 16 provide that as answers to begin with. We 17 debated about how we could possibly do it in 18 the time frame we had because they were 19 having their first meeting in July. And then 20 when we asked Denny Ellerman if he thought we 21 should go ahead with this, he's the chair of 22 the external review team, he said he didn't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 35 1 think that we needed to do it. So we 2 breathed a great sigh of relief and didn't do 3 it so we didn't take your suggestion but we 4 did think about it a lot. We can do 5 something more like the preliminary review in 6 future iterations of the external review 7 team. 8 Some of your suggestions, however, 9 did get rolled into a new document we 10 prepared particularly for the external review 11 team. That's the overview document that 12 describes EIA and its products and we will 13 send it to you for a review. I think it's a 14 very nice document. We haven't really had 15 something that looked at EIA as a whole to 16 describe what we do and why we do things. 17 Future thoughts are that we should take that 18 document and edit it and make it available on 19 our website so any comments you have on the 20 document would be most welcome. 21 This is just, again, the list of 22 people from the external review team and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 36 1 their affiliations. It's really a great 2 group. Denny is fantastic. I'm chairing the 3 support team and just this week he sent out 4 assignments to all of his teammates about 5 what they're supposed to talk about at their 6 meeting in November so I think we're going to 7 have some really interesting results and 8 we'll talk to you about what their 9 recommendations are and what we plan to do 10 about it in the spring. 11 This is everything else. You've 12 given us some recommendations on the short- 13 term energy outlook and we've accepted and 14 implemented some of your suggestions and 15 you're going to hearing a lot more about STEO 16 from Margot. We asked you about incentives 17 in the residential energy consumption survey. 18 I believe that the incentive experiment has 19 been approved by OMB and we're going forward 20 with that. 21 The way these things go it will 22 probably be a long time before you hear about BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 37 1 the outcome of that experiment. This is a 2 very large survey, it takes a long time to 3 run, but we'll tell you the outcome of the 4 experiment once we get there. 5 On survey evaluations this was 6 Brenda Cox talking about coming up with a 7 template for evaluation of survey programs. 8 We're going to have her continue to finish 9 off the petroleum marketing ones and then 10 we've to digest what we can do with the 11 ideas. One of the things that we've already 12 done with template is to use it as an outline 13 for documentation for new surveys. So even 14 if we could have our documentation in a more 15 consistent form that's not a bad outcome of 16 that exercise. 17 Last time on the EIA 826, the 18 electric power sales and revenue report, we 19 talked about an experiment predicting annual 20 data from annual data so we could actually 21 find out how various estimation methods 22 worked. And your recommendation was that we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 38 1 should do a similar experiment to the best we 2 could with monthly data, which is a grand 3 idea, we do plan to do that, we haven't done 4 it yet. 5 You're going to be hearing a talk 6 tomorrow morning that is looking at the 826 7 and the 861 but they're trying to look at how 8 to tie together time series methods with more 9 stationary regression methods in that survey. 10 And that work resulted from an analyst who 11 looked at our historical data and noticed 12 some funny bumps in it. 13 The EIA 914, Guy has talked about 14 that. One of the interesting things, I think 15 it was on a slide but you probably didn't 16 catch it is in the State of Texas, as you 17 know, Randy's student did a lot of work. 18 That state had wonderful data with patterns 19 that could be used in predicting for the 20 future and she helped us come up with a nice 21 way of doing that. 22 Well, in January 2005 Texas changed BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 39 1 the way they're reporting data so there is 2 now a huge change in Texas data and it makes 3 us even more thankful that we now have the 4 914 to help us come up with our new 5 estimation methods. And with that I'll 6 introduce Preston, who's going to give you an 7 update on the 914. As you recall, he gave 8 you some simulation results a couple of 9 meetings ago and you made some 10 recommendations so this is really just 11 finishing up the report on what worked and 12 what didn't work in trying to estimate 13 natural gas production from a sample survey. 14 So Preston. 15 MR. McDOWNEY: Good morning. I'm 16 going to update for the simulation results 17 for the EIA 914. In the spring 2004 meeting 18 Inderjit Kundra introduced the development of 19 the 914 survey to collect monthly data 20 considering a probability proportion to size 21 sample at the time and it goes with the 22 estimated US production with a coefficient of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 40 1 variation of 1 percent at the US level and 5 2 percent at the regional level. 3 We have seven regions, Texas to 4 Federal Gulf, Louisiana, New Mexico, 5 Oklahoma, Wyoming, and the rest of United 6 States excluding Alaska. We're going to use 7 the presumed optimal allocation formula to 8 determine the sample size based on the 9 desired CV. We calculated that the formula 10 of a sample of 176 companies will give us the 11 1 percent CV at the US level but we needed 12 358 companies to get the desired CVs in each 13 region. 14 With that sample in 2004 at the 15 fall meeting I presented the simulation 16 studies. We used simulated four estimation 17 procedures. The first was the probability 18 proportion to size estimator, the weighted 19 least squares estimator, which is the ratio 20 estimator, ordinary least squares estimator, 21 and a difference estimator, which uses the 22 prior, in this case years, response as a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 41 1 predictor for the future. 2 For the weighted least squares and 3 the ordinary least squares procedures we had 4 two methods, the first using the noncertainty 5 group only in estimation procedure and the 6 second using both the certainty and the 7 noncertainty groups in estimation procedure. 8 For both methods we had three variations, one 9 using all the operators in the method, the 10 second removing the outliers, and the third 11 removing outliers and influential variables 12 observations. 13 The results of that simulation, 14 these are the percent errors for the total 15 natural gas production at the US level. All 16 of the procedures, methods, and variations 17 had a negative bias. The probability 18 proportion to size estimator had the smallest 19 bias but a very large variation and range of 20 errors. The weighted least squares, ordinary 21 least squares and the difference estimator 22 all had smaller ranges but a higher negative BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 42 1 bias. So we concluded in the fall that the 2 PPS sampler wouldn't give us the desired 3 accuracy. The ASA committee recommended 4 using the Hajek estimator as a nonbiased 5 estimator. We decided to use a 90 percent 6 cut-off sample at the national level and John 7 Wood made a presentation on the need for a 8 bias adjustment. 9 In the spring 2005 meeting John 10 Wood presented the preliminary January and 11 February 2005 data. He demonstrated the 12 estimation procedure, which is a variation of 13 the ratio estimator, and he presented the 14 adjustment for the negative bias. He 15 reported a 99.4 percent response rate that's 16 volume rated. 17 So after the fall meeting we 18 started Stage 2. We had a new frame and we 19 wanted to see if the frame made a difference 20 in our simulation procedures. We also 21 implemented the Hajek estimator and we 22 implemented a stratified sample. And this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 43 1 time we only did two regions, Texas and the 2 Federal Gulf, and they're the two largest 3 regions. 4 In the first stage we had annual 5 data in the frame and the sources were the 6 EIA-23 survey data. We had state data and 7 multiple third party data. And Stage 2 was 8 monthly data and this time we had a single 9 third party as the data source. The first 10 time we had the seven regions in a way 11 integrated so if we picked a company in Texas 12 if they were put in any other region we took 13 their data also. In the second stage we 14 simulated each region individually so there 15 was no integration here. 16 With the new frame we used the same 17 formula as in the first one but we got 18 different sample sizes. In Texas we had 96 19 companies and 20 in the Federal Gulf. In 20 Stage 2 there were 61 companies and 19 in the 21 Federal Gulf, and that's due just to the 22 frame. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 44 1 We had the same four estimation 2 procedures and we added the Hajek and the 3 stratified. This is the Hajek estimator, the 4 stratified sample versus random stratified 5 sample with replacement. We had six non- 6 certainty strata in Texas, two non-certainty 7 strata in the Federal Gulf. 8 We're going to summarize the whole 9 project by showing the percent errors in the 10 total estimated regional production and I did 11 that by just summing all the monthly 12 estimates within a region. The stratified 13 sample had the strongest negative bias and a 14 fairly large range of errors. The PPS sample 15 had a much reduced negative bias but a very 16 large range of errors. 17 The Hajek was a little less biased 18 and smaller range of errors. The difference 19 in the model-based methods, the difference 20 estimator had a very small bias and very 21 small range of errors. 22 I'm going to skip to the last two. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 45 1 These are the model-based methods. When you 2 add the certainty and the non-certainty 3 together you really reduce your range of 4 errors and moderately improve on the bias as 5 opposed to when you don't add the certainty 6 companies. 7 We get a similar story in the 8 Federal Gulf. The stratified sample, that's 9 the best prospect but there are only two 10 strata and seven companies in the non- 11 certainty category. The Hajek does give you 12 a much smaller ———————— than the PPS sample 13 in the Federal Gulf. The difference 14 estimator performance is pretty much the same 15 and the model-based does reduce the range of 16 errors but, unlike in Texas, actually 17 increases your bias. 18 So in general the probability 19 methods had the largest variation in many 20 cases especially you need to go back and look 21 at the months themselves with the most 22 negative bias. The difference estimator and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 46 1 the model-based estimators with the certainty 2 and the non-certainty companies had the 3 smallest variation and least bias. And we 4 didn't show them but outlier and influential 5 variables made little difference in the 6 results. So that's my update. 7 DR. HENGARTNER: Thank you very 8 much. We will now also hear the EIA regional 9 short-term energy outlook by Margot Anderson. 10 MS. ANDERSON: Good morning, 11 everyone. I want to thank Nancy and Bill for 12 inviting me to come and speak with you today 13 updating you on what we've done to the 14 regional Short-Term Energy Model. I was here 15 in April to talk to you about work that was 16 currently underway. As Guy mentioned in his 17 presentation, it's become a workhorse for us 18 over the last couple of months. 19 Back in the halcyon days of August 20 before the hurricanes when we were just 21 starting to launch the regional model little 22 did we know how useful it might be during the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 47 1 hurricanes and after to look at the aftermath 2 and to be able to disaggregate some of the 3 results at a finer level of detail than we 4 were able to before. 5 So I want to give you an update. I 6 want to recap from 2005. I want to look at 7 the schedule because there is still work for 8 us to do. I want to give you an overview of 9 the key components and some of the challenges 10 that we're facing as we go forward. I remind 11 you that the purpose was to expand the 12 national model into more regional detail than 13 we had had before. Clearly we are 14 constrained by the amount of subregional data 15 that we have available to put in to the 16 model. 17 In many cases we can go to the nine 18 census regions, as Guy mentioned. In some 19 cases we can't break it down into that much 20 detail but we do try to regionalize to the 21 extent that we can regionalize for most 22 energy sectors. And we did this in the hopes BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 48 1 of providing additional relevance to the 2 model, to enhancing our ability to look at 3 regional supply, demand, and prices, clearly 4 to answer the kinds of request we get at the 5 regional level. 6 Guy is absolutely right. I think 7 people would like a household projection. 8 We're unable to do that. We get a lot of 9 requests still, of course, at the state level 10 and the regions are not fine enough for a lot 11 of our customers who would like us to go into 12 more detail. We just don't have the state 13 level data to do that in most cases. 14 We want to add depth, greater 15 understanding of what's going on. It's not 16 just to feed the customers but it's for us to 17 understand how these markets are interacting 18 particularly at a regional level. And there 19 are lots of disparities in supply, demand, 20 and price across the regions and we thought 21 it would be helpful and enrich our analysis 22 of the short-term energy situation and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 49 1 outlook to have a more regionalized model. 2 So that was our motivation. We 3 talked about wanting to launch in the summer 4 of '05. I'm happy to report we were able to 5 launch in August, as I said, the halcyon 6 days. Before the hurricanes we thought it 7 might be smooth sailing, we would launch 70 8 percent of it in August, we would continue to 9 do the updating in September and October, in 10 November we'd have the whole thing ready. 11 Well, we got hit by Katrina and by 12 Rita and it meant that we weren't able to do 13 some of the development that we wanted to and 14 continue working on the electricity dispatch 15 model, which is still not operational but 16 will be shortly, and we wanted to focus more 17 of our efforts on using the regional model to 18 put out the September and the October STEO. 19 In September we did something 20 unusual because it had been just a few days 21 since the hurricane. For one of the first 22 times ever we did a non-single estimate. We BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 50 1 did a forecast that looked at a slow, medium, 2 and fast recovery, which isn't something we 3 normally do in the short-term energy outlook. 4 We tend to have a baseline outlook. But we 5 were able to look at three different recovery 6 patterns because in those first days we 7 didn't really know where it was going, we 8 didn't have a good assessment of the damages, 9 and we didn't want to commit ourselves to a 10 single baseline forecast so we had three 11 forecasts. 12 By the time Rita hit we had about a 13 good three weeks to figure out what was going 14 on in the Gulf and the October forecast went 15 back to a single baseline, a medium recovery 16 forecast, but again was also coupled with the 17 winter fuels outlook. You saw many of the 18 slides that were used in the winter fuels 19 outlook in Guy's presentation and we were 20 able to look at heating expenditures for the 21 winter in most cases for the nine census 22 regions, something we had not been able to do BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 51 1 before. 2 So, as I said, we're going to get 3 the electricity component working in the next 4 couple of months. The delay was really due 5 to the fact that the hurricanes just 6 sidestepped us from the development angle to 7 the use angle. We are ongoing in our expert 8 review. We have all of that documentation 9 ready to be reviewed and we are working with 10 SMG to put together a panel to review the 11 documentation. We made the decision to go 12 live without having the documentation 13 reviewed. 14 We just didn't feel that we could 15 do them in the sequence that we would 16 normally do, review and then go live. We 17 wanted to go live and we will look at this as 18 a living model. If we need to make the 19 adjustments based on comments we get back 20 from the expert reviewers we're more than 21 happy to do so but we really wanted to 22 implement the regional model so the expert BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 52 1 review will occur in '06, Bill, '06. 2 The recap, as I said earlier, we've 3 broken it into these sectors and we have the 4 regional components associated with each 5 sector, far more information than we had 6 before. We also have an expanded query 7 system associated with the STEO models so it 8 allows you to go in and get the data that we 9 use for the subregions that we're looking at. 10 We did get rid of the interactive 11 model that people could use to generate their 12 own estimates in STEO. It's too unwieldy to 13 make the regional model available in that 14 kind of a PC format. If we ever get there we 15 might be able to make it available sometime 16 in the future but for now we have not made 17 that feature available so that is one service 18 we're not providing to customers but we're 19 giving you all the detail in terms of the 20 data that we can and will be as transparent 21 as possible in the website with the 22 documentation. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 53 1 Regional detail varies. I 2 mentioned this earlier. It just depends on 3 what region and what field you're looking at 4 to determine how much data we can get to a 5 more disaggregated level. We talked last 6 time about there were strong regional 7 differences, availability of data, more 8 limited data for propane and for heating 9 fuels. 10 For heating oil it doesn't matter 11 as much because a lot of those fuels are used 12 in a very concentrated area. So sometimes it 13 doesn't matter as much that you don't have 14 the regional detail that you'd like to have. 15 These are the regions. I'm sorry it doesn't 16 show up better. It didn't color on the STEO 17 page. But this gives you an idea. We have 18 the census regions and census divisions and 19 this is how we've broken out in the winter 20 fuels outlook in the October STEO so you can 21 look at prices and expenditures for those 22 regions which has been very helpful this last BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 54 1 time out because we were able, as I said, to 2 go into more detail which matters more when 3 prices were higher and supplies were tighter. 4 In other years where you might have had 5 expected a more normal year there probably 6 would have been less emphasis on the regions 7 because there wasn't as dramatic a change 8 between this year and last year in terms of 9 expenditures. 10 And I think that showed up nicely 11 on Guy's presentation where he showed the 12 damage associated with the hurricanes and the 13 expectations on prices that are attributed in 14 part to the hurricanes, in part to the 15 general situation we find ourselves in tight 16 global markets and tight domestic markets. 17 But the winter fuels outlook 18 mattered more this year because of these 19 dramatic changes in the energy sector so it 20 was nice to be able to have additional 21 information for people to make decisions on 22 and for policy makers to consider where there BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 55 1 might be regional disparities associated with 2 heating expenditures. 3 Observations, I've been at EIA a 4 year, about 14 months, and so I'm looking at 5 it as somebody that jumped into the 6 development of the regional model while it 7 was ongoing. I was not there at the 8 beginning when a lot of the work was really 9 developed so I certainly can't take any 10 credit for that nor could I take credit for a 11 lot of the day to day work that goes on in 12 it. That is attributable to Mark Rodekohr 13 and Dave Costello, who are part of the STEO 14 team and lead the STEO team. 15 It is resource-intensive to do this 16 kind of work. That sounds like an obvious 17 observation, but I think probably more 18 resource-intensive than we might have 19 anticipated when we walked down this path; 20 however, the value that you get out of a 21 regional model is quite large. 22 You asked earlier about are we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 56 1 capable of looking at some of these energy 2 emergencies. The regional models can allow 3 us to do that better. As we develop it even 4 more and more it's even going to add to that 5 capability. 6 So we use something called the 7 rules of thumb when we look at international 8 supply disruptions and it's a fairly simple 9 way to look at supply shortages and how they 10 might affect GDP in the United States and how 11 they might affect oil prices. This model 12 would be able to help us look at disruptions 13 whether they are happening in the United 14 States due to a weather event or somewhere 15 else; it's almost irrelevant where it 16 happens. 17 The regional model will allow us to 18 look at some of these contingencies. So I 19 think this is a powerful tool particularly, 20 because we didn't have this capability on the 21 natural gas side or on the electricity side 22 which we will have in the not too distant BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 57 1 future. So, again, I said earlier, looking at 2 regional disparities and conducting 3 contingency analysis, it's going to be a very 4 powerful tool. 5 And it can also be a fulcrum for 6 data collection and data sharing within EIA. 7 My office is very dependent upon all the 8 other offices in EIA to get the data to put 9 into the model and so it means that there 10 often has to be more cooperation and 11 coordination across the offices to make sure 12 that we're getting the data that we need in a 13 timely fashion to put into the STEO. 14 So I think it enhances the 15 collectiveness of the data and how the data 16 are used in a forecasting and modeling 17 framework within EIA and has raised a lot of 18 questions about data that are collected and 19 the turn-around of data to get them prepared 20 and model ready for the STEO model. So it's 21 raised some issues and some challenges that 22 we have here in EIA on dealing with some of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 58 1 the data issues of making sure that STEO is 2 ready to go every month without fail. We 3 can't delay STEO. It's on the same kind of 4 schedule that our weekly data series are on. 5 It's not optional. It has to go out every 6 month. 7 I urge you to look at the regional 8 STEO. You can look at it in the new web page 9 in the not too distant feature. The STEO is 10 now completely redesigned and we think it's 11 much easier to navigate through the tables, 12 through the text, through the links. The 13 query system is there. We've added some 14 interesting little features that I think make 15 it a much more user-friendly product and a 16 much more attractive product. It's always 17 been a high quality product and remains so 18 today, I think, and the regional has only 19 enhanced it. 20 So I can answer any questions. Do 21 you want me to answer questions on the 22 regional model at this point before I go into BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 59 1 forecasting? 2 DR. HENGARTNER: Yes, Moshe. 3 DR. FEDER: Margot, do the regional 4 STEOs add up to the national STEO and how is 5 it accomplished if so? 6 MS. ANDERSON: Yes and no. We 7 still have a national forecast and sometimes 8 it's an adding up and sometimes it's a 9 national forecast that is decomposed down. 10 So it isn't such a seamless system that every 11 little regional area adds up to a national 12 estimate. Mark, stop me when I go into an 13 area I don't need to go into. 14 DR. FEDER: Whether you aggregate 15 up or disaggregate down it always adds up, 16 right? There's no -- 17 MS. ANDERSON: Well, it has to. I 18 mean, there is no separate national forecast 19 and then we match it to a separate system 20 that's a regional forecast. They're an 21 integrated system and so the real question is 22 are we getting a different national forecast BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 60 1 than we would've gotten without the regional 2 model. 3 DR. FEDER: That's a good question. 4 MS. ANDERSON: That's an 5 interesting question. We haven't answered it 6 yet. Mark -- 7 MR. RODEKOHR: Yes, you are. 8 MS. ANDERSON: Thank you, well, we 9 are. 10 MR. RODEKOHR: Yes, we do add 11 everything up but the regional area is 12 differed by the fuel. For example, petroleum 13 is based on a pad structure and natural gas 14 is based upon a census division structure. 15 But we do add it up and it's not a top-down 16 notion where you get a national forecast and 17 you share it out. It's the other way around. 18 We get the regional forecast and we build 19 them up. 20 MS. ANDERSON: But the question 21 about whether it's the same as we got with 22 just the national model. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 61 1 MR. RODEKOHR: When you add up 2 things over certain time spectrum they're not 3 a lot different in many respects. A good 4 example is what Guy did here with the fuel 5 outlook. When you look at the regional 6 differences on weather patterns and you add 7 them all up it really doesn't make a lot of 8 difference. But it's important to the 9 regions themselves as to those difference. I 10 don't know if that's helpful. 11 MS. ANDERSON: If we could really 12 disaggregate, looking at forecast errors 13 before the regional model and forecast errors 14 after the regional model, that might help you 15 determine levels of accuracy when we were 16 using just the national model or whether we 17 were using just the regional model. What 18 matters now is the model that we're using is 19 as accurate as we can possibly get it. 20 MR. RODEKOHR: Can I say one more 21 statement? It may be easier to get a much 22 better accurate forecast at a national level BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 62 1 because all sorts of things are balancing out 2 but from a regional perspective it really is 3 important to get those things better and 4 that's what we could not do before. 5 DR. HENGARTNER: Mark? 6 MR. BERNSTEIN: I might say I 7 really liked when you did the different 8 scenarios after Katrina stuff and I would 9 encourage you to think about continuing 10 something like that rather than giving a 11 single estimate. Even in the short term as 12 you go along there is always something that 13 you look out the next couple of months and 14 say well, there's something we're not 15 particularly sure of but there is some 16 uncertainty critical out there in the next 17 few months and it would be nice to recognize 18 that and perhaps let that one float. And it 19 may be different ones in different months 20 where is this winter going to be particularly 21 cold or something else. 22 And so I would encourage you to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 63 1 think about having a range even in the short- 2 term outlook. So that's just one comment. 3 The question is in the short- term 4 energy outlook are the regions the same for 5 fuels and electricity or are they like NEMS, 6 the regions are different? 7 MS. ANDERSON: No, the regions are 8 different but, getting back to the first 9 point, it's a source of some debate within 10 EIA or at least friendly conversation about 11 how much our forecasting models ought to try 12 and hit a bull's-eye. Now, the parenthetical 13 here is that, of course, in the winter fuels 14 outlook we do a colder and a warmer scenario 15 so that we give estimates depending on 16 whether we expect it to be 10 percent warmer 17 or colder from the NOAA forecast. 18 But many other forecasters will put 19 out reports that might look at cleverly 20 titled worst case scenario or intensive 21 hurricane season forecast and so they will 22 give this range of estimates depending on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 64 1 what they think might be something that would 2 disrupt the market over the coming months. 3 Certainly we know that these markets are not 4 smooth sailing and if you look back one year, 5 you look back five years, you look back 10 6 years there is always some unanticipated 7 event. And to be able, maybe, to anticipate 8 those better or to give a range on estimates 9 is certainly something that the NEMS model 10 does on a regular basis. 11 They run 30 cases or so every year. 12 They may highlight the reference case and 13 talk mostly about the reference case but 14 they're always doing these other scenarios. 15 We have tended not to do that in the STEO 16 framework mostly because of time constraints 17 because it has to get out every month and 18 then the text constraint of being able to 19 talk about those other scenarios but I don't 20 think it's an issue that we've put to rest 21 yet. 22 I'm always concerned about making BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 65 1 it appear as if we're trying to hit a 2 bull's-eye every month when in fact what 3 we're really trying to do is generate a 4 reasonable story and a discussion about what 5 we think is impacting the markets and to give 6 a fairly accurate assessment of what we think 7 the interactions are and what the prices are 8 going to be and demands, et cetera. 9 So I think there might be some 10 value in that because often people will take 11 the estimate and really misuse the point 12 estimate and they won't be nuanced by what we 13 all know as forecasters and researchers that 14 there are some wide bands here. And it's our 15 obligation to educate people about the 16 uncertainty that might be associated with 17 these estimates. So I think it's a point 18 well taken. 19 DR. HENGARTNER: Yes. 20 MR. RODEKOHR: One last point, if 21 you do look in the last 100 STEOs you will 22 notice on world oil prices there is a red BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 66 1 band and it has always been there. And so 2 you can't say we've ignored that uncertainty. 3 People just don't focus on it. 4 MS. ANDERSON: The confidence 5 interval that we put on some of the prices. 6 MR. RODEKOHR: Yes, on some of the 7 prices, right. 8 MS. ANDERSON: But we don't focus 9 on it either. I mean, we have the red band 10 but we don't talk about what that really 11 means and what's motivating us to do that and 12 how people should interpret that information 13 so point well taken. 14 DR. SITTER: How are you generating 15 that red band? Is that scenario variability? 16 MS. ANDERSON: Mark. 17 MR. RODEKOHR: The red band is 18 generated upon the standard deviations from 19 the model we use to do prices. Now, again, 20 that's the model we use but the real point 21 estimate on the prices is done by more of a 22 Delphi approach where we sit around and we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 67 1 talk to each other and we get that but we do 2 have a model and it does give you two 3 standard deviations away from it and that's 4 where the red band is generated from. 5 And I would like to say if you went 6 back historically and looked at how well we 7 did versus that red band the model's probably 8 pretty close. In other words 95 percent of 9 the time you're going to be within that band 10 understanding the limitations, there are 11 monthly prices, there are averages, et 12 cetera. So that's how we do it. 13 MS. ANDERSON: But we could test 14 that. 15 MR. RODEKOHR: Well, it's very 16 time-consuming, yes. 17 DR. HENGARTNER: What I liked about 18 Mark's initial comment that started this 19 discussion is that in fact the model is 20 changing. You're changing the assumption in 21 the model and so this red band although tied 22 to standard deviation is useful. The BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 68 1 question is how sensitive is it to model 2 departures that can come from external shocks 3 like the hurricanes. 4 MS. ANDERSON: Well, the red band 5 is only on a couple of price variables that 6 are exogenous to the model. 7 MR. RODEKOHR: That's right, 8 exactly. It's not all-encompassing. That's 9 correct. It's on the most price variable 10 that a lot of people come to the STEO for a 11 WTI. 12 DR. SITTER: Right, that was the 13 point I was trying to make is that's not 14 really addressing this issue. This is a 15 completely different measure of variability 16 than the issue of different scenarios. 17 Different scenarios could actually be handled 18 in a more Bayesian framework if you wanted 19 to. 20 MS. ANDERSON: If we use the STEO 21 mostly to generate the short-term energy 22 forecast I'm hopeful and my vision is that we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 69 1 would be able to use the STEO to do analysis 2 outside of just generating the forecast so 3 you would see that you might put in a couple 4 of different scenarios of what-ifs associated 5 with either some kind of an event or a change 6 in policy and be able to look at a short-term 7 impact by using the short-term model. 8 We've tended to use it for the STEO 9 product itself as opposed to a workhorse of 10 energy analysis and then in that case you 11 would certainly want to explore alternatives 12 but it's a point well taken. 13 DR. HENGARTNER: Well, thank you 14 very much. 15 MS. ANDERSON: You're welcome. 16 DR. HENGARTNER: Before continuing 17 I'd like to introduce to you Tom Rutherford, 18 who just joined us. He is a new committee 19 member from Colorado, correct? 20 MR. RUTHERFORD: Michigan. 21 DR. HENGARTNER: So welcome very 22 much to our committee. Those members of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 70 1 audience who have joined us since we started, 2 if you could please introduce you into the 3 microphone for the records, please. 4 MR. SINGPURWALLA: Darius 5 Singpurwalla, LECG. 6 MR. BLAIR: Johnny Blair, Abt 7 Associates. 8 MR. SCHIPPER: Mark Schipper, EIA. 9 MS. McGUCKIN: Nancy McGuckin, 10 Department of Transportation. 11 MS. WAUGH: Shawna Waugh, 12 Statistics and Methods Group. 13 MS. TAYLOR: Yvonne Taylor, EIA. 14 MR. RODEKOHR: Mark Rodekohr, EIA. 15 DR. HENGARTNER: Thank you very 16 much. Before continuing I again want to ask 17 if any member of the committee has some 18 questions for Nancy's or Preston's 19 presentations. I know it's a little bit back 20 but I'd like to wrap up the update for the 21 committee section. 22 DR. SITTER: I have one question BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 71 1 for Preston. Is it possible to get a 2 write-up together with the previous write-up 3 because this was too terse for me to 4 understand what has been done since the 5 previous write-up. 6 MR. McDOWNEY: Sure. 7 DR. SITTER: Thanks. 8 DR. HENGARTNER: Anything else? 9 Okay, thank you very much. So let's 10 continue. The next presentation will be on 11 short-term forecasting performance measure 12 and accuracy evaluation by Margot, yes. 13 MS. ANDERSON: I'm going to talk 14 about the STEO performance indicators 15 project. I have my partners in crime here as 16 usual, Mark Rodekohr and Mike Morris, who 17 have done all the yeoman's work on this. I'm 18 just here to try and provide context and put 19 into an approach that not only talks about 20 some of the technical matters but why we're 21 doing it and what we hope to do with it in 22 the future. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 72 1 This is a growing program. It came 2 out of a conversation that we had last time 3 we here, in April, about what to do, about 4 really understanding how well we're doing 5 with our forecast. And it isn't so much to 6 be a report card and to say you got an A or a 7 B or you were absolutely right or absolutely 8 wrong. But we really needed a way to do some 9 performance indicators to help us improve 10 what is now a much more complicated model as 11 well as to take a look at how well we're 12 doing. So it's really to serve a couple of 13 purposes to develop a systematic way to look 14 at performance indicators and forecast 15 errors. 16 So when I was here last in April we 17 discussed the necessity for developing some 18 diagnostic tools for gauging forecast 19 accuracy and we've got 1,500 variables in 20 this thing now that we could look at and that 21 we're forecasting and do we look at all of 22 them, do we look at every conceivable way to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 73 1 measure forecast accuracy or do diagnostics, 2 and what might be the minimum set to give us 3 the maximum information? So we had some 4 conversations about that. 5 We discussed how to use the 6 information not only to troubleshoot data 7 problems but to improve the model, maintain 8 confidence in EIA's forecasting capability. 9 We tend not to report forecast errors on a 10 regular basis for STEO to the general public. 11 We do keep some track internally to meet some 12 OMB requirements but we have not tended of 13 late to have a program that is transparent to 14 the users of STEO about how are you doing. 15 We also discussed how we might 16 share these results. It can be burdensome to 17 do this every month and to have a long report 18 about what we've got and what it means but 19 there may be a low cost way or a relatively 20 straightforward way to tell our readership 21 what we are showing in terms of forecast 22 errors and diagnostics. We think that's a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 74 1 good goal. 2 So what did we do? Well, since 3 April while we were working on the regional 4 model we also talked about doing some 5 preliminary work on looking at some forecast 6 errors and choosing some metrics and taking a 7 look at what they were telling us. These are 8 not final results. These are not looking at 9 every variable. I think we looked at about 10 40 variables. I'm going to show you about 15 11 to show you the kinds of metrics we're 12 looking at and what some initial results are. 13 So it's an ex-post analysis. We 14 examine how are the projections compare to 15 the data but we don't go back and revise the 16 data in this particular presentation. 17 Whether we need to do that in the future is 18 determined by how much did the data change 19 from revision to revision to revision. 20 That's time-intensive. 21 This analysis starts with monthly 22 forecast in '03. We can go back longer on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 75 1 many series. We just did '03 for this 2 exercise. And what we're going to be looking 3 at is our one-month, our six-month, and our 4 12-month ahead forecast errors so I'm not 5 going to be talking about any diagnostic 6 tools of looking at outliers and 7 distributional concerns about the data. This 8 is primarily a discussion about forecast 9 errors. 10 We looked at a couple of standard 11 measures, we looked at the mean absolute 12 error, we looked at the mean absolute percent 13 error, we looked at the bias to tell us the 14 percent of times that we were under or over- 15 predicting. In some cases we're under- 16 predicting 100 percent of the time but only 17 by a little bit so you want to look at a 18 couple of things to determine how well you're 19 doing. We looked at an F-statistic to look 20 at the statistical significance between the 21 actual data and the predicted data to tell us 22 something about how off we were, given the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 76 1 incredible variation in some of the data 2 series alone. 3 So some initial observations, no 4 surprise to anybody in this room. The price 5 data, particularly the international price 6 data like WTI, are really hard to predict and 7 we're usually wrong. Sometimes we're wrong 8 by a little bit, sometimes we're wrong by a 9 quite a bit more, but we're pretty good and 10 pretty consistent when we're looking at 11 variables that are within our control in the 12 sense that we have modeled the markets 13 carefully. 14 The demand, the consumption 15 variables, for most sectors are relatively 16 stable. The price variables are the ones 17 that give us fits. So those are some initial 18 observations. Obviously those that are in a 19 self-contained US system we tend to be able 20 to model that and where it tends to be less 21 variability there tends to be more knowledge. 22 The data also tend to be better. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 77 1 I hope you can read this. Mark, 2 you're the one that's most far but can you 3 read this at all? 4 MR. RODEKOHR: Barely, yes. 5 MS. ANDERSON: Well, squint. I'll 6 try and walk you through here. What we're 7 looking at, the month ahead is the 1, the 6, 8 the 12. The first three columns are the 9 absolute percent errors that tell you on 10 average how much you're off. Then there's 11 the bias that looks at the percent of time 12 that you're under or over and then the F- 13 statistic, which is testing the difference 14 between the data series and the forecasted 15 series to determine whether you are 16 statistically different. If so you better go 17 look and find out what's going on. 18 These are the prices, the WTI and 19 motor gasoline and heating oil and the 20 natural gas prices, and you can see that the 21 bias in some cases is 100 percent. So for 22 WTI since 2003 100 percent of the time we've BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 78 1 been under-predicting. When we're one month 2 ahead we're under-predicting 23 percent. The 3 error widens the further out we go. So not 4 so bad one month ahead, not so great six 5 months ahead, even worse 12 months ahead. 6 And that's the case on the bias as well. Our 7 predictions are progressively off as we go 8 out for the one or the six or the 12 month 9 ahead. 10 Again, that's not really surprising 11 and I would venture to say that if you were 12 looking at other forecasters that are playing 13 this game they've got a similar track record. 14 That doesn't mean we can't do better and in 15 fact we should be able to use this 16 information to circle back into the modeling 17 framework and say if we're under-predicting 18 all the time maybe we ought to examine what 19 our priors are about why we're under- 20 predicting. 21 If we're under-predicting just a 22 little bit, 1 or percent, I tend to be less BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 79 1 worried about it. If we're under-predicting 2 by these larger percentages that's something 3 that we need to examine in how we're coming 4 up with some of these price variables. 5 Again, the WTI is determined exogenously, 6 outside of the model. The others are related 7 to WTI so they feed off wherever we start 8 with WTI. While natural gas has its own 9 markets and prices are much more determined 10 within US borders it is still often pulled up 11 or triggered by what's going on in WTI. 12 So this is giving us some insight 13 into how we're doing. And one of the 14 questions that I'm hoping you guys can help 15 us with is to say well, why aren't you 16 looking at these four other statistics. Why 17 didn't you do this kind of analysis? We 18 would like some feedback on what might be 19 other approaches that we could take to 20 looking at these variables that would lend 21 themselves to fairly easy interpretation so 22 that if we put them out on a website BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 80 1 customers would get an easy understanding of 2 what this meant. We don't really want to 3 make this purely an academic type of 4 exercise. Jae. 5 DR. EDMONDS: Margot, would you 6 just remind me what's WTI again? 7 MS. ANDERSON: It's the West Texas 8 intermediate crude price. It's a benchmark 9 for crude oil prices. 10 DR. EDMONDS: Thank you. 11 MS. ANDERSON: Motor gasoline is 12 what you get at the pump. The blue line is 13 the one-month, the next dotted line is the 14 six-month, and, of course, the 12-month is 15 the shorter line, less at the series. You all 16 have these in your handouts. I'm giving you 17 some more statistics about what we're looking 18 at. The solid line is zero so it's showing 19 you how much above and beyond we are 20 predicting depending on how far out we go. 21 Again, it's not clear to me whether 22 the best estimate is the one-month ahead or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 81 1 the six-month ahead. Is it the 12 month 2 ahead? What is the best measure of a 3 forecast error if you don't want to flood 4 your audience with every conceivable 5 permutation of every kind of statistic or 6 variable to look at? So we have some 7 deciding to do about that minimal set of 8 variables and statistics to give maximum 9 information. 10 There are gasoline prices again. 11 This is just really a picture of what that 12 summary table showed you. Mike, correct me 13 if I'm wrong. The scales on these are all 14 different so your eyeball can't compare one 15 to another but the price variables, again, 16 have the higher F-statistics than some of the 17 demand variables, which I will get to. 18 Wellhead natural gas prices, this 19 is indicative of what a volatile market this 20 has been over the last couple of years and 21 how difficult it has been to anticipate these 22 changes in the natural gas market and I think BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 82 1 that this picture really demonstrates that 2 story. 3 A couple of demand variables, we 4 looked at natural gas and looked at 5 electricity sales data. Again, it looks a 6 whole lot better in terms of the F-statistic. 7 The bias is telling us that as we move out we 8 were over-predicting on the natural gas 9 demand. I suppose this is intuitive. The 10 further out you go the more difficult it is. 11 I guess that makes some sense. If the one- 12 month aheads are really based on last month 13 you may tend to always do better one month 14 ahead than you would six months ahead or one 15 year ahead. I'm thankful I'm not in John 16 Conti's shoes because he has to predict 20 17 years out and that's a difficult thing to do. 18 We've got pictures of these as well 19 of the petroleum demand. Again, we're much 20 closer to where we need to be, which is zero, 21 which is, as I said earlier, on these demand 22 and consumption variables we do tend to do BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 83 1 better, on natural gas demand a bit more off 2 the mark related to what I said earlier about 3 the vast changes in this market. 4 Total electricity demand, our 5 regional model will help us with this. None 6 of these are regional, by the way. These are 7 all national level estimates. It goes back 8 to the question that we were talking about 9 earlier and what Mark was saying about the 10 ability of the regional models to hit their 11 mark. We haven't yet incorporated into this 12 regional forecast errors because we don't 13 have the time series to do so but eventually 14 we would want to look at how the regional 15 models are doing as well. Right now it's 16 just on the national side. 17 So we are in the early stages of 18 this and we are looking for help and looking 19 for feedback on what we can do. We want to 20 write this up in a paper. As we move on we 21 want to write it up in a more academic paper 22 and justify why we're picking what we're BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 84 1 picking and being able to show a bit more on 2 what these data look like in terms of their 3 distributions, in terms of outliers, and in 4 terms of just some basic diagnostic tools of 5 looking at the data series. 6 I think being able to automate a 7 lot of this will really help us now that we 8 have this much larger model and to pick out 9 some anomalies that are inevitably going to 10 creep in when we have over 12,000 variables 11 that are in the model, maybe predicting 12 1,400, so it's a very complicated model. 13 So what statistic should we look 14 at? What variables should we look at? The 15 obvious ones are on the front page every day 16 but there are a lot of other interesting 17 variables that can provide tremendous insight 18 as to how well the model is doing that ought 19 to be added to our menu of variables that we 20 want to check out. It's, again, not just for 21 public consumption. It's to feed back into 22 making the model a better model. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 85 1 We certainly want to focus on the 2 areas where there are significant error and 3 where we can do something about it. In some 4 cases there is not much we can do about it. 5 There may not be that many variables or 6 structural equations in the model that we can 7 change depending on where the variable comes 8 from if it's determined exogenously. But 9 there are certainly some ways that we can 10 consider options to reduce the uncertainty or 11 the forecast error by adding variables. 12 There is seasonality in the models 13 and we haven't adjusted for that in these 14 forecast errors. And it might be wise if we 15 talked a little bit about the value of doing 16 some turning point analysis. There are quite 17 a few turning points in these kinds of data 18 and some of these metrics are not so great 19 when you get close to a turning point. And 20 so maybe doing turning point analysis might 21 be something that would be useful on some of 22 these series so that's something to consider BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 86 1 as well. 2 And we've got to figure out a way 3 to publish this without creating a huge 4 burden on the staff to do this but creating 5 as much transparency as we can. I think 6 we're obligated to our customer to let them 7 know how well we do and we hope that that 8 would engender others in this game to play 9 along. There are lots of people forecasting 10 and there are lots of people that don't have 11 the credibility that we have and don't have 12 the obligation to be transparent and 13 responsive. 14 And so I think this is a part of 15 EIA's overall agenda, to reveal what we can 16 reveal and to keep the discussion open and to 17 generate feedback, solicit feedback, and make 18 the adjustments where we can. These are 19 official government statistics and we need to 20 feel confident of them. I've said all along 21 when we started this process that it wasn't 22 so we could have an uh-huh and point a finger BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 87 1 at somebody and say see, I knew you were 2 wrong. 3 I have great confidence in this 4 model and I think we're doing probably pretty 5 well compared to what others do and I think 6 we're pretty accurate to the extent that we 7 can be. There is always room for improvement 8 but I would like to have a tool to be able to 9 say not only am I relatively confident but 10 that Guy Caruso gets up somewhere and talks 11 about the results from this model that he has 12 the confidence that he needs because he has 13 the statistics behind him to show that what 14 we're doing is as good as we can do given the 15 volatility of these markets and the sometimes 16 unpredictability that goes on in these 17 markets. So we're looking to you guys to 18 help us out on making this a better program. 19 Mike or Mark, did you want to add 20 anything to this presentation? 21 MR. RODEKOHR: Just one little 22 thing, I think exactly what you said is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 88 1 right. We used to publish the forecast 2 errors and I agree we ought to started it up 3 again. Budget reasons stopped it but 4 nevertheless I do think that that adds 5 something to what other people don't do. You 6 will find that the other private consultants 7 will always tell you when they were right. 8 They will never tell you when they were 9 wrong. And to the extent we can add an 10 historical record and be transparent about it 11 and say here is what we did right and here is 12 what we did wrong I think that's a value 13 added to what we're doing. That's all. 14 Michael. 15 MR. MORRIS: Again, my name is 16 Michael Morris. I work with the STEO team 17 and I look at petroleum demand issues 18 primarily. I don't want to duplicate or 19 replicate what Margot or Mark has said. I 20 just say that I have output for the 37 series 21 that we currently cover which are all 22 national in nature. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 89 1 We'll do the regional, perhaps, 2 starting in a year or two from now when we 3 have more of a track record behind us. And 4 if anybody wants to look at my wonderful F- 5 statistics and biases and all the rest of it 6 I'd be happy to show you so you get a flavor 7 of what this is about and I'd be happy to 8 stick around and show it to any interested 9 parties. Thank you, that's all I have to 10 say. 11 MS. ANDERSON: Thank you, Michael. 12 MR. MORRIS: Any other questions 13 for me or anybody else on this? 14 DR. HENGARTNER: So actually, we 15 have two discussants right now and you're one 16 of them, Tom. 17 MR. RUTHERFORD: Sure. 18 DR. HENGARTNER: So you may as well 19 start with your comments and you can weave in 20 your questions and then Moshe also will have 21 some comments and I probably will open up the 22 discussion to the floor after that. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 90 1 MR. RUTHERFORD: I'm a newcomer 2 here so I wasn't completely certain of what 3 my role was going to be but I'm very happy to 4 be here and I really found the prese