AMERICAN STATISTICAL ASSOCIATION (ASA) MEETING OF THE COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION (EIA) Washington, D.C. Thursday, April 28, 2005 2 1 COMMITTEE MEMBERS: 2 NICOLAS HENGARTNER, Chair Los Alamos National Laboratory 3 MARK BERNSTEIN 4 RAND Corporation 5 CUTLER CLEVELAND Center for Energy and Environmental Studies 6 JAE EDMONDS 7 Pacific Northwest National Laboratory 8 MOSHE FEDER Research Triangle Institute 9 BARBARA FORSYTH 10 Westat 11 NEHA KHANNA Binghamton University 12 NAGARAJ K. NEERCHAL 13 University of Maryland Baltimore County 14 SUSAN M. SEREIKA University of Pittsburgh 15 DARIUS SINGPURWALLA 16 LECG 17 RANDY R. SITTER Simon Fraser University 18 ALSO PRESENT: 19 MARGOT ANDERSON 20 Energy Information Administration 21 LORI ANITI Energy Information Administration 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 3 1 ALSO PRESENT (CONT'D): 2 STEPHANIE BATTLES Energy Information Administration 3 COLLEEN BLESSING 4 Energy Information Administration 5 BILL BOSTIC United States Census Bureau 6 ERIN BROEDECKER 7 Energy Information Administration 8 CHARLES BROENE Energy Information Administration 9 SUSAN BUCCI 10 United States Census Bureau 11 GUY CARUSO Energy Information Administration 12 DAVE COSTELLO 13 Energy Information Administration 14 HOWARD BRADSHER-FREDRICK 15 Energy Information Administration 16 FRED FREME Energy Information Administration 17 CAROL FRENCH 18 Energy Information Administration 19 DWIGHT FRENCH Energy Information Administration 20 KAREN FRIEDMAN 21 Energy Information Administration 22 NICKI HAITOT United States Census Bureau BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 4 1 ALSO PRESENT (CONT'D): 2 SUSAN HOLTE Energy Information Administration 3 CRAWFORD HONEYCUTT 4 Energy Information Administration 5 RICK HOUGH United States Census Bureau 6 PAUL HSEN 7 United States Census Bureau 8 ALETHEA JENNINGS Energy Information Administration 9 JIM JOOSTON 10 Energy Information Administration 11 NANCY KIRKENDALL Energy Information Administration 12 TANCRED LIDDERDALE 13 Energy Information Administration 14 RUEY PYNG LU Energy Information Administration 15 PRESTON McDOWNEY 16 Energy Information Administration 17 RENEE MILLER Energy Information Administration 18 MARK RODEKOHR 19 Energy Information Administration 20 REGINALD SANDERS OnLocation, Inc. 21 LAWRENCE STRAUSS 22 Energy Information Administration BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 5 1 ALSO PRESENT (CONT'D): 2 AMY SWEENEY Energy Information Administration 3 PHILLIP TSENG 4 Energy Information Administration 5 BILL WEINIG Energy Information Administration 6 JOHN WOOD 7 Energy Information Administration 8 9 10 11 12 13 * * * * * 14 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 6 1 C O N T E N T S 2 AGENDA SESSION: PAGE 3 Updates Since Fall 2004 Meeting 14 4 Greetings and Remarks 24 5 Regionalizing the Short-Term 41 Energy Outlook (STEO) Forecast 6 STEO Performance Indicators: 78 7 Diagnostics and Forecast Errors 8 STEO Electricity Modeling 97 9 ASA Committee Discussants 161 10 Regional STEO Propane and Heating Oil Modules 174 11 ASA Committee Discussant on 233 12 Regional STEO Propane and Heating Oil Modules 13 Follow-Up on Frames Team 235 14 Activities 15 EIA's Proposed Strategy for 265 Addressing Declining Response 16 Rates in the Residential Energy Consumption Survey (RECS) 17 ASA Discussant on EIA's Proposed 325 18 Strategy for Declining RECS Response Rates 19 Hands-On Usability Testing of 341 20 EIA's New Website Design 21 22 * * * * * BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 7 1 P R O C E E D I N G S 2 DR. HENGARTNER: Ladies and 3 gentlemen, members of the committee, members 4 of the audience, this is the first meeting of 5 our committee since it was determined that we 6 are not covered by the provisions of the 7 Federal Advisory Committee Act. The DOE's 8 Office of General Counsel has reexamined the 9 statute of the Committee and advised that 10 only committee subjected to the department's 11 actual management or control are covered by 12 the act. And in fact it does absolutely 13 change nothing for us in the way our 14 committee is working. So this is simply how 15 we're perceived by the DOE. It's not 16 internal on how we're working and how we're 17 appointed and so on. So this should be 18 transparent to all of us except for 19 management. 20 Moving to the real agenda, let me 21 remind you that this is an ASA Committee, as 22 is obvious and not an EIA committee, which BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 8 1 periodically provides input and advice to the 2 EIA. This meeting is open to the public and 3 public comments are welcome. Time will be 4 set aside at the end of each day and morning 5 and afternoon for public comments. All 6 attendees, EIA employees and committee 7 members and members of the public, should 8 sign the sign-in sheet at the entrance and 9 please write down your phone number and 10 e-mail. 11 The other thing I would like to let 12 you know, there are restrooms. If you go out 13 of the door in to the hallway you turn to the 14 left and it's at the end of it. That's 15 important for the new committee members. 16 There is also a telephone here. We will come 17 to that in a minute. There is also a 18 telephone feed here if anybody needs to have 19 someone call you. The telephone number, I am 20 giving it you, it's 202-586-3071 or 586-6273 21 or 586-6202. Now, you don't need to remember 22 these numbers. I have them written down BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 9 1 here. If you need them, just come and ask 2 me. 3 Kathleen Wert is in charge of the 4 ASA and is helping the committee members with 5 reimbursement and other logistical questions 6 and she has a new help, Margarita Navos, who 7 will be helping her in that respect. On the 8 technical side each committee member is asked 9 to speak loudly and clearly, to articulate, 10 and please speak in the microphone such that 11 the recorder here can actually help in his 12 transcriptions. Now, finally before we start 13 I would like to welcome our two new committee 14 members, Cutler Cleveland from Boston. 15 Welcome. 16 MR. CLEVELAND: Thank you. 17 DR. HENGARTNER: And Mr. Darius 18 Singpurwalla. He recently moved to DC so 19 welcome. 20 MR. SINGPURWALLA: Thanks. 21 DR. HENGARTNER: Now that I have 22 introduced the new members I think it's just BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 10 1 befitting that we all introduce ourselves 2 starting with myself. This includes members 3 of the public here present. My name is Nick 4 Hengartner, I am from Los Alamos National 5 Laboratory, and I am serving here as the 6 chairman of this committee. 7 MS. KIRKENDALL: Nancy Kirkendall, 8 I am Director of Statistics and Methods Group 9 in EIA. 10 MR. CARUSO: Guy Caruso, 11 administrator of EIA. 12 MR. WEINIG: Bill Weinig, 13 Statistics and Methods Group, EIA. 14 MR. BERNSTEIN: Mark Bernstein on 15 the ASA Committee. 16 MR. CLEVELAND: Cutler Cleveland, 17 ASA Committee. 18 MR. EDMONDS: Jae Edmonds, ASA 19 Committee. 20 DR. FEDER: Moshe Feder, ASA 21 Committee. 22 MS. FORSYTH: Barbara Forsyth, ASA BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 11 1 Committee. 2 MS. KHANNA: Neha Khanna, ASA 3 Committee. 4 MS. SEREIKA: Susan Sereika, ASA 5 Committee. 6 MR. SINGPURWALLA: Darius 7 Singpurwalla, ASA Committee. 8 DR. SITTER: Randy Sitter, ASA 9 Committee. 10 MR. STRAUSS: Lawrence Strauss, 11 Statistics and Methods Group, EIA. 12 MR. WOOD: John Wood, Director of 13 Reserves and Production Division, EIA. 14 MS. ANDERSON: Margot Anderson, 15 EIA. 16 MS. FRENCH: Carol French, EIA. 17 MS. MILLER: Renee Miller, EIA. 18 MS. BLESSING: Colleen Blessing, 19 EIA. 20 MR. PSENG: Phillip Tseng, EIA. 21 MR. BROENE: Charles Broene, EIA. 22 MS. JENNINGS: Alethea Jennings, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 12 1 EIA. 2 MS. FRIEDMAN: Karen Friedman, EIA. 3 MS. SWEENEY: Amy Sweeney, EIA. 4 MR. WADE: Steve Wade, EIA. 5 MS. HOLTE: Susan Holte, EIA. 6 MR. HONEYCUTT: Crawford Honeycutt, 7 EIA. 8 MR. JOOSTON: Jim Jooston, EIA. 9 MS. BOEDECKER: Erin Boedecker, 10 EIA. 11 MR. MCDOWNEY: Preston McDowney, 12 EIA. 13 MR. FREME: Fred Freme, EIA. 14 MR. LU: Ruey Pyng Lu, EIA. 15 DR. HENGARTNER: Thank you very 16 much for introducing yourself. Nancy here is 17 very important to this meeting because she is 18 the designated federal officer. That means 19 that she must attend the meetings and she 20 may, if she wants, chair the meeting or call 21 a stop to the meeting if she deems it 22 necessary, and she must approve all the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 13 1 agendas and call the meetings of the ASA 2 Advisory Committee. So Nancy here is very 3 important. 4 MS. KIRKENDALL: The only thing is 5 that's probably not true any more because I 6 think that's a FACA thing. 7 DR. HENGARTNER: Now that I've 8 painted you in such a good light, Nancy. 9 I think this is going to be a good 10 meeting. I looked at the agenda that Nancy 11 provided and this looks like a really 12 interesting meeting. I think the last few 13 meetings generally have been superb. I hope 14 that this will continue and I hope that we 15 are all going to come away with some good 16 recommendations and some advice that will be 17 useful for the EIA. 18 So I will be the taskmaster. I 19 will keep you on schedule. That means we are 20 10 minutes late as we are. Guy Caruso will 21 give a talk about the state of EIA. 22 Unfortunately he is looking for his slides BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 14 1 and therefore Nancy Kirkendall will start as 2 presenter. 3 I think we are going to switch 4 Nancy and Guy and Nancy will start if that's 5 all right with you. And the other feature 6 for this morning is short-term range in 7 forecasting systems and most importantly I 8 want to remind the committee members there is 9 lunch provided, so the economists are wrong, 10 there is a free lunch, and it is going to be 11 located in the usual room, which is EA 226 on 12 the first floor. So you want to start, 13 Nancy? 14 MS. KIRKENDALL: As Nick said, we 15 are no longer under FACA. There are new 16 rules by the government ethics office. That 17 is why we got out from under FACA and they 18 were that the ASA president appoints members 19 and it's true we make recommendations but he 20 still sneaks a member sometimes but that is 21 more little challenges in communication than 22 anything else. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 15 1 And then the other main reason was 2 that we do not ask the committee to provide 3 consensus recommendations. We want your 4 technical recommendations and when of course 5 you all chime in and agree that carries more 6 weight than if it's one voice but we really 7 do like to hear a variety of opinions and, as 8 Nick said, we don't expect to operate any 9 differently than we have in the past. We 10 thought it has worked pretty well so we do 11 not see any reason to. 12 So my purpose today is to give you 13 updates, a preview of coming attractions 14 including this meeting and the next meeting. 15 So these are topics that you are going to 16 hear more about today. You have heard about 17 them in the past. And particularly this 18 morning we have a lot of discussion about the 19 short-term energy outlook in the past. 20 Phil Tseng and Dave Costello have 21 talked about their efforts in the modeling of 22 electricity. This is a big effort in EIA in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 16 1 the past. Our short-term energy forecasts 2 have all been done at the national level and 3 now we are trying to move to the regional 4 models so we have got a lot of issues to talk 5 about with you. 6 Margot Anderson is going to give 7 you background information and talk about big 8 picture items. Phil Tseng and Dave Costello 9 will tell you more about where they are on 10 the electricity regional modeling and Tanc 11 Lidderdale will talk about all the rest of 12 the regional modeling. So I think that will 13 be a very interesting session. 14 We have also talked a lot about 15 assessing EIA frames. This is an item in our 16 strategic plan is to take a look at our 17 frames and see how good they are. Howard 18 Bradsher-Fredrick will be talking about the 19 results from an EIA look at our frames. It's 20 really hard to have a good quantitative 21 assessment so it had to use a little 22 qualitative judgment in it but he will tell BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 17 1 you the results and what we've learned from 2 that. 3 We also have our colleagues at the 4 Census Bureau doing some comparisons of some 5 frames and you will have an update of that. 6 We will also have more information about that 7 project at the fall meeting because they'll 8 look at some of our bigger surveys. I think 9 today they're talking about two of our 10 smaller surveys where they've completed the 11 work. 12 As a number of you have heard, we 13 are doing usability testing of a new website 14 design. I think all of the committee members 15 should be there, two separate break-out 16 sessions so all of the committee members will 17 get to participate in the testing, and that 18 will be valuable information. 19 You will hear a new talk. This is 20 a new talk. Did we skip one? I did skip 21 one. This is continued. We are going to 22 hear about the natural gas production survey. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 18 1 There's a new survey. It collects monthly 2 natural gas production. The committee has 3 heard a lot about how we have estimated 4 natural gas production in the past 5 particularly in certain states, in Texas. 6 Randy Sitter's student did some work that we 7 actually used. 8 You heard about our efforts to 9 estimate in the Gulf of Mexico where there is 10 very little data and now what we have is a 11 new monthly survey. John was going to be 12 talking about the results. It started 13 operation in January. The first data are 14 just being processed and are close to being 15 ready to release. So he is going to ask some 16 advice on when do we decide that they should 17 be our official data, how do we decide that 18 they are accurate enough to release, and that 19 sort of thing. 20 826 is a monthly survey that 21 collects information on electricity sales and 22 resale and we are trying to work with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 19 1 estimation methods. Do we use outlier 2 detection, you've heard about that. We will 3 talk about it on Friday. We will talk about 4 it some more next time, I'm sure. 5 So this is an ongoing project and 6 then another major effort that's going on, we 7 talked last time about external evaluations. 8 As we told you then, we had a poor report in 9 the OMB's PART report. That's a performance 10 assessment rating tool. They go through and 11 ask a bunch of questions and we flunked in 12 terms of being able to say that we have 13 regular external reviews that look at our 14 relevance programs, are we doing the right 15 things, and we had two discussions last time. 16 One was Brenda Cox talking about 17 doing an evaluation of a survey program, and 18 she will give us an update on that work, and 19 the other one was even more exciting. The 20 committee came back and told us we needed to 21 have a high-powered review team modeled on 22 academic evaluations and so we actually have BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 20 1 a plan. We're going to go forward with it 2 and we will hear about that on Friday too. 3 So that one is particularly exciting. 4 I think I covered the first one. 5 The second bullet is declining response 6 rates. Our Residential Energy Consumption 7 Survey, Dwight French, they're looking at a 8 lot of different initiatives that might be 9 used for improving response rates for RECS. 10 I think one of the most interesting things is 11 that they are going to be doing an incentive 12 experiment with it. They're also looking at 13 other things that I think will be more 14 applicable to some of our other surveys so 15 interesting things going on there. 16 So these are topics that you will 17 probably hear about at the next meeting but 18 we did not have time to squeeze them in. 19 Either one of two things, we either did not 20 have time to talk about them this time or we 21 weren't ready to talk about them again. 22 There's post-stratification for the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 21 1 manufacturing energy consumption survey. 2 This is collaborative work with the Census 3 Bureau and so they'll come back and talk to 4 us again, at least we hope. 5 We had a group from my office did 6 some cognitive testing on the Form EIA 920. 7 That's a survey that goes to basically 8 manufacturing plants that asks about their 9 electricity generation and consumption. They 10 used cognitive interviews to redesign the 11 form and so this test is to try to figure out 12 what was the impact of that work. We have 13 done a lot of work since then and hope to 14 have a nice presentation next time. 15 Then the last item, customer 16 evaluations of our forecasting and analytical 17 products, these are surveys we have talked 18 about a couple of times. Last year we did a 19 survey of participants in our NEMS Conference 20 to get an assessment of our annual and 21 international energy outlook. 22 In the fall we did a similar survey BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 22 1 but we used a sample of subscribers to the 2 list serve for the short-term energy outlook 3 and we have decided that that methodology 4 works better so we are going to use the same 5 methodology in the next month or two to 6 assess the AEO and the IEO. One of the 7 recommendations of the committee last time 8 was that we should not do an annual 9 assessment of each product and so maybe in 10 the fall we can talk about how to schedule a 11 rotating assessment so that we can capture 12 things in a less burdensome manner for 13 everybody. 14 These are just other things that we 15 talked about. Work is going on, George Latey 16 (?) is continuing his work on diagnostic 17 modeling on NEMS solutions to try to 18 illuminate how good the forecast are to get 19 some sensitivity analysis for our 20 forecasting. Howard Bradsher-Fredrick is 21 going to present a paper on dual system 22 estimation at the Federal Committee on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 23 1 Statistical Methodologies Research 2 Conference, which will be in the fall, so 3 that should be interesting. 4 And then we have a session at the 5 joint statistical meetings in August on the 6 new natural gas production survey. It's 7 going to be an introduction to the problem, 8 description of the survey methods. We will 9 have done an extensive simulation including 10 the estimators that you recommended and then 11 John will also talk about what we're doing 12 now and how it's all going. And that is on 13 Wednesday at 8:30 in the morning so if you're 14 at the joint statistical meetings you should 15 come and hear the talk. 16 And that is all that I have so if 17 we have Guy's slides then I can turn this 18 over to Guy. 19 DR. HENGARTNER: I would like to 20 introduce Guy Caruso, Administrator of EIA, 21 who will give us an overview of what the EIA 22 is currently doing as well as tell us what we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 24 1 are going to pay for gas. 2 MR. CARUSO: There is a direct 3 correlation between the price of gasoline and 4 the amount of requests the EIA gets in hits 5 on the website. First let me congratulate 6 you, Nick, for promotion to chairman and 7 welcome, Cutler and Darius, to the committee. 8 I appreciate all of your time that you put 9 into this committee as colleagues that had 10 been involved for a number of years. 11 We really take your advice 12 seriously and it clearly makes a difference 13 in many of the things we do, as you'll see 14 from the agenda, in some of the products that 15 we've changed and the methodological changes 16 we've made so a big thanks for the time you 17 take to do this and I know the pay isn't that 18 great. 19 I want to talk a little bit about 20 some of our activities since the last meeting 21 and it's on the first slide and I think, as 22 Nick's comment implied, when energy prices BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 25 1 are high there is a real effect on our work, 2 not only the day to day products like our 3 weekly petroleum numbers, weekly natural gas 4 numbers, but currently once again an energy 5 bill is being debated on the Hill and we get 6 asked to do a lot of service reports related 7 to the bill or elements that are connected to 8 the bill and two weeks ago we released the 9 analysis of the National Commission on Energy 10 Policy that was a report put out in December 11 of '04 which makes a lot of policy 12 recommendations, many of which go well beyond 13 anything the Congress has actually passed or 14 even contemplates passing like substantial 15 increase in CAFE standards and things like 16 that. 17 But in any case it's an example of 18 things we're being asked to do. We're being 19 asked right now to do some work for Senators 20 Dorgan and Jeffords and Imhofe as well. So 21 there has been a lot of activities directly 22 related to debates on the Hill. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 26 1 To answer the next question 2 directly, the summer fuels outlook which we 3 released two weeks ago, we say that the 4 gasoline price is going to average $2.35 in 5 May which will be the peak. By Labor Day it 6 should be down to 2.20. 7 One of the things that occurred 8 since our last meeting was a substantial 9 revision in one of the weekly natural gas 10 storage numbers, actually Thanksgiving week, 11 and we've taken a hard look at our revisions 12 policy on the weekly data and just yesterday 13 on the web the Federal Register notice was 14 released with our new revisions policy and if 15 anyone is interested in getting into more 16 detail I'll be happy to do that but it's 17 another issue that I know we've discussed in 18 this committee at that specific report about 19 revisions policy in general, and then you'll 20 have on the agenda the natural gas production 21 survey which John Wood has talked to you 22 about in previous meetings and we'll get some BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 27 1 more reviews on that. 2 Nancy mentioned the new development 3 with an external study team being formed and 4 we'll have that on the agenda some new 5 developments with respect to the strategic 6 plan and I'll give you a few updates on where 7 we are with our '06 budget. In fact this 8 week we're meeting with all the officers to 9 talk about '07 so the budget's clearly an 10 important aspect of getting some of these 11 things down that we've talked about in this 12 committee. 13 The next slide just talks about 14 some of the things we've done for Congress. 15 I mentioned some of the service reports but 16 then there is the regular products such as 17 the winter fuels outlook which we do every 18 October and then the annual energy outlook in 19 both its reference case and the full report 20 which was released in late February. 21 We are working on the international 22 energy outlook now and will have that out by BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 28 1 early July. This next slide shows a couple 2 of these service reports, one on the mercury 3 controls that Senators Imhofe and Voinovich 4 requested. I mentioned the National 5 Commission on Energy Policy already. That 6 was requested by Senator Bingaman and there 7 are a couple of others in the pipeline now. 8 So I think the main point here is 9 that EIA is being asked to do even more on 10 what-if scenarios with respect to the current 11 energy bill and other energy policy changes 12 that are being contemplated and the modeling 13 and other techniques that have come before 14 this committee are being used extensively. 15 I already talked a little bit about 16 this, the average price for the summer, 2.28 17 with a peak in May. Of course, it seems to 18 have gotten more political attention. The 19 President's speech yesterday clearly was an 20 example of that and, of course, the bill was 21 passed by the House. The House version, HR 22 6, was passed last week. The Senate will BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 29 1 debate their version in the third week of May 2 and they're hoping to have something to go to 3 conference so that the President can sign it 4 by August, that's the goal. 5 The next one again talks about how 6 often we are being asked to provide data and 7 an analysis not only to Congress but the 8 administration and certainly lots of press 9 attention and our products continue to be 10 heavily used by the financial community and I 11 think we talked about last time that there 12 are actually now derivatives based on the 13 actual numbers that the EIA releases on a 14 weekly basis. 15 You can bet on what our natural gas 16 number's going to be at 10:30 this morning. 17 An actual financial instrument has been 18 created and is now up and running and we were 19 concerned about what impact that might have 20 and I think the main impact we have seen so 21 far is this very close scrutiny to the 22 numbers and any kind of deviation. For BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 30 1 example, the large revision I mentioned in 2 the week of Thanksgiving last year created a 3 huge stir and any time there is any kind of a 4 technical problem where the numbers are even 5 two minutes late from 10:30 we get a huge 6 amount of criticism from the media. 7 So I think for us as an 8 organization managing the information system 9 what it's done is put on more pressure for 10 both not only accuracy but timeliness and 11 things like revisions policy, which one 12 wouldn't think would get that much attention, 13 was a subject of a lot of trade journal 14 attention which I think in the long run is 15 good as long as the resources are there to 16 deal with it. 17 The real problem is increased 18 detention and requirements for greater 19 timeliness, even more attention to data 20 quality, which, as you know, in this 21 committee we pay a lot of attention to under 22 declining real resources so it's a real BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 31 1 challenge —————— storage I've already really 2 referred to that. That's really what we are 3 talking about here in not only the financial 4 instruments but the attention to the 5 revisions policy. 6 Next one is the production survey. 7 John Wood will go into more detail on the 8 later agenda item here but we are very 9 hopeful that that will provide more timely 10 and accurate monthly production data which, 11 as you know from past discussions in this 12 committee, we have relied exclusively on 13 state data. So this will be the first time 14 EIA collects that on a monthly basis directly 15 from the operators. The sample, the frames, 16 and the methodology we've discussed in this 17 committee. 18 We're very excited about the 19 initial results but I won't take anything 20 away from John Slater's comments so I just 21 want to mention that. The external review 22 team, I'm not sure if you had mentioned, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 32 1 Nancy, this was the outcome of the PART 2 process. Our OMB review downgraded us 3 because we have not had an independent review 4 of the full EIA program by outside experts. 5 We have had a number of independent 6 expert reviews on specific issues, 7 methodologies, and other more narrowly 8 focused reviews. There was a National 9 Academy Study in '91, but that was really 10 focused on NEMS so even that NAS wasn't a 11 full program review. So what we have done is 12 we have established a concept for an external 13 study team of five very senior people. Up to 14 now and I think I'd be announcing to the 15 public for the first time that Denny Ellerman 16 of MIT will chair that study team and he will 17 be the one selecting the other four team 18 members so there is indeed truly an 19 independent set of experts, and that will be 20 on the agenda later on so I won't go into any 21 more detail on that. 22 On the strategic plan you've heard BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 33 1 about that, a number of other meetings. We 2 have added a fourth goal which started out as 3 an IT, information technology, goal but as it 4 evolved it certainly has broadened to include 5 a number of other cross-cutting issues and we 6 will be discussing with you some of those 7 issues in more detail. The next slide shows 8 some of the things. 9 Four groups have been established 10 within EIA to discuss survey frames, IDC, 11 Internet data collection, things like 12 disclosure limitations and database 13 commonality and I think what's particularly 14 interesting about this is that there is a 15 great deal of interoffice cooperation on this 16 goal 4 and we see lots of opportunity for 17 improvements of efficiencies and the number 18 of things that we've been hoping to use the 19 IT goal to lead to so I think from the 20 perspective of the managers we're looking 21 forward to implementing that goal 4. 22 On the budget we have roughly an BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 34 1 $86 million request before Congress for FY 2 '06 and the House Energy and Water Committee, 3 which as of this budget our budget was moved 4 from Interior to Energy and Water so now we 5 are in the same budget as all of DOE except 6 for the weapons and NSA part of DOE and so 7 this will be a new process for us, a new set 8 of Congressional members to deal with, and I 9 have been spending a lot of time up there 10 meeting with many of the members of the House 11 Energy and Water Committee and they are going 12 to mark up our budget, I believe, May 12 is 13 the latest date I heard. So we are hopeful 14 that roughly 86 million, 85.9, is about a 15 $2-million increase over FY '05. 16 And a number of specific things 17 around the board, they were about what's in 18 this request for '06. We want to fund the 19 consumption surveys. We have been trying to 20 get by with a minimal amount of funding and 21 we've decided that it is affecting the 22 quality or could potentially affect the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 35 1 quality so we put some more money in '06 to 2 assure continued quality of the quadrennial 3 consumption surveys and the issue of cyber 4 security has clearly become even more 5 important and in particular with the 6 increased sensitivity of the weekly petroleum 7 and natural gas data. We hope to spend some 8 more money on that in '06. 9 There was a new voluntary reporting 10 of greenhouse gas, the 1605B, that Jae 11 Edmonds knows better than anyone else in the 12 room here. The EIA will be asked to collect 13 that data and it will add to our budget 14 requirements in '06. 15 So let me close by saying again how 16 much we appreciate your comments and advice 17 on the items we're going to be talking about 18 over the next day and a half. The next item 19 will be short-term energy outlook. Margot 20 will talk to you about some of the things we 21 are doing there and I mentioned the external 22 review team. Production survey, John Wood is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 36 1 with us. 2 We also are working hard to make 3 our website even more usable and 4 maneuverable. We've done a number of 5 customer surveys and we are trying to 6 implement those and John Weiner is heading up 7 an effort within EIA to do that and we look 8 forward to hearing from you about that as 9 well. And then there will be a number of 10 other statistical and modeling issues. 11 Welcome back to Washington for 12 those of you coming from out of town and 13 thanks once again and, Nick, I appreciate 14 your leadership on this. 15 DR. HENGARTNER: Don't go away. 16 MR. CARUSO: Any questions besides 17 the price of gasoline? 18 DR. HENGARTNER: I mean, the whole 19 audience is silent. That's a bad sign. 20 DR. FEDER: Guy, I noticed that you 21 had a lower bound on the price of barrel of 22 oil that was 50. Is there any upper limit? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 37 1 MR. CARUSO: Well, there are some 2 economists in here. In the short run it's 3 probably not much of an upper limit but in 4 our long-term outlook we have raised the 5 expectation where oil prices are going and 6 it's partly because of the increased cost of 7 finding and developing crude but in the short 8 run the real issue is that there is hardly 9 any unused productive capacity in the world 10 so there is no cushion to deal with shocks to 11 the system other than strategic petroleum 12 reserve and that, of course, is limited. So 13 in the upper bound there has been a report in 14 recent weeks from Goldman Sachs using $105 15 per barrel price. 16 If you put any price there in the 17 short run I think the real answer is at some 18 point there would have to be some reaction to 19 use the SPR to invoke the IEA short-term 20 response system so we have a statistical 21 boundary on our short term and the STEO does 22 that every month but I think that's a 95 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 38 1 percent confidence interval so it doesn't 2 cover the more extreme cases even. 3 DR. HENGARTNER: Neha. 4 MS. KHANNA: Since we're talking 5 oil I have to ask a question. What do you 6 think are possibilities of the effectiveness 7 of the open price band? I mean, it's been 8 violated ———————— the last two years but they 9 stuck to it for 15 years. What is the EIA's 10 expectation about where the price band is 11 going and how effective it will be? 12 MR. CARUSO: They suspended the 13 price band I guess it was in the fall of '04. 14 OPEC has a committee that supposedly is to be 15 reviewing this. We have some contact with 16 the members of that committee and they don't 17 want to set a new band partly because no one 18 knows where the price is going to settle out 19 so I'm not sure when or if OPEC will 20 reestablish that band. 21 I mean, when you talk to the Saudis 22 privately and even some of the public BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 39 1 comments seem to be they're thinking in the 2 $40-50 range but they are formally saying 3 that but that seems to be what they're 4 thinking. 5 We did in the AEO this year 6 something we haven't done before partly 7 because some people thought it might be too 8 politically sensitive, look at what the OPEC 9 revenues would be under various price bands 10 and what it showed was three high price cases 11 above the reference case. In two of those 12 cases the OPEC revenues were higher at lower 13 production which has always been ———————————— 14 revenues —————————————— 15 DR. HENGARTNER: Thank you very 16 much. Before we continue I'd like the people 17 who just walked into the room or who came 18 after we introduced ourselves to come up to 19 the micro and tell who you are and what your 20 affiliation is. 21 DR. NEERCHAL: I apologize for 22 walking in late. I think I am the committee BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 40 1 member who lives closest to Washington, DC. 2 I have the right to be walking in the latest, 3 I think. I'm Nagaraj Neerchal from the 4 University of Maryland Baltimore County. 5 MR. HOUGH: My name is Rick Hough. 6 I'm with the US Census Bureau. 7 MS. HAITOT: Nicki Haitot, US 8 Census Bureau. 9 MR. BOSTIC: Bill Bostic, US Census 10 Bureau. 11 MR. HSEN: Paul Hsen, US Census 12 Bureau. 13 MS. BUCCI: Susan Bucci, US Census 14 Bureau. 15 MR. BRADSHER-FREDRICK: Howard 16 Bradsher-Fredrick, EIA. 17 MR. FRENCH: Dwight French, I run 18 those pesky quadrennial consumption surveys 19 in the EIA. 20 DR. HENGARTNER: Thank you very 21 much. It's my pleasure to invite Margot 22 Anderson to come and talk to us about BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 41 1 regional and short-term energy outlook. 2 That's the STEO. 3 MS. ANDERSON: Good morning, 4 everyone. Thank you for having me. It's a 5 pleasure to speak with you all day about two 6 topics very close to my heart and the 7 business that I am required to do within EIA. 8 I am relatively new to EIA and I get to 9 oversee many of the activities that you are 10 going to hear about today which deal with the 11 short-term outlook and the quadrennial 12 surveys as well. I want to talk today a bit 13 about the project we have underway to 14 regionalize our short-term energy outlook. 15 I think in ASA meetings in the past 16 you have had a presentation on various 17 components. I want to put the rewind button 18 on and go back and do an overview of what 19 we're trying to accomplish and why we're 20 regionalizing and what some of the challenges 21 are. 22 Later today you are going to hear BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 42 1 from some of the folks that are very deep in 2 the actual modeling and you'll hear about the 3 electricity component of the regional steel 4 and you'll hear, I believe, about the propane 5 component and other components as well. 6 With us in the room are several of 7 the folks intimately involved with regional 8 steel, Mark Rodekohr over here who is leading 9 the charge, I see Tanc here and Phil and Dave 10 Costello and I am sure there are others but I 11 haven't put my glasses on. So when we get to 12 any question that I can't answer, I know that 13 they will help me out. 14 I want to talk about the purpose 15 and objectives, I want to talk about our 16 schedule, and I want to talk about the 17 overview of the actual model and the key 18 components of the model so you can see how it 19 all fits together, and then I want to briefly 20 address some of the challenges that we're 21 facing as we move from a national model to a 22 regional model, might I say, at the same BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 43 1 time. A lot of the work that we're doing to 2 regionalize this model is occurring while 3 we're putting out the monthly short term 4 energy outlook, all done with about the same 5 staff, and it has presented quite a challenge 6 to us to be going back and forth from the 7 building of the regional to putting out the 8 monthly actual outlook. 9 Why are we doing this? What is our 10 purpose and what are our objectives? We got 11 into this about mid-2003 and the objective is 12 to maintain a national perspective while 13 providing some regional richness to our 14 short-term outlook. For years we've been 15 presenting just the national numbers but we 16 know that the national number isn't fully 17 representative of what's going on in each of 18 the regions and so we would like to be able 19 to tell folks in the northeast or the 20 southwest or in California why they might be 21 facing a different set of energy prices, how 22 their demands are different vis-à-vis the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 44 1 entire US demand level or against each other 2 in the regions. 3 So it provides the kind of richness 4 that we often get asked about when people say 5 what's going on in my area, what's going on 6 in my State, how can you tell me something 7 more about how it will affect me as opposed 8 to a national average, so it provides this 9 richness and relevance to what we think key 10 customers want. In addition it adds this 11 depth. We gain a better understanding of 12 local regional supply problems. By zeroing 13 in and looking at what's happening in these 14 regions it helps inform the national 15 perspective as well. 16 So it provides a whole new set of 17 options for us to look at where there may be 18 problems that we want to zero in on whether 19 on the demand side or on the supply side. It 20 may add, and the caveat here is may, accuracy 21 to the national level estimates. 22 It's not clear to us at this time BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 45 1 whether the national forecast will be 2 significantly different than it would be from 3 using the national model that we do now but 4 we do think that by adding this regional 5 richness we may add some improved accuracy to 6 the national level forecast and I'm going to 7 be talking about accuracy in forecasts in my 8 second presentation today. 9 So this is some of the motivation 10 for why we got into this. Let me review the 11 schedule with you. Again, we began in 12 mid-2003 and we are scheduled for completion 13 in late summer of 2005. What are we doing 14 now? We're continuing to do the model 15 development. The very large part of doing 16 the regional model is dealing with the 17 electricity side of the model and that's why 18 we're spending so much time on it and why 19 you're going to hear about it in much more 20 detail today. 21 So we're undergoing model 22 development. Certainly testing goes on all BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 46 1 the time as we build modules and we preparing 2 the documentation. So in essence we're going 3 through a testing period, we're going to have 4 expert review of the documentation, and we 5 want to go live in late summer 2005. 6 We've got a lot of issues about how 7 we communicate what we're doing to the 8 public. We want to continue, of course, the 9 monthly release of a short-term energy 10 outlook. What will that look like? Will it 11 be more extensive than what we're doing now? 12 Well, sure, we're going to have a lot more 13 tables associated with the regional 14 components of the model. How much that 15 translates into a textual description of 16 what's going on is still undecided yet, how 17 much do we expand or contract or reformulate 18 the actual short-term outlook that you see. 19 It clearly is a lot more information but 20 trying to organize it in a way that will be 21 useful to the readership is an open question. 22 We currently have a downloadable PC BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 47 1 version of the short-term energy outlook. 2 Are we going to continue that when we have a 3 much more complicated model? This is an open 4 question for ourselves. We have a relatively 5 small but a hardcore set of users that like 6 to download the model and whether we're going 7 to be able to offer this under current budget 8 and resource constraints we don't know. We 9 would like to able to offer it to customers 10 in some version but this is again an open 11 question of who will be able to get access to 12 a downloadable version. 13 We'd like to expand the web 14 presentation and the query system to include 15 the regional detail. We may not be able to 16 publish all the information every month but 17 we're investigating whether there may be 18 additional information in an improved format, 19 i.e., a query system, that allows the 20 readership to get at the questions that they 21 want to get at without us having to package 22 it for them in output that we would release BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 48 1 every month. 2 So by adding this great deal of 3 complexity of the model we suspect that we 4 will have an increased number of customers 5 who are interested in the regional component 6 and how we present that information for 7 consumption is a big issue for us. Clearly 8 we will provide all the documentation as we 9 get it expert reviewed and available to those 10 folks that are interested in looking at the 11 specifics of what we did. 12 I suspect that while we will go 13 live we will continue to upgrade this model 14 over time. I don't think that once we're 15 done we're done. It will be an evolutionary 16 process of refining the model as we get 17 comment back and as we understand how people 18 are using it and as we can improve our access 19 and analysis of the data. So I'd like to 20 look at it as evolutionary with big points at 21 which we release things to the public. 22 Some of these things I've already BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 49 1 covered but I think most of you know that the 2 system that we use now, the STIF system, is 3 moving to the RSTEM system, which expands on 4 the STIF systems to include these regional 5 demand and price determinants for most energy 6 products. It's an econometric model that 7 captures demand, inventories, prices, and 8 it's used to forecast the things that Guy was 9 talking about earlier. It's used to support 10 the monthly outlook and it's used to support 11 the summer fuels and the winter fuels outlook 12 as well which will continue now with regional 13 richness. 14 RSTEM will allow more detail 15 treatment of the trends in the regions far 16 more than was possible under STIFS. The 17 frequency of the outlook will remain the 18 same. I should have mentioned that, I think, 19 in the previous slide. We are not going to a 20 longer or a shorter forecast period. We're 21 still going with the 12- to 24-month look 22 ahead. So that will stay the same under the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 50 1 new system. As an aside we often get asked 2 about forecast for the three to five-year 3 period. This model is not it; it is a 4 short-term energy outlook model. 5 What are the components of the 6 model? The components are similar to what 7 they are now only they're regionalized and it 8 makes it a whole lot more complicated and I 9 have a slide in a minute to demonstrate just 10 how more complicated it is. We have the 11 electricity side that looks at demand prices, 12 low generation, and dispatch model. Phil's 13 going to be talking about this later today. 14 A natural gas component that's looking at 15 prices in storage, supply, and demand, a 16 petroleum component looking at heating oil, 17 gasoline, propane in a macro bridge. 18 This is a gross simplification of 19 what's going on. There's a tremendous amount 20 of detail for each one of these components. 21 Some are more regional than others, some are 22 more detailed than others, some go from a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 51 1 national level and allocate out to a regional 2 or state level, and some build back up. It's 3 not the case that every component in this 4 model is a bottom-up approach. There's a 5 mixture and a back and forth between national 6 and regional components. Not all the regions 7 are the same for each sector. 8 The degree of regionalization 9 varies considerably. We have significant 10 regional detail and data for some components 11 of this model but not for others. And so in 12 some cases we can get down to more regions 13 and in other cases we are constrained by our 14 ability to have the data necessary to support 15 the kind of forecasting we think is 16 appropriate in this case. So you will see 17 significant detail in the natural gas and 18 electricity side of the model, less so for 19 some petroleum products and for some end 20 uses. 21 So there is a varying degree of 22 regionalization. I think that's critical to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 52 1 remember that not all regions will get the 2 same level of detail that they may be 3 expecting by going to a regional model. It 4 will vary tremendously based on the data 5 requirements and the data needs and the data 6 quality that we have for the different end 7 uses and for the different markets. 8 This is a little graphic that shows 9 you that in the STIFS model we're dealing 10 with about 1300 variables and we add up the 11 variables of what we got to keep track of. 12 This is not just the variables that are being 13 forecast. These are the variables and the 14 data that we got to keep track of within the 15 model to a model that's looking at about 16 15,000, almost 16,000, variables. The level 17 of complexity has increased many, many times 18 over. 19 So it's a much more complicated 20 model and the data requirements are huge and 21 a lot of the issues that we deal with are 22 really on the data side as opposed to on the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 53 1 model side making sure that the data that we 2 have go through the appropriate diagnostics 3 that are clean, that are in the shape that we 4 want them at the time we need them, and these 5 are all issues that we have to grapple with 6 not only within our office but we are 7 dependent upon others within EIA to get the 8 data that we need at the time that we need it 9 to make sure that they make it into the 10 monthly model. So there's a lot of 11 logistical problems that show up in moving 12 from a more simplified national model to this 13 much more complicated regional model much 14 less some of the complexity of the modeling 15 particularly on the electricity side. 16 What do we do when we go through 17 the usual procedures when we're building the 18 separate components of the model to judge the 19 accuracy and the capability of the model to 20 predict? We use generally accepted criteria, 21 we evaluate data obviously for measurement 22 error, we look at overall fits, we look for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 54 1 independence of residuals, all the classic 2 things, the good practices that you do when 3 building econometric or time series models, 4 to check to make sure that they're tracking 5 in sample forecasts and that they are 6 accurately predicting the future. Once you 7 put it all together you need to run the kind 8 of tests for the national and regional levels 9 as well as to look at the forecast as the 10 model interacts with other components and I 11 want to talk a bit more about some of those 12 complexities in the second half of my 13 discussion today. 14 Clearly there are lots of 15 challenges with upgrading the model and 16 moving to a much more complicated system. As 17 I've said three or four times already, the 18 electricity side is really the most 19 complicated, the most detailed information 20 that is available, and the most complicated 21 type of modeling that we do is in the 22 electricity sector of trying to determine low BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 55 1 generation and dispatch and regional demands 2 and regional supplies and thankfully we have 3 Phil and several others that are leading us 4 through this. 5 Another problem is avoiding 6 aggregation errors. When we're adding up 7 from a lot of different pieces there are 8 problems associated with whether what you get 9 at the top level doesn't contain all the 10 biases that you might be embedding in each of 11 the individual systems or the individual data 12 and equations. 13 Maintaining model and data quality 14 is another big issue. When we're talking 15 about as many variables as we are it takes a 16 considerable amount of resources to check for 17 measurement errors and data quality, on a 18 monthly basis and to try and build into this 19 system diagnostics to make sure that you're 20 not making gross errors in data is a key part 21 of this. A single person or a couple of 22 people can't eyeball all of this data to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 56 1 determine where there may be outliers or 2 missing data and trying to come up with 3 systems that can do this electronically is 4 certainly something we do on individual bases 5 but to put it together in a systematic way so 6 that we have the confidence that we're not 7 making data errors is absolutely critical. 8 I talked earlier about augmenting 9 the model through time and I think this is a 10 challenge that we will have as we move 11 forward and interact with our customers and 12 with analysts about how to improve the model 13 as we get better information on how people 14 are using it and getting the data in a more 15 timely fashion I think will change the way we 16 do the model over time. 17 Developing publicly accessible 18 outputs and products, this is a big deal for 19 us at EIA because we pride ourselves on our 20 transparency and our ability to get 21 information out to folks about how we do 22 stuff. We want people to open us up and take BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 57 1 a look inside and we need to develop an 2 appropriate approach to how we're going to 3 make what is now going to be a whole lot more 4 information available to the public and to do 5 it in a way that is digestible. And so 6 building on the website that we have to make 7 the documentation available to clearly allow 8 people to chart through what will be a lot 9 more data is going to be an important part of 10 communicating with our customers and getting 11 feedback from our customers about how well 12 this new system is working for them will be 13 an important part of this feature too. 14 So I hope to able to come back at 15 some time in the future and talk about how 16 we're going to ask people in a more formal 17 basis how are we doing, how's it doing, how's 18 the model doing, and is it doing what you 19 want it to do, and is it as good as what we 20 were doing before when we just had the 21 national model so assessment will be a big 22 part of our going forward program. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 58 1 Obviously we need to develop 2 appropriate diagnostic and model evaluation 3 tools. I alluded to this in talking about 4 model and data quality but model evaluation 5 tools are also critical and it's really the 6 issue that I want to get to in my next talk 7 about best practices for evaluating forecasts 8 in general and in essence to solicit some 9 advice on what we ought to be doing to 10 develop a program in EIA certainly within the 11 regional model that helps us gauge forecast 12 accuracy and do a report card on ourselves as 13 well as be able to compare ourselves with 14 other forecasters to get a sense of how we 15 might improve the model based on some 16 diagnostics and some forecast accuracy tools 17 that really ought to be a part of our core 18 program. 19 So I would like to end this 20 presentation and maybe answer a few questions 21 or go into the next one. What do you want me 22 to do? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 59 1 MR. HENGARTER: That's the nice 2 thing of having to talk separate in two 3 pieces. It allows the public and the members 4 of the committee to ask questions in the 5 middle without interrupting your flow. 6 MS. ANDERSON: Great. 7 MR. HENGARTER: So if there are any 8 questions it's a good time. Yes? 9 DR. NEERCHAL: I'm sure the data 10 quality and the availability vary from region 11 to region quite a bit. Was that also part of 12 the consideration when you decided the degree 13 of regionalization? 14 MS. ANDERSON: Sure, particularly 15 availability, yes, it was, and I don't know 16 what more to say about that but certainly the 17 data quality and the data availability have a 18 lot to do with whether you're breaking it 19 down by PADD regions or whether you can get 20 down to some state level data for some 21 components of the model. And we hope that 22 supply might create its own demand or that by BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 60 1 having the model and by making the model 2 available and by people saying that it could 3 be a useful tool for predicting at a regional 4 level might help upgrade the quality of some 5 of the data or expand some of the data that 6 some people are collecting whether it's us or 7 whether it's others. 8 DR. NEERCHAL: So how much worse 9 does it get when you go from this national to 10 regional? 11 MS. ANDERSON: Well, depending on 12 the sector you're looking at it can get real 13 bad. I mean, in many ways we're aggregating 14 up from not very good data, aggregation 15 problems. In other cases we're just using a 16 much more aggregated number and parsing out 17 by what we know about demand or supply in 18 individual regions. We don't really have any 19 regional richness but we may have to infer in 20 some areas based on historical data. 21 MR. HENGARTER: Jae? 22 MR. EDMONDS: Margot, can you just BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 61 1 give us a sense of the big numbers that are 2 going to be coming out and that are going to 3 be the focus of attention that could come out 4 of the short-term model and what the key 5 numbers going in to the model are just to get 6 a sense of the kinds of problems you have to 7 deal with? 8 MS. ANDERSON: Well, in terms of 9 output I think we're going to focus on the 10 kinds of things we do now in the national 11 model, prices, demands, supply. I mean, 12 right now what we report out in monthly are a 13 long conversation about what we think the 14 prices are going to be for natural gas, 15 petroleum products, electricity, coal, and we 16 also have a discussion on the demands for 17 each, somewhat less discussion on supply for 18 each of those fuels although we usually have 19 in a paragraph or two about world oil 20 balances because that seems to be a primary 21 focus. 22 In some ways your question raises a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 62 1 question for me about whether the shift in 2 the emphasis will change when we go from 3 national to regional. Right now our focus is 4 really on oil markets and has been for some 5 time lately because of all the increases but 6 it may be that customers are going to want to 7 rebalance our emphasis in a sense of looking 8 at spending more time looking at regional 9 breakouts of prices as opposed to a discourse 10 on the national level. 11 So the outputs will be generally 12 the same. They'll just be more detail on 13 them and so we would be able to look at 14 gasoline or propane prices by our regions and 15 be able to let customers know that if you're 16 living on the west coast your gas price path 17 is likely to look different than if you're 18 living in the southeast. 19 In terms of the input to the model 20 the numbers are similar. The information 21 that you need to predict it at a regional 22 level is similar to what you need at the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 63 1 national level. There's just a whole lot 2 more of them and you may have to fill in 3 where there are some gaps of missing data at 4 the more regional or state level. Does that 5 get at what you're asking? 6 MR. EDMONDS: Actually I was just 7 thinking if you could enumerate the key 8 drivers to the short term forecast. You've 9 got some big benchmarking drivers like you're 10 macro-economic variables coming in but then 11 what are the other things that you're going 12 to be looking for on the outside to drive the 13 development of price forecast? 14 MS. ANDERSON: Mark, you want to 15 answer? 16 MR. RODEKOHR: Yes, there are 17 several factors. One is weather. Clearly if 18 we have a national weather forecast that it's 19 going to be a normal winter -- 20 MR. HENGARTER: Mark, come to the 21 table. 22 MS. ANDERSON: If you volunteer you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 64 1 got to use the mic. 2 MR. RODEKOHR: There are several 3 factors and the weather is one of them. For 4 example, you can have a national weather 5 forecast that says it will be normal for the 6 winter; however, that masks regional 7 differences, colder in the northeast, 8 whatever the case may be. So that can and 9 has made an impact on average prices for the 10 country and certainly prices in regions and 11 we've known this for years now. That's one 12 factor. 13 Another factor is regional 14 differences in prices with related things 15 like gasoline. Refinery outages here and 16 there can make a difference. We're not 17 tracking every refinery outage for every week 18 but nevertheless you can have some 19 differences. 20 In terms of electricity it's 21 transmission constraints, it's distribution 22 constraints. For example, you have first run BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 65 1 plants. We've disaggregated our electricity 2 dispatching into four categories roughly. 3 First of all you're going to use all your 4 hydro. You're going to use all your 5 renewables, which is next to nothing. 6 First the real issue on electricity 7 dispatching is what you do with coal, oil, 8 and natural gas and those depend on relative 9 prices and a lot of other factors and we're 10 trying to get a handle on that. And that's 11 probably the biggest cost part of the 12 regional STEO is trying to get handle on 13 that. We're also focusing on four states, 14 Texas, California, New York, and -- what's 15 the other one? 16 MR. TSENG: Florida. 17 MR. RODEKOHR: Florida, very 18 helpful, thank you, and Phil's going to tell 19 you all about this this afternoon. 20 MS. ANDERSON: Mark, talk a little 21 bit about how we disaggregate the macro 22 component that we're using at the national BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 66 1 level to look at the regional level so we can 2 perhaps try to break down the -- 3 MR. RODEKOHR: Well, I don't know a 4 whole lot about that but we are trying to 5 look at regional macro components as well. 6 Clearly some of the industries are obviously 7 affected differently by higher energy prices 8 in other industries and those are very 9 regional in nature and we are trying to get a 10 better handle on that as well. As Margot 11 said, whether this leads to a better national 12 forecast is one issue. 13 The other issue is can this lead to 14 a better forecast in terms of what policy 15 makers are worried about and that's one of 16 the key considerations that we're after 17 policy. Many policy people are very worried 18 about what happens in their particular region 19 or part of the country and that's what we're 20 trying to address with this effort. That's 21 part of what we're trying to address. 22 I personally have no view if we're BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 67 1 going to have a better national forecast but 2 clearly when you look at the press, when you 3 look at the policy issues, they're not all 4 just national in order. Margot, do you want 5 to -- 6 MS. ANDERSON: Are there other 7 questions? John? 8 MR. WOOD: Do I have to come to the 9 mic? 10 MR. RODEKOHR: The mic's over 11 there, by the way. 12 MR. WOOD: I would just mention 13 that a component that we've looked out a lot 14 in the last five years, natural gas, detail 15 of the quality, how well you have to know 16 some of the inputs to have a good chance at 17 really capturing natural gas prices, for 18 example, is amazingly what, tight. It looked 19 like just a couple of percent difference 20 between natural gas supply and demand moved 21 the natural gas price from $9 an MCF to 2 in 22 the 2001-'2 period and you can get a feel for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 68 1 what moves it and then where the natural gas 2 supply might be on a regional basis would 3 have given you a lot of insight. During that 4 period of time there was plenty of natural 5 gas supply in the Rocky Mountains but there 6 were shortfalls in the Gulf of Mexico with 7 the main distribution system is so I think 8 that idea of the regionalization gives you a 9 lot better insight into how ———————————— 10 MS. ANDERSON: Well, I hope so. 11 It's a be careful what you ask for kind of a 12 thing. We're going to have a lot more to say 13 about what's going on in the country and the 14 pressure to be as accurate and timely as 15 possible has increased considerably so 16 there's just a lot more to look at and I 17 think there are lot more sensitivities when 18 you are looking at it on a regional or state 19 basis than there would at a national level. 20 So it's going to be a very interesting year. 21 MR. HENGARTER: Last question, 22 Mark? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 69 1 MR. BERNSTEIN: More concerns and 2 observations and I know you realize I'm 3 really concerned about that 16,000 number 4 down there. I'm sure we'll get into it with 5 Phil because the bulk of that is the 12,000 6 dispatch number but that's a real concern on 7 a monthly basis to be able to keep track of 8 things and make sure they're consistent. I 9 know you realize that but I just needed to 10 say it. 11 I do believe you're in the right 12 direction on disaggregating regionally. 13 We're doing some work now on regional 14 disaggregation of price elasticities and we 15 see significant differences between regions. 16 So it will clearly affect forecast but the 17 other observation I want to make is more of a 18 broader policy issue in EIA is the problem of 19 forecasting versus assessing the risks of 20 problems. 21 What I would want to get out of a 22 regional short-term energy outlook are things BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 70 1 like what's the risk in southern California 2 of having blackouts this summer, assessing 3 the risks next winter in the northeast of 4 having significantly higher heating oil 5 prices, things like that as opposed to a 6 forecast which no matter how good we are at 7 forecasting we know there are problems but 8 more in a sense of what are the conditions 9 likely to lead to problems so that we can 10 have red flag signals. 11 It's the same issue I have with the 12 annual energy outlook too but I think it's 13 more relevant for the short term energy 14 outlook and so I'd like you to be thinking 15 about and considering a different approach to 16 presenting the information and evaluating 17 options and I think there are hot point 18 things that are problems and there are 19 regions where there aren't any problems and 20 you can change your efforts to emphasize 21 things where you notice there are going to be 22 problems versus places that are not. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 71 1 MS. ANDERSON: Can I comment on 2 that? 3 MR. HENGARTER: Yes. 4 MS. ANDERSON: I think that's a 5 great point. We certainly tend to do more 6 risk assessment on the international oil 7 side. We do choke point analysis, et cetera. 8 We do at DOE we have an office of energy 9 assurance, energy transmission and assurance 10 I think it just changed it's name to, that 11 was an office that came out of the first term 12 of the Bush administration to assess 13 vulnerabilities to the energy system and 14 certainly they do some of that work. Whether 15 EIA can be of assistance in using a model 16 like this to identify where there might be 17 these kinds of constraints certainly would be 18 worth looking into. 19 MR. BERNSTEIN: They don't put out 20 stuff that -- 21 MS. ANDERSON: They won't put it 22 out and they may not put it out for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 72 1 sensitivity reasons or for policy reasons 2 because they may be assessing other kinds of 3 threats as well but you're right. I don't 4 think you'd find a published place where you 5 could put a probability although the policy 6 office used to do some of that as well of 7 trying to assess where there may be working 8 with NERC to look at the summer electricity 9 outlook but you're right. There's no 10 systematic way of doing that and it's not 11 model based. So the points well taken that 12 models may be able to be used for that and 13 then what is the obligation of an EIA of 14 putting that kind of information, making it 15 that available. It's a good point. 16 MR. BERNSTEIN: The key is that 17 just there's nobody out there responsible for 18 noting that there may be problems out in the 19 future. The only place that actually has 20 that information is EIA. 21 MS. ANDERSON: What about NERC on 22 the electricity side? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 73 1 MR. BERNSTEIN: NERC's stuff is 2 pretty poor. My one anecdote example is in 3 2000-2001 time frame in California when NERC 4 said that there are going to be no problems 5 in California in the next six months and then 6 all hell broke loose. 7 MS. ANDERSON: Well, that leads us 8 to our next discussion about how do you 9 forecast well. 10 MR. CLEVELAND: Just a brief 11 comment to follow up on Mark's, I think that 12 once you demonstrate this capability to do 13 this regional type of work you will be given 14 this kind of work because in essence the 15 special requests you get are often regionally 16 driven, many of them, and so now all of a 17 sudden if you are going to be able to say 18 things specifically about the California 19 electricity market or the New England heat 20 oil market you're going to get requests from 21 people from those regions to do this type of 22 work. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 74 1 I think Mark's point is a good one 2 that should you be doing it pro-actively I 3 think is an important issue to discuss but 4 you'll be doing a lot more of that type of 5 work and you're going to be under even more 6 scrutiny because you're going to be saying 7 things that are with a lot more specificity 8 and as soon as you start saying things with a 9 lot more specificity your models or your 10 numbers are going to be scrutinized even more 11 closely so be ready. 12 MS. ANDERSON: Dave, Tanc, anything 13 you want to add to that? Mark? 14 MR. RODEKOHR: I'd like to add a 15 little bit to that. Two things I think 16 you've said, I think Mark has made a very 17 good point in the sense that we could add to 18 our STEO I won't say a model-based assessment 19 of risk but where we think things could 20 potentially be problems and I see no problem 21 with that at all because we know what it is. 22 Is it based on a model? Maybe yes, maybe no. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 75 1 Setting expectations for the 2 regional model, I think, is very important 3 and the things you just said are the things 4 that we're going to need most. We do not 5 have transmission link models embedded in 6 this. We have a regional forecast but we 7 don't have all the transmission links. 8 And we've already, trust me, been 9 getting many questions about if pipeline goes 10 down for X number of days what's going to 11 happen to the world and we keep trying to 12 answer these questions but we don't have a 13 modeling technique to do it. So I think we 14 need to set our expectations better but I 15 agree with you. A risk assessment is saying 16 in a summer or winter outlook what are the 17 problems we think could happen not based 18 necessarily on a modeling exercise because we 19 do not have the detail of a NEMS or another 20 huge model that looks at all the 21 transportation links. We never pretended 22 that we had this is not a bad idea at all. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 76 1 But it is more notional than it is 2 modeling based and the regional model has 3 never been built as something that will 4 capture all the transportation links and all 5 the things that certainly policy makers at 6 times are worried about. Let me give you a 7 few examples here. 8 Our projections are monthly. Often 9 times events happen that last a few days or a 10 week. We have no way of knowing those few 11 days or a week in a modeling exercise are 12 going to happen so we need to tailor our 13 product to what they need but nevertheless 14 never over-promise at the same time. But the 15 notion of a risk assessment, I think, at 16 certain times a year is not a bad idea at all 17 but we have not simply buried in certain 18 sentences or paragraphs some of that stuff. 19 MR. EDMONDS: A quick point of 20 clarification here, one of your slides when 21 you did the variable count you distinguished 22 between econometric and dispatch variables. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 77 1 I know what an econometric variable is but 2 what's a dispatch variable? 3 MS. ANDERSON: Phil, help me. 4 MR. TSENG: I'll spend ———————— 5 minutes with you. 6 MS. ANDERSON: And on the numbers 7 that 12,000 number may go down considerably 8 if we can find a way to streamline on this 9 dispatch model. So we throw that up to 10 demonstrate where we are now, having to track 11 all that. 12 MR. EDMONDS: So what it's saying 13 is that that's an electricity dispatch? 14 MS. ANDERSON: Yes, it is, 15 electricity -- 16 MR. EDMONDS: And basically it's 17 all coming in from outside? 18 MS. ANDERSON: Well, all coming in 19 on that one component of the model that's 20 dealing with trying to find the optimal way 21 that the plant just sends out the power at 22 what time. And we have all this plant data BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 78 1 and that's generating all these different 2 variables that can be used to assess who's 3 getting electricity when based on demands and 4 on prices. And so because we have so much 5 detail you've going to have plant by plant by 6 plant all these variables that are 7 contributing to that number but it is 8 electricity dispatch. 9 DR. HENGARTNER: I have the feeling 10 we'll be hearing a lot more of this. The 11 next topic -- 12 MS. ANDERSON: I'll talk really 13 fast. 14 DR. HENGARTNER: Yes, it's a 15 pleasure to continue Margot's exposition. 16 MS. ANDERSON: Thank you. I want 17 to talk about performance indicators, 18 diagnostics, and forecast errors. This is an 19 issue that I've been thinking about is, I 20 have come to EIA about what can we do to 21 improve our ability to assess the accuracy 22 and the quality of our forecasts. The caveat BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 79 1 here isn't to beat ourselves up if we're not 2 perfect every month. 3 The idea about doing this is just 4 to get a better sense of how we can improve 5 our own models by coming up with a program 6 that continually allows us to assess what 7 we're doing and what's the appropriate report 8 card that we ought to use. So I want to talk 9 a bit about past and current practice, 10 principles and tools for forecast evaluation, 11 and some things we're thinking about as we 12 consider designing a good practices program 13 for assessing our forecast. 14 Well, I've hinted at why evaluate. 15 I mean, to many this is an obvious thing to 16 do. We want to assess the accuracy of what 17 we're doing, we want to improve the data in 18 the models, we want to enhance accuracy and 19 credibility, we want to increase 20 transparency, we want to help demonstrate the 21 complexity of forecasting. 22 Unless you're constantly reviewing BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 80 1 what you're doing it's difficult to 2 communicate why you might not be doing it as 3 well as some people think you ought to be 4 doing it who are not in the game. Having 5 said all of this, it's not so easy just to 6 come up and say here are a couple of tools 7 and if you just do this every month you'll be 8 better off and you can tell people you're 20 9 percent, 30 percent, 2 percent off. 10 It's using the information to 11 evaluate what you're doing and it's not 12 looking at all 12,000, 1400, whatever it is, 13 number of variables. It's really designing a 14 program that looks at some key things that 15 can really help you better understand what 16 you're doing within the models and why or why 17 not you may be missing what the actual 18 outcome is and there may be obviously some 19 really good reasons that you're not going to 20 ever capture within a model. 21 But I think there is an impetus to 22 do this at varying levels. You can have a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 81 1 $20 program and you can have a $100 program 2 and it's deciding which program would be best 3 under this context. During the '80s and the 4 early '90s we routinely published an annual 5 supplement to what was then a quarterly 6 short-term outlook model and we put out a 7 series of forecast errors and looked at some 8 key variables and said here's how we're doing 9 and here's why it's different than what it 10 actually was and explained what might have 11 been going on in those markets and why we 12 missed it and talked a bit about what we 13 might do to improve models. And so there was 14 a good general discussion that came out every 15 year for people to take a look at. 16 That was discontinued in the early 17 '90s due to a variety of reasons, the biggest 18 of which was budget. Again, STEO was a 19 quarterly model. Accuracy, it was typically 20 absolute average errors and percent errors. 21 We didn't go any deeper to look at any 22 regression-based tests to determine what's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 82 1 going on and what's missing and what you 2 might get from additional information. It 3 was a fairly simple way of looking at what 4 did you predict, what was it actually, how 5 much were you off. 6 Now, currently several of the 7 programs, NEMS does publish forecast errors 8 with their annual report but we in STEO don't 9 publish our forecast errors. Each analyst 10 certainly keeps track of forecast errors for 11 their components and we keep track internally 12 of forecast errors for some key variables. 13 We don't have an externally 14 published document nor do we typically share 15 our forecast errors or our discussions about 16 forecast errors. We haven't done that some 17 time and that's one of the open questions 18 that we'd like some guidance on, on what 19 might be an ideal program to get back into 20 this game and what should we be doing. 21 I have some examples of what we 22 keep track of internally and, again, these BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 83 1 are just the percent differences between a 2 quarterly forecast, a quarterly average, 3 because the STEO is, of course, now monthly 4 and what the actual was. And I'm putting 5 these up here not so you could say they were 6 really off. I'm just putting these up here 7 to show you what we keep track of and we have 8 a series of about 12 or 14 of these graphs 9 that we keep track of and take a look at to 10 just tell ourselves are there any obvious 11 things that we're overlooking, are there any 12 trends going on here, but we don't do more 13 analysis on this. Again, we don't look at 14 correlation among the errors. We eyeball 15 this, we say where have we been under- 16 looking or overlooking, no surprise that in 17 areas like international oil prices were less 18 good than were on some of the demand 19 equations or on some of the demand forecasts 20 which tend to be more stable over time. 21 So I got a couple of them here. 22 Here it is for gasoline price. This goes BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 84 1 from '95 to 2004. You look at it and worry 2 whether there are any more systematic things 3 or gee, what happened back there in 2001 when 4 we were really off. The scales are different 5 for each graph. The percent is different for 6 each one. They're not comparable by eyeball. 7 Some are moving in a much tighter band, some 8 are in a much smaller band. Here is for 9 petroleum band where the band is really much 10 tighter. This is 3 percent error on either 11 side. So this is just an example of what we 12 have internally but you don't see this. 13 Should a forecasting program show 14 this to you every month? Should we show it 15 to you once a year? What should we show to 16 you? What should be the number of variables 17 that we look at? How deeply should we look 18 at them in what context are questions that we 19 have. 20 Obviously there are lots of 21 candidate metrics, some more appropriate for 22 time series, some more appropriate for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 85 1 econometric modeling, and there are any 2 numbers that you can pick from moving from 3 something fairly simple to something much 4 more complicated doing modeling on the errors 5 themselves to try and figure out what's going 6 on. 7 So there's a variety to choose 8 from, none of this new to you, and I want to 9 talk a little bit about basic principles, 10 about if you were going to design best 11 practices for forecast performance what would 12 you be worrying about. You'd be worrying 13 about unbiased forecast, you'd be worrying 14 about efficient forecast, and you'd be 15 worrying about uncorrelated errors. 16 What's interesting is we often talk 17 to other modelers who have a whole lot more 18 data than we do, access to maybe a whole lot 19 more data, particularly in the commercial 20 sector, and the question always comes up in 21 my mind if we had access to that data too, 22 which are often very expensive, and maybe BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 86 1 highly detailed data about production in 2 every field and every country would we be 3 able to have a better forecast if we had that 4 data. What's the value of getting that 5 additional information and is it worth EIA to 6 go out and get that information, purchase 7 that information from commercial sources, 8 because it will improve our forecast? 9 Certainly at first blush it sounds as if we 10 need more data, we need better data. Without 11 really doing an assessment of what that 12 better data might buy you it's difficult to 13 make that tradeoff. 14 So these are some of the things 15 that you'd want to be able to accomplish in a 16 forecast evaluation program. And if you were 17 designing a forecast evaluation program from 18 scratch what would you worry about? You'd 19 worry about which tests to apply. You can't 20 apply 20 tests. What's the smallest number 21 of tests that you could apply to give 22 yourself the confidence that you need to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 87 1 answer your core questions? How often should 2 you do this, every month with all those 3 numbers that are in the regional STEO? Is 4 once a year okay? Is every quarter okay? 5 Which forecast, just a one step ahead, or you 6 do an average over your year or over your 7 quarter or over two years? Which data do you 8 use, the data that you are using when you 9 make the forecast or your real data when you 10 go back and you adjust for all the numbers 11 that were preliminary when you made your 12 forecast? 13 There are two schools of thought on 14 which are the best data that you judge your 15 actual from your forecast number. Which 16 variables? We have hundreds of variables 17 that we will be predicting the outcome. 18 Which variables are we going to look at? 19 What are the most important? When we move 20 from the national to the regional model that 21 changes dramatically and there are going to 22 be a lot of interest for some regions and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 88 1 less interest for others. How do we pick in 2 an objective way which variables we're going 3 to track or that we're going to test? 4 How do you use the results? Again, 5 the results aren't necessarily just for a 6 report card to the public to say we're better 7 than somebody else. The results are to help 8 us do diagnostics and improve the model. My 9 first goal is really for ourselves to do 10 better and to have fairly simplified 11 electronic ways to improve what we're doing 12 particularly as we move from a national to a 13 regional model. 14 It's not to flagellate ourselves 15 and the public. It's a diagnostic tool. 16 Certainly my gut feeling is that our forecast 17 can stand up to anybody's forecast and that 18 we're typically in very good shape on a lot 19 of variables and I think that we've 20 demonstrated that over the past. It's not a 21 competition. It's about improving what we 22 do. So that's how to use the results. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 89 1 Should we compare ourselves with 2 commercial forecasters? We like to do that a 3 lot, that we were more accurate than X, Y, or 4 Z. Should we do that in a more formal basis? 5 Should that just be an internal thing? Why 6 would we want to do that? How will we use 7 their information to improve our forecasts? 8 And how do we communicate with 9 customers? Again, it goes back to the issue 10 I mentioned earlier. Is this something we 11 publish every year, is it something we put on 12 the web, do we have a discussion with our 13 customers, how do we communicate what this 14 program might look like to those that might 15 be interested in assessing what we're doing? 16 I'm a manager in EIA. I always got 17 to worry about the bottom line and who do we 18 have to do this and how we're going to get 19 this done under, as Guy said, declining real 20 bud