AMERICAN STATISTICAL ASSOCIATION (ASA) MEETING OF THE AMERICAN STATISTICAL ASSOCIATION COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION (EIA) DAY ONE Washington, D.C. Thursday, October 5, 2006 2 1 P R O C E E D I N G S 2 (8:40 a.m.) 3 MR. HENGARTNER: Good morning, all. 4 It seems I need a gavel here to call us to 5 order. Of course, I don't have it. 6 Good morning. Welcome all. This 7 is a -- I'd like to open the meeting. And I 8 want to do that by reminding you that this is 9 an ASA and not an EIA committee, and it meets 10 periodically to provide input and guidance 11 and advice to EIA. 12 This meeting is open to the public. 13 I hope there is someone here. And public 14 comments, of course, are welcome. There's 15 time set aside for public comments at the end 16 of each day as well as the opportunity to 17 submit written comments either at the end of 18 the meeting or throughout the year. Your 19 comments can be sent either to the ASA or the 20 EIA. 21 All attendees, including guests and 22 EIA employees, should sign in at the register BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 3 1 in the hall and include your e-mail address. 2 Restrooms are at the end of the 3 hall. You go out that way or this other way, 4 and you know where they are. If not, I'll 5 show you. 6 There's a telephone in this room. 7 It is a shared number. Let me give you that 8 number in case you need to be reached. It's 9 (202) 586-6273. Again, (202) 586-6273. 10 Kathleen Wert and Donna Renee 11 Arrington are with the ASA committee. Are 12 they here? No, they're outside manning the 13 table. And if you have any questions about 14 expenses or travel reimbursements or so 15 forth, please address those questions to 16 them. They will be glad to help you. 17 In the past, we had someone writing 18 a transcript. We are still doing this, so I 19 please ask you to speak up, and if you talk 20 from the seats in the back, go to the 21 microphone. That will help us all to get the 22 transcript. There was an indication how to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 4 1 use a microphone. I will skip this, but I 2 will keep the opportunity to remind you that 3 if you don't do it right, I'll go and read 4 this explicitly. 5 Finally, I'd like to 6 announce -- before we start, I'd like all of 7 us to go around the table and introduce 8 ourselves as well as the members of the 9 audience, and I will start with myself. 10 Again, please use the microphone. 11 So, my name is Nick Hengartner and 12 I'm from Los Alamos and the chair of this 13 committee. 14 MS. KIRKENDALL: I'm Nancy 15 Kirkendall. I'm director of the Office of 16 Statistics and Methods Group, whatever office 17 I'm in. 18 MR. CARUSO: Guy Caruso, 19 administrator of the EIA. 20 MS. KHANNA: Neha Khanna, State 21 University of New York at Binghamton. 22 MR. BURTON: I'm Mark Burton, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 5 1 University of Tennessee. 2 MR. SINGPURWALLA: Darius 3 Singpurwalla, I'm at Klemm Analysis. I got a 4 new job. 5 MS. SEREIKA: Susan Sereika, 6 University of Pittsburgh. 7 MR. NEERCHAL: Nagaraj Neerchal, 8 UMBC. 9 MR. CLEVELAND: Cutler Cleveland, 10 Boston University. 11 MR. EDMONDS: I'm Jae Edmonds at 12 the Joint Global Change Research Institute 13 and Pacific Northwest National Laboratory. 14 MR. FEDER: Moshe Feder, RTI. 15 MS. FORSYTH: Barb Forsyth, WESTAT. 16 MR. HILL: Walter Hill, St. Mary's 17 College of Maryland. 18 MR. McDOWNEY: Preston McDowney, 19 Statistics and Methods Group. 20 MR. HENGARTNER: And the audience, 21 if you want to please walk to -- 22 MR. HELKIE: Bill Helkie, senior BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 6 1 advisor at EIA. 2 MS. HOJJATI: Behjat Hojjati, EIA. 3 MR. TSENG: Phillip Tseng, EIA. 4 MS. MILLER: Renee Miller, EIA. 5 MS. WAUGH: Shawna Waugh, 6 Statistics and Methods -- 7 MS. BLOMBERG: Carol Blumberg, 8 Office of Oil and Gas, EIA. 9 MS. WAUGH: Shawna Waugh, 10 Statistics and Methods Group, EIA. 11 MS. JENNINGS: Alethea Jennings, 12 EIA. 13 MR. BROON: Tom Broon, SMG at EIA. 14 MR. BRADSHER-FREDERICK: Howard 15 Bradsher-Frederick, EIA. 16 MS. SWEENEY: Amy Sweeney, EIA. 17 MR. STROUD: Lawrence Stroud, 18 Statistics and Methods Group, EIA. 19 MR. RASMUSSEN: Eric Rasmussen, 20 EIA. 21 MR. RUTCHIK: Bob Rutchik, EIA. 22 MR. WEINIG: Bill Weinig, EIA. I'd BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 7 1 like to mention that we have a constant 2 ringing in the room. We know about it. It's 3 not reparable and I'm sorry for the 4 inconvenience. It's with the air 5 conditioning system, so I think we're going 6 to have to suffer through it. I'm sorry, but 7 that's just reality. 8 MR. HENGARTNER: Okay. Thank you 9 very much, everybody. I'd like to inform you 10 that under the terms of the DOE grant to the 11 ASA, Nancy Kirkendall may chair, but must 12 attend each meeting. 13 And she's authorized to adjourn 14 this meeting if she determines this to be in 15 the best public interest. Must approve all 16 meetings of the advisory committee and every 17 agenda. She may also designate a substitute 18 in her absence. 19 Overall, I think we're going to 20 have a very good meeting again. This meeting 21 seems to be -- I mean, I looked at the 22 agenda. I was interested by seeing what's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 8 1 going to be on the plate. There seems that 2 there's a lot of room for discussion this 3 time around, and I hope that we will all feel 4 free to comment and elaborate on the points 5 that are brought forward to us. So just to 6 let you know, I think it's going to be good. 7 The first session this morning is 8 going to be by the EIA administrator, Guy 9 Caruso, who is going to tell us about the 10 general affairs at the EIA. 11 Welcome, Guy. 12 MR. CARUSO: Thank you, Nick. Good 13 morning. Thanks again, Nick. I always start 14 my remarks by saying how much we really 15 appreciate the ASA committee's work. And I 16 think it's -- you know, my -- this is I guess 17 about the eighth meeting for me, and every 18 time, I feel as though that EIA is a 19 significant beneficiary of this committee's 20 work and the advice you've given us. 21 And as you point out, Nick, the 22 agenda for this meeting is no exception in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 9 1 that it lays out a number of areas where we 2 can really seek the advice of this committee 3 and the membership and goes beyond just this 4 meeting. Clearly we're going to hear a lot 5 from your members in the sessions themselves, 6 but I also want to thank you and the members 7 for the continuing relationship we have with 8 you outside the formal meetings, and that we 9 continue to seek the advice of the committee 10 and the membership. So thanks very much. 11 Good news I think for bringing you 12 all up to date as to what's been happening at 13 EIA is that we're a lot less -- it's been a 14 much calmer six months this October than when 15 I stood here last October right after the 16 aftermath of Katrina and Rita, which so 17 dominated our own work and the Congress' and 18 the Administration's demands on EIA, and that 19 tended to overshadow a lot of the ongoing 20 core activities. 21 The last six months, we've I think 22 been able to get back to doing the things BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 10 1 that we need to do to follow up on many of 2 the suggestions that we've heard from you. 3 And you're going to hear 4 immediately after my remarks, and Nancy 5 brings you up to date with the activities, 6 where we are on probably the most exciting 7 new development in EIA on the forecasting 8 side in a couple decades. And that is we 9 have received the approval and support of DOE 10 management to do a complete redesign of the 11 National Energy Modeling System. 12 And of course, budgets being what 13 they are, this is highly dependent on our 14 getting the monies that we've asked for, but 15 we're optimistic. And I'll talk a little bit 16 about specifically on the budget, but I 17 mentioned this the last time, but we are six 18 months closer to actually implementing the 19 redesign. And Susan Holte will go into a lot 20 more detail about that. And I think it's 21 very much an activity which we can benefit 22 from, this committee and your membership, so BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 11 1 we're real excited about that. 2 We've been talking about the 3 reporting on greenhouse gases for a number of 4 meetings now, and the redesign of the 5 voluntary reporting survey is well underway 6 now and we're hoping to implement that in 7 '07. And as you know, it's been about a 8 two-year process, and again, benefiting from 9 your advice. 10 And then there is always the 11 ongoing energy markets and how they impact 12 what EIA does. And just a couple of 13 activities, a couple of events that highlight 14 the effect on our work, and that is 15 the -- we've been involved to some extent in 16 this whole debate about peak oil that's been 17 underway in the analytical community now for 18 several years. 19 And we at EIA, as I think you've 20 heard from John Wood and others of John 21 Wood's staff in Dallas, are of the view that 22 the resource base, whether it be oil or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 12 1 natural gas, is sufficient to meet the 2 demands that our forecasting models indicate 3 will be there, whether it be domestically or 4 internationally. 5 And of course, we follow new 6 developments. And the most recent one on the 7 upstream side has been this very deepwater 8 discovery and the lower tertiary trend in the 9 Gulf of Mexico, the so-called Jack 2 well 10 operated by Chevron. And our models have had 11 the reserves from that lower tertiary trend 12 in our long-term outlook for some time now. 13 So in a way, it kind of supports our view, 14 which doesn't necessarily mean it's 15 ultimately where it's going to come out. 16 And then going to the other extreme 17 of the spectrum on oil is the downstream. 18 Not only have we had this transition from 19 MTBE to ethanol, which is significant for EIA 20 because of the data quality issues with 21 respect to collecting and understanding 22 what's going on with blending of ethanol, and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 13 1 that we certainly did not have in our system 2 when EIA was founded -- so that's been an 3 issue; we'll talk about that -- but other 4 things like the lower sulfur gasoline, the 5 ultra-low-sulfur diesel, and then the 6 increased emphasis on flexible-fuel vehicles, 7 such as biodiesel, all have important 8 implications of the work of EIA and 9 particularly the data quality on, in this 10 case, oil. 11 I'll also mention where we are on 12 the FY 2007 budget, which is, unfortunately, 13 Congress went out without passing an '07 14 budget for energy and water, where our budget 15 resides. So we have to live with the 16 continuing resolution at least through 17 mid-November, but the smart money in 18 Congress -- insiders are saying it won't be 19 until the new Congress in January before they 20 act on an '07 budget, which is difficult for 21 an organization that has as its core function 22 collecting data, which uses about 50 percent BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 14 1 of the budget. And we can't stop doing that 2 or even diminish it, so it means other things 3 really get squeezed. 4 We talked about the external study 5 team that was headed by Denny Ellerman. We 6 have, since our last meeting, gotten the 7 formal report, and those reports are I think 8 available if you haven't gotten one. And 9 we'll talk a little bit more about the 10 recommendations they've made, some of which 11 we'd like your advice on about how to 12 implement that most effectively. 13 And we are beginning, believe it or 14 not, another five-year strategic plan. That 15 means that I've been here almost five years, 16 because I was here when we started the last 17 one. 18 You know, Susan's going to go into 19 a lot more detail about the NEMS system, the 20 redevelopment or redesign. And some of you 21 were around and remember when this current 22 version was started in 1991. And you think BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 15 1 about what's happened in the energy markets 2 since '91, there's been an enormous amount of 3 change, restructuring, deregulation. And as 4 I mentioned, in oil and gas, there were 5 virtually no biofuels in 1991. It didn't 6 exist in our database. 7 On the gas side, there's been an 8 enormous change. In 1991, coal bed methane 9 was less than 1 percent of our gas supply; 10 it's 10 percent today. Other unconventional 11 gases, if our model is correct, will 12 represent 40 percent of our gas by 2030. 13 So clearly, you try to keep up with 14 these changes by making adjustments in the 15 algorithms and add-ons, but oftentimes when 16 you do that in an existing large-scale 17 mathematical model, as many of you know much 18 better than I do, it creates a lot of 19 unintended consequences and you come out with 20 results that, quite frankly, don't make any 21 sense unless you go back and relook at the 22 whole system. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 16 1 And a few of us were at the energy 2 economists meeting in Detroit, and Bill Hogan 3 made an excellent luncheon talk. And he made 4 some of these points about how energy 5 modeling can be useful, but clearly, in his 6 case, he related it to electricity 7 deregulation, there are a number of things 8 that we need to do in that area to bring NEMS 9 up to date, and many others, as I've talked 10 about. 11 Where we are now is Susan Holte, 12 who will speak to you at 9:30, is heading up 13 our internal design and planning team for the 14 redesign of NEMS. And we've sent letters out 15 to key stakeholders who are interested in 16 NEMS, and asked for their input. And 17 obviously by virtue of this meeting, we're 18 asking for yours. And we think it's probably 19 about a three-year project. We're hoping 20 that if all goes well by maybe the 2010 21 Annual Energy Outlook, the new NEMS would be 22 up and running. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 17 1 Something that's very near and dear 2 to the heart of Jae on this committee, the 3 voluntary reporting on greenhouse gases is 4 underway and been redesigned. And I think if 5 all goes well, that will begin collecting 6 that data in '07. And thank you all for your 7 help in the work leading up to this. 8 I already mentioned the deepwater 9 discoveries. And I think these are events 10 that we have to respond to, whether it be 11 because we're asked to analyze what does it 12 mean for energy markets -- but in the case of 13 the deepwater discovery, it brings up another 14 point which is directly related to the 15 redesign of NEMS, and more specifically, to 16 forecasting in general, and that is 17 technological change. 18 Now we try to do our best in 19 capturing technological change in every 20 aspect of the industry, and we have utilized 21 independent expert groups to come in and say 22 are we doing the right thing in trying to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 18 1 capture what's going on in whatever sector it 2 happens to be. But I don't think anyone 3 could have anticipated the drilling in 12,000 4 feet of water and producing from the lower 5 tertiary trend in 1991, when NEMS was 6 designed. 7 So it's that kind of thing that 8 we'll be looking to all of the -- not only to 9 this committee, but to all of those I 10 mentioned that we've sent letters out to 11 asking for input how best to capture 12 technological change. I mean, we've been 13 using I think a reasonable system to do that 14 in terms of recent trends, what's going on in 15 each sector, but we'll certainly be looking 16 for outside advice on that as well as in the 17 downstream sector. 18 You know, we talked about the 19 Energy Policy Act of last year, and some of 20 the implications that has had for a number of 21 things. I mean, certainly, the one you hear 22 the most about is biofuels and things like BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 19 1 particularly ethanol and the subsidies and 2 incentives for renewables, and we certainly 3 will capture all of that in our Annual Energy 4 Outlook 2007. But many of these items that 5 were in the EPAct 2005 were authorization 6 language, but no appropriations. So indeed, 7 many of the things that we think are going to 8 occur, we can't really model without knowing 9 what the actual budgets and appropriations 10 will be. 11 But we are working with others 12 within the federal government, such as 13 Agriculture, to try to understand better 14 what's going on in the production of biofuels 15 and what it means for our projections. For 16 example, we don't collect on a regular basis 17 the ethanol production, but Agriculture has 18 some collections on their system, so we're 19 working with them on that. 20 And clearly we need to improve 21 that, since ethanol, we think, will represent 22 about 8 percent of total gasoline demand in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 20 1 our long-term forecast. So it's extremely 2 important that we get the -- we capture all 3 that blending that's going on outside of 4 refineries. Our systems were designed to 5 capture data at the refinery, at bulk 6 terminals, but we never anticipated -- you 7 know, I think there's about 2,500 blenders at 8 last count that are not in our -- that are 9 increasingly -- we're trying to capture that 10 data, but it's increasingly important and 11 necessary. 12 So clearly, our intention is to 13 improve both the oil and gas data quality in 14 both the areas of ethanol and biodiesel. We 15 are proposing in the '08 budget, which is now 16 at OMB, a system to collect the biodiesel 17 data, beginning with the '08 budget. 18 Bringing you up to date, as I 19 mentioned, Congress left town without having 20 voted on the energy and water budget, and our 21 request was about $90 million. The Senate 22 actually added $3 million to that in their BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 21 1 subcommittee. The House approved the $89.8, 2 so we're hoping the Senate prevails when they 3 come back and they get in conference 4 committee. 5 One of the issues that came up 6 during the budget process was that at that 7 roughly $90 million budget, we said that we 8 couldn't keep doing everything and keep our 9 quality up, as we've talked about before this 10 committee previously. 11 So we proposed the dropping of 12 three surveys, petroleum marketing surveys 13 mainly affecting the price or the pricing of 14 both the first purchases of crude oil in the 15 domestic market and imported prices for 16 crude, as well as electricity Form 767. 17 Interestingly enough, even though those three 18 forms are relatively small in terms of the 19 total budget, we estimate about $1 million to 20 do those three. 21 It kicked off kind of I think a 22 reasonably strong pushback from users of that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 22 1 data, including some within the federal 2 government, such as IRS, which uses the 3 pricing data; EPA, which objected to 4 the -- they use the data from the electricity 5 survey. So we got some pretty strong support 6 for keeping these surveys. And indeed, that 7 was one of the reasons the Senate added 8 $3 million in addition to it. The two 9 reasons the Senate gave the Senate Budget 10 Subcommittee on Energy and Water for adding 11 $3 million was to restore these three surveys 12 and to add collection data on ethanol and 13 biofuels. 14 So we're hopeful, again, that that 15 works out, because as you know, the trend in 16 the EIA's budget that we've talked about for 17 a number of meetings here has not been 18 favorable, but we hope this is the beginning 19 of the turnaround and the increase that we 20 hope that will come to be passed for the '07 21 budget. And we have reason for optimism for 22 '08 as well. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 23 1 We've talked about that external 2 study team, which -- the design of that team 3 actually was very much influenced by your 4 advice in that we model it after kind of a 5 visiting -- outside visitors kind of program, 6 as they do in many of your universities. 7 And Denny Ellerman, heading that up 8 from MIT, and then an excellent supporting 9 cast that we talked about previously, I think 10 it really worked really well, in our view, 11 because it forced us to pull together the 12 information they needed to do their 13 assessment or their study and kind of forced 14 us to do almost an internal review before we 15 could prepare for the external review. And 16 there are three main recommendations that 17 come out of the study. 18 And again, the first one is that we 19 should do more analysis. And they said that 20 it was obvious when you look at that, what 21 suffered in that long downward trend in real 22 prices was less analysis. We had to focus BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 24 1 obviously on the core data collection. And 2 up until recently, we haven't dropped any 3 surveys, and as I mentioned, we decided that 4 couldn't continue. We had to either drop 5 some things or the quality would continue to 6 decline. What they're saying is that there's 7 a linkage between the data collection and the 8 analysis. And if you kind of shortchange the 9 analysis, you actually don't keep up with 10 what data you need to keep the analytical 11 function current. So that was one 12 recommendation. 13 And then the IT recommendation was 14 in response to a trend in at least the 15 federal government that there be a 16 consolidation of IT functions within each 17 department under one CIO, chief information 18 officer. And they thought in EIA's case that 19 didn't really work well because we needed to 20 maintain that independence, and we certainly 21 agreed with that. 22 And then the one where I think we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 25 1 may be asking -- I think for sure we'll be 2 asking you advice in this: Making greater 3 use of the external market information. And 4 the team thought that one of the areas where 5 EIA may not have kept up to date as well as 6 we could have is taking advantage of the 7 information that's out there in the 8 marketplace and using that in our analysis. 9 And we think we have some 10 ideas -- and they gave us some on how we 11 might do that, but it's an area where we 12 certainly will seek your advice, among the 13 many other areas that we'll be talking to you 14 about these two days and in future meetings. 15 And finally, we are launching -- I 16 think in two weeks, we'll have our first 17 meeting to begin our next five-year strategic 18 plan. And again, we're trying to use this as 19 a way of, one, implementing some of the ideas 20 that have come out of advice from groups like 21 yours and from the external study committee, 22 as well as the recognition and need to keep BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 26 1 up with the change that is always underway 2 with respect to what EIA does. 3 So again, I want to close by 4 thanking you all for holding this meeting and 5 being willing to spend some time in advising 6 us. And we looking forward to the next day 7 and a half. Nick, again, thank you. And I'm 8 certainly available for questions now or over 9 the next two days. 10 MR. HENGARTNER: Any questions? 11 Jae? 12 MR. EDMONDS: Yes. Guy, I was 13 wondering if you'd take a couple minutes and 14 elaborate a little bit more on the Ellerman 15 committee findings. 16 I think the first and the last 17 intrigued me. The first was do more 18 analysis. And what does that mean? 19 And then the third one was use 20 external data sources more, and I'm not quite 21 sure what that means. So if you could 22 elaborate a little bit on what the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 27 1 implications for EIA are, I'm sure that Denny 2 put a few more pages than that on the topics. 3 MR. CARUSO: Yes, and I think the 4 full report's available, yes, to everyone 5 here. What they were reacting to is that 6 when they looked back on the last -- well, 7 especially the last 10 years, they saw that 8 the EIA used to publish a lot of standalone 9 analyses as either working papers or, up 10 front in many of our publications, there used 11 to be an opportunity for EIA staff to publish 12 analytical work on that particular -- like, 13 for example, in the Petroleum Supply Monthly, 14 you might have an article on deepwater 15 drilling or, on the other extreme, we'd have 16 things on the refinery sector, individual 17 analytical pieces that could either be part 18 of a statistical publication or a standalone. 19 And there had been a significant decline in 20 that. That's what they meant. 21 MR. EDMONDS: And these were issues 22 that EIA was taking on itself as opposed to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 28 1 Congress asking EIA do a special study on X? 2 MR. CARUSO: Yes, exactly. Yes, I 3 should have mentioned that, Jae. It was 4 self-initiated analytical work as opposed to 5 Sen. X saying please do a study on carbon 6 restrictions or on the NCEP. 7 The third recommendation about 8 using information that's out there in the 9 marketplace, they felt that maybe that's one 10 area where EIA has not kept up as well as we 11 might have. And that is because so much 12 information's available now out -- and they 13 were specifically -- and the examples that 14 came up during the study team were things 15 like futures and forwards markets options, 16 and whether or not we're capturing -- when we 17 do our short-term energy outlook, whether 18 we're really making use of that kind of 19 information in trying to project, in this 20 case, particularly oil, oil prices. 21 And we're saying, yes, it's a great 22 idea, but we're not sure how to do that. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 29 1 That's why I said we'll be looking for 2 advice, and what's the implications of hedge 3 fund activity and the commodity price index 4 funds. We all talk about it, but no one 5 really knows how to incorporate that into our 6 projections. 7 MR. CLEVELAND: Guy, just to follow 8 up on the first point, I feel like the 9 decline in discretionary analysis is a real 10 loss, I thought, for oil and gas in 11 particular. And what we only get now is 12 whatever's blowing in the current political 13 winds oftentimes. And whereas there used to 14 be -- in the oil and gas supply module, for 15 example, you'd often -- it was a way to get 16 into the guts of NEMS when someone working on 17 it would think about, well, how do you model 18 tautological change? And so there might be a 19 paper on that or what are the trend and find 20 development costs? That'd be a way of not 21 having to get into the code of NEMS, but to 22 start -- a way of users, at least some of us, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 30 1 thinking about some of the issues that we 2 struggle with in building these models. And 3 I think that's a significant loss, and I'm 4 sure it's true across every sector of NEMS. 5 MR. CARUSO: Well, I think Denny 6 and Paul Joskow, I think they're the two that 7 felt the most strongly about exactly what you 8 just said. And at least in my experience of 9 four years here is that it was -- we had a 10 choice between continuing to try to put the 11 resources on the core data collection and do 12 as much of the analysis as possible, and that 13 always was the discretionary analysis, to 14 follow up on Jae's point, that suffered. And 15 I agree. And then we certainly -- if we get 16 additional money, we'd certainly like to 17 spend more resources on that. 18 MR. HENGARTNER: Susan? 19 MS. SEREIKA: On the work that you 20 did here or the work that the external team 21 do, did any of this come through your request 22 for funds for next year? Like, for example, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 31 1 that first point. 2 MR. CARUSO: Well, yes, I think 3 if -- particularly for in '08, in the 4 budgetary request that we submitted, if 5 everything goes as requested, we would 6 definitely have more money for analysis in 7 addition to a redesign of NEMS, and the third 8 area where a lot of that additional funding 9 would be expended would be improving data 10 quality, particularly on oil, where as I 11 mentioned, things like biofuels and other 12 changes really were not captured in our 13 system. 14 MR. NEERCHAL: Coming from an 15 economic background, our department, for 16 example, is going to go through one of these 17 external reviews this year. 18 And it went through one of those 19 seven years ago. Usually the reports usually 20 sit on a shelf somewhere. So I was wondering 21 what is the next steps for this report and 22 who is the audience? Who is going to get it? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 32 1 And if we feel that we need to support some 2 of these points, how do we forward our 3 support to those points? 4 MR. CARUSO: There are two things 5 very specific. One is that we have, as I 6 said, at least a little -- some effect 7 already is in our request for the '08 budget. 8 Secondly, the strategic plan that we're 9 launching in the next couple weeks will use 10 some of the recommendations in there to guide 11 us in that. 12 But there are a couple of other 13 things that they recommended. One is there 14 were a couple of areas that they felt that 15 really weren't essential to EIA's mission. 16 One was the financial reporting system, and 17 they recommended that we drop that. I didn't 18 include that in these highlights, but that 19 has been a controversial program, for 20 example. And I think there have been three 21 attempts to drop that since it was in the 22 authorizing legislation of 1977, and all BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 33 1 three met stiff resistance from Congress. 2 And we thought that if we were to propose 3 that, let's say in our FY '08 budget, to drop 4 financial reporting of energy companies at a 5 time when we're not too far removed from 6 Enron and other activities, that it really 7 wasn't politically feasible. 8 And then the collection of the 9 renewable transportation -- 10 MS. KIRKENDALL: Alternative-fuel 11 vehicles -- 12 MR. CARUSO: Alternative fuel. 13 There was another form -- alternatively 14 fueled vehicle form. They said that that 15 really probably belonged in EPA or was not 16 really an EIA function, not in energy. But 17 again, at this time when so much emphasis on 18 biofuels and renewables that's probably not 19 in the cards. So we aren't necessarily 20 following up on -- implementing -- planning 21 to implement everything. 22 MR. SINGPURWALLA: Did they give BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 34 1 any tips on implementing the stuff or was it 2 just recommendations? 3 MR. CARUSO: It was mostly 4 recommendations. You know, I mean, we had 5 more in the offline conversations with Denny 6 and Paul and Jay. Jay Hakes, the former 7 administrator, of course, has been through 8 this and understands the difficulties, but 9 was extremely valuable to have him there. 10 And they all said that they would be happy to 11 continue discussing this. 12 MR. SINGPURWALLA: Do you think 13 it's primarily a manpower issue? I was 14 looking at the top point again specifically. 15 MR. CARUSO: That one is, yes. 16 Yes, I think it's a combination of having 17 enough people and then the right people. And 18 as we've talked about in this group 19 previously, there is a significant turnover 20 of people with -- about one-third of our 21 staff at the senior level turning over in 22 '06, '07, '08. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 35 1 MS. KHANNA: Well, yes, sort of 2 related to that, which is does this affect in 3 any way the hiring priorities? 4 MR. HENGARTNER: You stole my 5 question. 6 MS. KHANNA: I mean, if you talk 7 about analysis, you're talking about a 8 different kind of group as opposed to people 9 who are really good at collecting data or 10 collating data. 11 MR. CARUSO: I think so. I think 12 we would hope to be able to get more 13 analytical positions, particularly I think 14 where it's -- if you look at -- and I don't 15 want to -- since there's a lot of staff here 16 I don't want to single out, but it's fair to 17 say that the office that suffered the most is 18 the oil and gas office in terms of the 19 analytical -- the personnel available to do 20 analysis. I think they're about one-half of 21 where they were -- certainly from '96. So in 22 a 10-year period, they're about one-half. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 36 1 MR. HENGARTNER: Related to hiring, 2 the one thing I didn't see in your 3 presentation is actually something that came 4 up several times in the last six years is the 5 turnover, a lot of people retiring and 6 getting new people into EIA. And I was 7 curious to see if we made progress or, if 8 not, if that is something that one should 9 focus in the next five-year plan? 10 MR. CARUSO: Oh, definitely, that's 11 critically important. The workforce planning 12 is the buzzword, but we lost 16 senior people 13 in '05. We're probably looking at around 20 14 in '06; a similar number, maybe a little more 15 in '07. And we have been I think reasonably 16 successful in attracting high-quality 17 applicants in the new -- so we have this 18 bimodal demographic distribution developing 19 now in EIA. We actually have some people in 20 their twenties again, and then a large group 21 in the fifties and even in their early 22 sixties. And so what we want to do is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 37 1 continue to attract good young people with 2 analytical skills, obviously 3 statisticians -- that we've talked about in 4 this group about how difficult it is to get 5 math stat -- high quality, qualified 6 applicants. 7 And then to the extent that people 8 who are eligible for retirement would stay a 9 bit longer, that would help, too. So it's a 10 combination of making it -- retaining people 11 a bit longer and attracting young people, and 12 then the mentoring end activities of 13 transferring that knowledge to the new 14 people. 15 We had a very active intern program 16 this summer, the most active -- I think it's 17 got to be -- in EIA history. We had more 18 than 20 interns in the summer of '06, and we 19 think it was very successful. 20 How many of the interns would 21 ultimately come to work for EIA? Hard to 22 say, but it's useful in recruiting as well BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 38 1 because some of them go back to their 2 universities and say we had a good time 3 working there, they gave us some interesting 4 work. It's not a bad place, you know? All 5 those old people aren't so bad after all. 6 So yes, I definitely should have 7 mentioned that because that's one of our 8 great challenges. As I said, probably a 9 significant -- at least one-third turnover in 10 three years. Nancy? 11 MS. KIRKENDALL: You might talk 12 about we've got the big recruiting -- they're 13 doing a lot of work right now. We have a 14 contractor helping us to improve our 15 materials for recruiting on the web, because 16 it's awful. 17 MR. CARUSO: Yes. 18 MS. KIRKENDALL: It has been awful, 19 and they're trying to put together better 20 descriptions of the jobs we have. They're 21 trying to make it sound like it's more 22 interesting. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 39 1 MS. FORSYTH: Potential for 2 analysis. 3 MS. KIRKENDALL: So there is an 4 effort and I think they've made some 5 progress. At least the things I've seen look 6 pretty promising. So you might occasionally 7 check the web and see if you think it looks 8 any better. 9 MR. CARUSO: Yes, we have hired a 10 recruitment coordinator. When you think 11 about it, 10 years ago, EIA was in decline. 12 So we hadn't hired -- there was a long period 13 there where there was a hiring freeze. And 14 to go from that to someone who actually 15 that's what they do full-time is -- and it 16 came out of our strategic plan and it 17 certainly will be an important part of the 18 next five-year plan. 19 MS. KHANNA: Actually, follow-up to 20 Nancy. The one thing I noticed, occasionally 21 either from you or from Bill, we'll get an 22 e-mail saying, hey, we have a job available BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 40 1 and please advertise it to some of your 2 colleagues and students. But at least for 3 economists, I mean, if I'm looking for a job, 4 I go to the central marketplace for 5 economists, which is job openings for 6 economists. 7 If you are trying to recruit, 8 recruit at least economists, that would be 9 the place to create a link to, because people 10 don't go anywhere else, literally. I mean, 11 if you're looking for an academic or a 12 federal government-type job or even private 13 jobs with an analytical component. And it 14 could be I think really important to get 15 hooked up to that. 16 SPEAKER: Get the advertisement in 17 the right spot. 18 MS. KHANNA: It comes out five or 19 six times a year and it's really the central 20 marketplace. 21 MR. CARUSO: Thank you. 22 Susan? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 41 1 MS. SEREIKA: For the 2 statisticians, you should go to the joint 3 statistical meetings, because they have lots 4 of opportunities for recruitment there. 5 That's where more statisticians will go. 6 MR. CARUSO: I know we always have 7 at least several people attending those. 8 MS. SEREIKA: Do you? 9 MS. KIRKENDALL: We don't always 10 recruit them, so that's a good point. 11 MS. SEREIKA: Yes, a lot of 12 students go there for recruitment. And those 13 interns, where are they in terms of their 14 training? Are they toward the end of their 15 training for their degree or are they 16 mid-way? 17 MR. CARUSO: They range from 18 juniors in college to some that were in 19 graduate school, and at least two of them 20 were the JPSM, right, Nancy, from the 21 University of Maryland's Joint Program of 22 Statistical Method? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 42 1 MS. KIRKENDALL: Yes. 2 MS. SEREIKA: Because I would 3 imagine if you caught them in their last 4 year, that's probably ideal. Because they're 5 right at that point where they're starting to 6 think about where they want to go next. And 7 if you give them that wonderful opportunity 8 to do the internship, that they might -- you 9 know, might capture them. I know that like 10 in engineering, when they do their 11 cooperative experience, it's in that last 12 year and a lot of people transition right 13 into that job. 14 MR. NEERCHAL: I think the issue 15 here I think is that you are competing with 16 many other recruiters, also. 17 MS. SEREIKA: That's true. 18 MR. NEERCHAL: I think we sent one, 19 Katherine (?), and she had a wonderful time 20 and she gave a very good report, but she 21 wants to do her Ph.D. in Iowa, so that's -- 22 MS. KIRKENDALL: That's what BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 43 1 happens when you get good ones. 2 MR. NEERCHAL: Yes. So I think 3 this is about competition. So I think going 4 to JSM and putting a recruiting booth is a 5 good idea, even when you have one opening, 6 but you kind of make your presence felt there 7 because then you know the recruiters from 8 year to year to year, the EIA's (inaudible) 9 recruiter. 10 MR. CARUSO: Thanks. 11 MR. WEINIG: I might mention to the 12 committee that at 3:40 this afternoon, we do 13 have a discussion chaired by the deputy 14 administrator, Howard Gruenspecht, on the 15 external study team. It will not be -- it's 16 a pretty bulky report, but we are printing 17 copies of that so that you'll be able to see 18 it later today. 19 MR. CARUSO: Thanks, Bill. I 20 should have mentioned that in response to 21 Jae's question. There will be an opportunity 22 to really dig into that a bit more. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 44 1 MR. HENGARTNER: Okay, thank you 2 very much for your patience with us. 3 MR. CARUSO: Thank you, Nick. 4 Thank you, everyone. 5 MR. HENGARTNER: Next, I'd like to 6 invite Nancy Kirkendall as the director of 7 the EIA's Statistics and Methods Group to 8 tell us what is happening to the advice and 9 input we provided to her over the last year 10 or so. 11 MS. KIRKENDALL: Actually, what 12 we're going to do is give out our thank yous 13 for our departing committee members, and then 14 I'll go. 15 The first one is for Susan Sereika. 16 Susan has been a member of the committee for 17 the last three years, and we'd like to thank 18 you for your advice. 19 MS. SEREIKA: Well, thank you for 20 having me. 21 SPEAKER: And a couple of things, 22 Susan. The certificate and the letter from BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 45 1 Sec. Bodman. 2 MS. SEREIKA: Well, thank you very 3 much. 4 MS. KIRKENDALL: A certificate and 5 a letter from Sec. Bodman. Okay, our next 6 one is Mark Burton. Mark has been a member 7 of the committee for six years. 8 SPEAKER: Forever. 9 MS. KIRKENDALL: Forever, yes, it 10 seems like. We've seen him through two jobs. 11 SPEAKER: Nancy, how about giving 12 him a (inaudible)? 13 SPEAKER: Sell it fast. 14 SPEAKER: I'd rather take 60 bucks. 15 SPEAKER: You're becoming an 16 economist. 17 MS. KIRKENDALL: Finally, Nick 18 Hengartner, who has also been a member of the 19 committee for six years and has served ably 20 as our chair for the last two years, and has 21 kept us entertained. 22 MR. HENGARTNER: Thank you, Nancy. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 46 1 This is the end of the era. 2 SPEAKER: And the beginning of a 3 better one. 4 MR. HENGARTNER: Thank you so much. 5 SPEAKER: Right, now we can do some 6 analysis. 7 MS. KIRKENDALL: All right, this is 8 just a little summary of what's happened 9 since the last meeting in terms of some of 10 the suggestions you gave us. 11 We had a session, just a 12 wide-ranging discussion about how modeling 13 could suggest data needs. And the intention 14 on this was to try to get some advice 15 on -- since EIA sort of uniquely have data 16 users as well as our data providers in-house, 17 so we thought maybe we could get some ideas 18 on how to establish better communications 19 between those two groups. Because the 20 modelers sit over here and complain about the 21 data side, but we don't -- you know, and this 22 is an oversimplification. We do talk to each BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 47 1 other. It's just -- it could benefit from 2 being more formal. 3 But the discussion ended up being a 4 little bit more trying to bring in outside 5 ideas on data needs, which is probably 6 useful, too, because we haven't really done 7 enough of that. So some of the ideas that 8 came up were engaging the energy modeling 9 forum in some kind of a formal way, perhaps 10 special issues in the energy journal, and 11 then outreach efforts. 12 One of the things that senior staff 13 has actually talked about -- we've had our 14 NEMS conference every year for a number of 15 years trying to get input from users, and 16 that's worked pretty well. The 17 recommendation is, well, why don't we have 18 EIA Week or something, just expand that 19 conference to a couple days to bring in the 20 data side, too? In recent years, it's been 21 more than just a NEMS conference. They've 22 brought in STEO issues. So it has been BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 48 1 expanding, and I think that that's something 2 that senior staff seems to be interested in 3 doing, too. So I think you will see more on 4 the user conference idea as we go forward. 5 That'll show up in strategic planning, I'm 6 sure. 7 We had a session on the new 8 representation for the SAGE Model, and you 9 had some suggestions. You're going to hear a 10 talk by our contractor tomorrow at 8:30, so 11 you'll hear more about what we have learned 12 on that. This is all work-in-progress, so 13 you're going to be hearing things as we go 14 forward. It's one thing about giving you 15 information -- we try to pressure people into 16 talking to you before they're really ready 17 to, but at least that way we are more 18 interested in listening to advice. 19 And on the SAGE Model, the 20 committee commented the model was too complex 21 and we're still working on a more empirical 22 approach. And hopefully, that'll come back BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 49 1 at a future meeting. 2 The 2006 manufacturing energy 3 consumption survey, you had some good 4 suggestions. As you know, the Census Bureau 5 is our data collection agent on that, and we 6 work very carefully with them so that we're 7 working through considering inexpensive 8 alternatives for incentives. They're 9 examining non-response follow-up letters. 10 They're trying to improve things in general. 11 We had a session on respondent 12 cutoff dates for electricity data 13 collections, and a very good discussion. And 14 what that has turned into now is you're going 15 to be hearing about Electricity 2008. That's 16 the proposals for going forward with the 17 electricity surveys, and the piece of that 18 that relates to this session is that now 19 they're talking about doing a sample of the 20 smaller respondents. 21 The 860 and the 861 are Census 22 surveys. They've been run every year. And BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 50 1 so what they're talking about doing is taking 2 the smaller respondents and putting them on a 3 three-year cycle instead of having it be 4 annual. So in the course of three years, 5 you'd have a census, but you'd have -- every 6 year, you'd do a sample of about a third of 7 them. And so that's the idea that's related 8 to this cutoff date thing, to try to reduce 9 the burden on getting in late response, 10 non-response for all these little tiny 11 companies. 12 The IDC. We had a good discussion 13 on the IDC, the functional requirements 14 document. They tell me they are very close 15 to having it done. They've incorporated a 16 lot of recommendations. This has actually 17 been a really good effort in EIA. I think 18 everybody is serious about trying to make it 19 happen and moving to a common way of doing 20 Internet data collections. 21 We are going to be implementing the 22 greenhouse gas program in this new IDC BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 51 1 system. And the only little challenge we 2 have to work through is that we have legacy 3 Internet data collection systems, and we have 4 to figure out how do we bring them along and 5 move them into the new system. So there's 6 still some things we have to work through to 7 try to actually end up with a good, 8 consistent IDC technology. 9 MR. EDMONDS: Nancy, what is the 10 greenhouse gas program? 11 MS. KIRKENDALL: What is it? 12 MR. EDMONDS: Yes. 13 MS. KIRKENDALL: It's the voluntary 14 survey of greenhouse gases that they're going 15 to put in this IDC. 16 MR. EDMONDS: Okay. 17 MS. KIRKENDALL: And then finally, 18 this was a study, a co-integration analysis 19 between crude oil and natural gas prices, and 20 you recommended that we continue to do 21 similar analysis. We have published the work 22 you saw. The authors, Jose and Fred, do plan BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 52 1 to look at that trend term. They haven't 2 managed to do that yet. 3 However, that study did generate 4 further interest in EIA. Andy Kydes has had 5 an intern work on it over the summer. They 6 did an analysis of crude oil, natural gas, 7 and coal prices, and trying to figure out 8 what does the model tell you and what doesn't 9 it tell you and what can you learn from it. 10 So we're interested in the 11 technology and trying to figure out how to 12 use it. And so it's always slow trying to 13 figure out how to do something new. You have 14 to bring it in and use it and work through 15 all the details. 16 So that is what I have. Any 17 questions? 18 MS. KHANNA: This may be a question 19 that needs discussion separately, but I was 20 just curious why is the EIA collecting the 21 greenhouse gas data as opposed to, say, the 22 EPA? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 53 1 MS. KIRKENDALL: Well, that was 2 sort of what the external study team asked. 3 I don't really know how we got tapped with 4 it, but Congress decided that we should do 5 it. Maybe they trust us. 6 MR. CLEVELAND: Well, I mean, it's 7 not that unreasonable, I mean, given that 8 it's intimately tied to energy production and 9 consumption. 10 MS. KHANNA: But there's plenty of 11 stuff that the EIA -- the EPA does. I mean, 12 it's just at a time when budgets are 13 declining and we're having to turn off 14 service, which only the EIA can probably 15 provide. 16 MR. CLEVELAND: Right. 17 MS. KIRKENDALL: Well, the other 18 thing is it's a voluntary survey. It doesn't 19 represent anything. Whoever wants to send in 20 stuff can send it to you. 21 MS. KHANNA: Thank you. 22 MS. KIRKENDALL: You know, so what BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 54 1 does it really mean? You know, it's a way 2 for somebody to get on record of having done 3 something. 4 MS. KHANNA: Well, that's even more 5 reason the EPA should be doing it, because 6 they have a huge voluntary emissions database 7 which has variable quality. 8 SPEAKER: That was charitable. 9 MR. NEERCHAL: Yes, that's what I 10 was going to say, have you seen the data EPA 11 collected? I think that in fact, most of 12 EPA's data is self-reported I think one way 13 or the other. I think part of (inaudible) 14 compliance and things like that, I think they 15 really do not do service like you do. I 16 think that maybe -- 17 MS. KIRKENDALL: They may do some, 18 but, yes. 19 MR. NEERCHAL: But I want to just 20 ask and make a comment. You mentioned the 21 legacy system. One of the things I think 22 sometimes people don't talk about -- what BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 55 1 you're building right now is going to be 2 legacy some day. And I think that dealing 3 with a past legacy, many times you are 4 dealing with the paper information that you 5 are trying to put up, and I think it's like a 6 one-time something you have to deal with. 7 But it's even more important to really look 8 at the system you're building right now and 9 be aware that one day, the technology is 10 going to change, and make sure that it does 11 not become as much of a burden as the legacy 12 of today. 13 MS. KIRKENDALL: Right. 14 MR. NEERCHAL: I think 15 that's -- you know, it is kind of one of 16 those megatrend kind of an issue and I think 17 it's very important to keep in mind. 18 MR. FEDER: Nancy, I just wanted to 19 comment that even voluntary service can be 20 made representative by use of some weighting, 21 such as propensity scores and things like 22 that. People do that with web-based surveys BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 56 1 where you have -- I mean, which are also 2 voluntary. I know Harris Interactive is 3 doing it with their surveys. 4 MR. EDMONDS: Yes, but these data 5 are just public relations. Let's face it. 6 SPEAKER: That's right. 7 SPEAKER: They're biased. 8 SPEAKER: Which is probably why 9 it's Energy and not EPA. 10 MR. EDMONDS: I was there, I was 11 involved with the original creation of this 12 stuff and it is -- there's just -- I mean, it 13 has no content. 14 MR. FEDER: But Jae, do you think 15 that the greenhouse emissions survey can be 16 made representative using propensity scores? 17 If you post-stratify by industry type and 18 things like that, you can at least improve 19 the -- 20 MR. EDMONDS: See, it's not -- see, 21 it's about emissions reductions, not about 22 emissions. And so it's a delta based on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 57 1 heaven knows what. 2 MR. NEERCHAL: Estimates. 3 MR. EDMONDS: Right. And then how 4 does that fit into the larger accounting? I 5 mean, that's why I sense some frustration is 6 that it doesn't -- I mean, I can't think of 7 any way you could use it even if it were 8 totally accurate. 9 MS. KIRKENDALL: The only thing 10 that might be interesting, I understand many 11 electric companies report on it and we could 12 match to our electric power data, so you 13 might be able to do something interesting 14 with that, but other than the electric power 15 industry, I don't think that you have enough 16 to do anything with. 17 MR. FEDER: That's a weighting 18 technique which we commonly use in service by 19 using auxiliary data, so yes. 20 MR. HENGARTNER: Well, thank you 21 very much. It's nice to see that our advice 22 is taken seriously. Thank you, Nancy. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 58 1 I think we are -- unfortunately, 2 we're half an hour behind schedule. This 3 doesn't mean -- that's all right, I think 4 it's good to have those discussions. 5 I'd like to segue into our first 6 presentation, which is done by Susan Holte, 7 which is one of the technical advisors to the 8 EIA. And she's going to talk to us about 9 energy modeling systems. And I think this is 10 one of those issues that I hope will generate 11 discussion within the committee. And so, if 12 need be, we'll just stretch out this session 13 until we're all talked out. 14 MS. HOLTE: Does that mean I blow 15 off my 10:30 meeting? 16 MR. HENGARTNER: Oh, you have a 17 hard (inaudible). 18 MS. HOLTE: Well, that I can do, 19 but I think I do have to be out of here at 20 11:00. 21 MR. HENGARTNER: Okay, thank you 22 very much. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 59 1 MS. HOLTE: I'm sorry. This is 2 late. Anyway, thank you. Is this coming 3 through okay? Okay. 4 I've been asked to head a small 5 team of modelers and analysts who consider 6 the future development of the National Energy 7 Modeling System, and this effort has been 8 underway for about six months. We are 9 seeking to get input from interested 10 stakeholders on modeling and analytical 11 issues. Given that input, our own 12 assessments and those of the NEMS analysts, 13 we will prioritize (inaudible) development 14 projects for the next two fiscal year budget 15 cycles and beyond. And we may also initiate 16 some research projects on our own. 17 In addition to me, the team members 18 are: Bill Helkie, who joined EIA this year 19 from the Federal Reserve Board; Jeff Jones, 20 who's a lead modeler for electricity markets 21 in NEMS, and he has more than 25 years' 22 experience in EIA modeling and analyzing BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 60 1 electricity markets; Andy Kydes is the senior 2 technical advisor for the Office of 3 Integrated Analysis and Forecasting, and he 4 has nearly 30 years of experience in energy 5 analysis and modeling, with some private 6 sector experience in natural gas markets; and 7 Dan Skelly has worked with EIA both as a 8 contractor and as a federal employee for more 9 than 25 years in a number of areas, and more 10 recently has done work with the industrial 11 and transportation modules of NEMS, and he's 12 responsible for the overall system 13 configuration of the model as well as the 14 greenhouse gas emissions and analysis. 15 Personally, I've worked on the 16 design and development of all the major 17 integrated models in EIA and its predecessor 18 organizations going back to Project 19 Independence, which has everybody under 35 20 saying why are you still here? Referring to 21 the demographic issues. 22 Anyway, NEMS was developed in the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 61 1 early 1990s, and the call for NEMS arose from 2 the National Energy Strategy, which was 3 released in 1991. The model that EIA used at 4 the time was largely econometric and was 5 deemed inadequate to analyze many potential 6 energy policies. 7 As a result, there was a lot of 8 publicity about a new, biggest and best 9 energy model that EIA was going to build with 10 public statements by the Secretary and the 11 Deputy Secretary. The Office of Policy, 12 Planning, and Analysis in DOE commissioned a 13 National Academy of Sciences study to provide 14 recommendations on a model. And there was 15 some activity by the Secretary of Energy 16 Advisory Board. 17 Finally, the administrator, Calvin 18 Kent, decided to create a NEMS project office 19 in late 1990. And this consisted of the 20 then-technical assistant, Bill Skinner; John 21 Conte, who is now head of the Office of 22 Integrated Analysis and Forecasting; me under BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 62 1 a different name; and John Holte, if you get 2 the connection there. 3 The mission of the project office 4 was to research models and methodologies and 5 plan the design of NEMS. And the mission 6 also included representing the project in the 7 National Academy and other organizations, and 8 make some budget and organizational 9 recommendations. 10 Then, EIA created the Office of 11 Integrated Analysis and Forecasting in 12 October of 1991, with a mission of developing 13 NEMS, producing the Annual Energy Outlook and 14 the International Energy Outlook, but leaving 15 the short-term energy outlook out of that 16 office's mission. OIAF developed various 17 design papers, including very detailed 18 component design reports on each of the 19 modules and submodules of NEMS, which were 20 circulated for review, both in and outside of 21 the department. 22 And OIAF also started working group BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 63 1 meetings in each subject area so there would 2 be periodic interaction with interested 3 people in the department, the national 4 laboratories, and some outside organizations, 5 and this practice continues today. 6 We're undertaking the project a bit 7 differently this time, for several reasons. 8 First, we have an office in EIA with the 9 central responsibility for midterm energy 10 modeling and analysis, and that was not the 11 case 15 years ago. We didn't have a 12 centralized organization. 13 Second, we believe that EIA has a 14 functional model for many analytical 15 requirements, although more functionality is 16 required. 17 Finally, it seemed unnecessary and 18 impractical to start from scratch. In an era 19 of tight budgets and many requirements upon 20 EIA, the cost would be very high. So we do 21 not have a separate formal project officer at 22 this time. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 64 1 Much has been made of NEMS being 15 2 years old, but the model has been extensively 3 developed and adapted to maintain relevancy 4 beyond just the simple updates and periodic 5 extensions of the forecast horizon. As 6 background information, we asked the NEMS 7 analysts for a list of the model changes that 8 have been made since the first incarnation of 9 the model, and received a very long list of 10 projects. 11 And I've listed a few here: 12 Restructuring of the electricity and natural 13 gas markets, mercury emissions, greenhouse 14 gas, the ability to put limits or prices on 15 greenhouse gases, technology learning, 16 additional standards and regulations, more 17 technological detail, integration of the 18 entire macroeconomic model within NEMS. So 19 these are just a few, but the model has 20 certainly been extensively adapted over the 21 15-year period. 22 At this time, the team has had BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 65 1 meetings with interested parties in the 2 department to ask for their comments and 3 suggestions on NEMS. I should note that many 4 people in the department are far more 5 interested in NEMS now than they were 15 6 years ago. 7 And this is largely due to the 8 Government Performance and Results Act of 9 1993. As part of their budget submissions, 10 many department programs are required to show 11 the expected outcome of their programs on 12 energy markets by analyzing the program 13 impacts through NEMS. So there's a very good 14 reason for them to be interested. 15 Inquiries were also sent to 16 relevant congressional staffers. Not 17 surprisingly, we have received no comments. 18 They use our analysis and our data, but would 19 not typically be interested in modeling down 20 at this level. Guy Caruso's also sent 21 letters to a variety of energy trade and 22 research organizations, and inquiries are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 66 1 going out about now to some of the national 2 laboratories. 3 We have also asked for suggestions 4 from the modelers within OIAF looking for 5 projects that may be beyond their ongoing 6 model development. So interest does vary 7 widely, and suggestions do vary widely 8 throughout the department. 9 After meeting with us, the Office 10 of Energy Efficiency and Renewable Energy 11 provided a 15-page paper of suggestions 12 ranging from outreach and communication to 13 broad topics of consumer behavior modeling 14 and technological change to some very 15 module-specific recommendations. 16 Fossil Energy was also very 17 interested with a list of recommendations, 18 including improved representation of all 19 alternative liquid fuels, improve our 20 representation of oil and natural gas 21 production technologies and distributed 22 generation, and transportation advances in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 67 1 vehicle designs. 2 The Office of Electricity Delivery 3 and Energy Reliability has a more narrow 4 focus. Their issues are basically related to 5 the difficulty that they have in representing 6 their programs within NEMS, such as 7 transmission reliability, load shifting, and 8 storage. Some of their concerns may be 9 nearly impossible to address in the absence 10 of a transmission model. Many of their 11 metrics for their benefits analysis concern 12 congestion pricing and the cost of electric 13 outages. Nuclear Energy is primarily 14 interested in expanding the number of nuclear 15 technologies represented and the uranium fuel 16 cycle from supply to reprocessing and 17 storage. 18 Policy really didn't provide us 19 with much feedback. As a somewhat different 20 suggestion, they appeared more interested in 21 some scenarios in the three-to-six time frame 22 for which NEMS is really not suited. And the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 68 1 Office of Science is not interested in 2 weighing in, but wanted more information on 3 NEMS for some new staffers. 4 I'm not going to really dwell on 5 this slide, but just point out that the 6 combined vision for NEMS is always rather 7 lofty: Do everything, have all sorts of 8 diagnostics and uncertainty and everything, 9 and, oh, by the way, solve it in a wink of an 10 eye. But working from that lofty vision, 11 there are some overarching topics that we 12 must consider. 13 EERE and Fossil Energy are very 14 interested in extending the model horizon. 15 Even if we have no desire to extend the 16 projections in the Annual Energy Outlook, we 17 could -- and I repeat that is a big 18 "could" -- consider doing this to service the 19 needs of the department. Currently, a 20 contractor does that for them for their 21 benefits analysis. 22 Extending the model horizon raises BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 69 1 many issues. Obviously, one that always 2 comes to people's minds first is resource 3 depletion, but there's also the question of 4 how to represent technologies in such an 5 extended time frame. Generic technologies, 6 which is always sort of a fallback in 7 longer-term models, generic technologies may 8 not be very useful for the department 9 programs because they have to analyze very 10 specific programs. There's also a question 11 of how relevant a lot of the detail in NEMS 12 becomes as you stretch the forecast horizon 13 further out. 14 There were several issues raised by 15 people within the department about improving 16 the feedback between the energy and 17 macroeconomic models. There were a wide 18 range of infrastructure issues that are of 19 interest to the department, including 20 transmission, pipelines, G terminals, and 21 water use, which was of particular interest 22 to Fossil Energy. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 70 1 Technological choice and 2 technological improvement are topics of 3 continuing interest to us and to the 4 department, and how consumers choose 5 technology and the related topic of foresight 6 are paramount. 7 In addition to these broad issues, 8 there are some very specific model 9 improvements that are potentially of interest 10 to us and others. EERE is sponsoring the 11 development of a hydrogen market module for 12 NEMS. EIA is part of that process, so that 13 it is done within our modeling structure. 14 The process for adopting that model into our 15 baseline version of NEMS has yet to be 16 determined. 17 We would like to consider improving 18 representation of world energy markets, one 19 goal being to represent the competition among 20 energy sources and energy uses outside the 21 United States to better improve our domestic 22 projections. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 71 1 Not to pick on any one area, but 2 offshore wind is another technology we're not 3 currently representing in NEMS. There are 4 sure to be many others, such as gas hydrates. 5 I think there is a serious question 6 about whether we can capture some of the 7 important points about electricity 8 transmission, reliability, and congestion 9 without incorporating a transmission model. 10 Also, the solution time of NEMS has 11 become a problem. This is actively being 12 addressed through some algorithm changes. 13 There were other topics related to 14 NEMS and NEMS analyses that we do not view as 15 strictly modeling issues. In our meeting, 16 several people raised the issue of 17 transparency. This can run the gamut from 18 diagnostic tools to documentation in order to 19 understand the model, and how changes in 20 assumptions filters through to the results. 21 However, when pressed for further 22 information, nobody in the department came BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 72 1 back with any really specific ideas. 2 Our discussions with both EERE and 3 Fossil Energy would indicate that some real 4 investment on their part in learning the 5 model is really necessary. 6 NEMS relies on information about 7 technology characteristics from external 8 experts in various areas. An issue raised by 9 several people concerns the equitable 10 treatment of technologies, the so-called 11 level playing field. There is a real 12 question about whether all the experts are 13 evaluating the technologies with the same 14 level of optimism, and that technology 15 evaluation may be a tradeoff between 16 consistency and expertise. 17 So I guess that leaves a question 18 about whether there are guidelines that can 19 be offered in order to solicit this input 20 that's very necessary for the modeling. 21 There are various policies and 22 programs that cannot be treated now in NEMS. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 73 1 Local policies and deployment programs are 2 two examples. 3 Further outreach was also 4 recommended as a two-way street: Us 5 providing more information about the model 6 and the results, and us contacting more 7 experts to solicit input. Similar to many 8 recommendations, this is subject to available 9 resources, money, and sometimes even more 10 scarce, time. 11 So next steps? We will have 12 further discussions with stakeholders, and 13 the team will be looking at all the proposed 14 projects, reviewing and prioritizing them, by 15 very early next spring. 16 There are several major design 17 projects underway within OIAF. The oil and 18 gas supply module has been redesigned, and 19 the design addresses some of Fossil Energy's 20 recommendations. The peer review of that is 21 scheduled for early December. 22 A redesign of the petroleum market BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 74 1 module, that's our refining portion, has been 2 initiated. And there's some talk of doing 3 similar for the natural gas transmission and 4 distribution module. 5 A workshop on consumer 6 decision-making is in the planning stages. 7 Additional module reviews may be conducted. 8 And addressing a solution time, we've 9 initiated testing of another optimization 10 package. 11 Thank you. 12 MR. HENGARTNER: Well, Cutler? 13 MR. CLEVELAND: Thanks, Susan. 14 It's an enormous, daunting task given the 15 complexity of the model and the way it's 16 been. It's like the air conditioning system 17 in my building. It's been renovated and 18 re-renovated about 15 times the last 20 19 years. And every time they do it, some new 20 contractor comes in and puts in a new switch 21 or a blower or something. And when you call 22 when it doesn't work right, which is pretty BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 75 1 much all the time -- that's not true with 2 NEMS -- but when it doesn't work right, you 3 call the VU (?) staff that come in and no one 4 knows how the whole system works; they know 5 their little blower part. And I know it's a 6 daunting task, but I'm glad you're doing it. 7 I think the transparency, from the 8 user's side, the transparency in 9 documentation is I think the really important 10 issue. It's really hard to find out a lot 11 about the details of most models. You either 12 have kind of the report for the public that 13 you all produce and then some PDF buried 14 somewhere written in contractorese. 15 And there has to be some way to 16 present to users in some kind of 17 understandable, digestible way a little bit 18 more of the details than what you see in the 19 annual outlook or whatever the publication 20 is. And I think that would go a long way to 21 improving the transparency issue, which I 22 know always dogs you. And the users can help BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 76 1 do that because we know what we want and what 2 would help them. 3 One of the things that I didn't see 4 up there in terms of some of the big issues 5 was the connection to international markets. 6 And I'm wondering -- was it up there? 7 MS. HOLTE: No, I did mention that, 8 yes. 9 MR. CLEVELAND: Okay. I must have 10 been writing or not paying attention. 11 MS. HOLTE: Yes, that is really a 12 big issue. I'd like to enhance that. Right 13 now, the various import-export markets are 14 handled in a very -- you know, sort of 15 independently by fuel -- coal exports, 16 natural gas imports, oil, you know? But as 17 you go out -- especially as you go out 18 further in the time horizon, we anticipate 19 more need to consider the competition among 20 those fuels out there as it relates to it. 21 MR. CLEVELAND: Who wants to go 22 past 2050? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 77 1 MS. HOLTE: What? 2 MR. CLEVELAND: Who's pushing for 3 the really long forecast? 4 MS. HOLTE: Well, right now, NEMS 5 goes to 2030. 6 MR. CLEVELAND: Right. 7 MS. HOLTE: So even 2050 is an 8 extension. 9 MR. CLEVELAND: But who wants it? 10 MS. HOLTE: Basically EERE and 11 Fossil Energy. They have -- some of their 12 programs, they view as -- well, definitely in 13 2030, just sort of on the edge or even beyond 14 2030. So when they're asked to do the 15 program benefits, they're sort of out of 16 luck. The contractor, as I said, does extend 17 the model, extend NEMS to 2050 in order to do 18 that analysis for them. 19 MR. CLEVELAND: Two other quick 20 points. One, you mentioned a lot and I just 21 wanted to reiterate how important I think it 22 is, modeling technological change. I think BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 78 1 you quite clearly understand that. I think 2 it's represented in a pretty ad hoc way 3 across many of the different NEMS models. 4 And I think there's a large 5 literature out there in the economics world. 6 In fact, this month's issue of Energy Policy 7 has a whole special issue related to how do 8 you represent technological change in carbon 9 abatement. And then you have the whole 10 engineering literature of learning curves and 11 experience curves. And it would be good if 12 you guys could somehow address this issue in 13 a more systemic and consistent way, not 14 necessarily that every module is going to be 15 represented the same, but it's pretty ad hoc. 16 And it would be good -- it would 17 give users more confidence if the 18 rationale/discussion of how technological 19 change is represented, for example, in oil 20 and gas finding costs, if that was connected 21 to this ongoing debate in the literature 22 amongst economists and engineers and between BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 79 1 the two groups on how best to do this. It's 2 not easy, I know, but I think you could do 3 better in that area. 4 And the other issue is statistical 5 methods used in some of these things. I 6 think, again, you need to make sure, as I 7 think you try to do with varying degrees of 8 success, stay up to speed with what's 9 happening in terms of the latest statistical 10 techniques, particularly when you're dealing 11 with forecasting issues, time series data, 12 just to make sure that you're up to speed on 13 what's happening out there in the forefront 14 of forecasting. 15 MS. HOLTE: Did you want to weigh 16 in, Andy, on technological change or not? 17 I'm not trying to put you on the spot. 18 MR. KYDES: Where we have 19 technological -- 20 MR. HENGARTNER: Sir, could you go 21 to the microphone, please? Thank you. 22 MS. HOLTE: We do have several of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 80 1 the NEMS analysts here in the audience if 2 they want to weigh in. 3 MR. KYDES: Can you hear me? 4 MS. HOLTE: Yes. 5 MR. KYDES: Yes. The model 6 actually is not a single framework. We try 7 and build the model in each area -- the 8 decision-making -- based on what we think how 9 the market responds, and also all the 10 available data that's available to us. In 11 the electricity markets, they're lumpy 12 investments. They're easy to identify. 13 There's a collection that we can do. 14 And I've been doing technology 15 learning since I guess about 20 years now, so 16 I'm aware of the literature you're talking 17 about. And it is the thing that we 18 incorporate within a market where we can 19 identify significant investments of that 20 kind, lumpy and large. 21 In the area of, say, buildings, the 22 technologies are a little more muddy. There BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 81 1 are about 90 million, for example, 2 heaters -- space heaters, I mean, gas-fired 3 heaters -- and so the technology learning 4 curve in terms of learning by doing in some 5 of those things, it really kind of flattens 6 out. What you're really seeing is the 7 incorporation of new features: Dual fans, 8 dual speed, that kind of stuff. And that's 9 what's really playing, and so we do try and 10 incorporate that where it is a new 11 technology. 12 Now, in the area you're interested 13 in, I know oil and gas, the problem there is 14 that we don't have explicit technologies 15 there represented. And so in that area, we 16 haven't figured out how to incorporate tech 17 learning except as changes to the parameters 18 that relate to finding rates, for example, 19 cost per unit, and so on. So we're still 20 studying it, but if you have ideas, we'd like 21 to hear them. 22 MR. CLEVELAND: Well, I guess BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 82 1 there's this very interesting debate, you 2 know. You use learning curves, for example, 3 with these new technologies. And of course, 4 the engineers love learning curves, but the 5 economists have a very different way of 6 looking at it and would suggest different 7 ways of trying to model technological change. 8 So it might be worth kind of 9 engaging some of those folks and talking 10 about how best to represent technological 11 change. I'm not necessarily endorsing how 12 they do it, but for example, if you look at 13 this latest issue of Energy Policy, which are 14 mostly economists, they would approach how do 15 we forecast the impacts of new technologies, 16 like wind or something, on carbon abatement 17 from a very different perspective than the 18 learning curve perspective. Just something 19 to be aware of. 20 MR. EDMONDS: Is this the Michael 21 Grubb special issue? 22 MR. CLEVELAND: I don't know if it BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 83 1 is or not. I'm not sure. 2 MR. EDMONDS: I reviewed some of 3 those papers and they're scary. Yes, what's 4 in some of those models is just scary stuff. 5 And I think Andy's right in the 6 sense that it does require sort of a 7 fairly -- you really need to get some 8 perspective on technology, because learning 9 is a real phenomena. I mean, it's been 10 reproduced in study after study. But there's 11 a really nice paper, and I don't know if 12 you've seen it or not, but it's by Clark (?) 13 and Wye (?), which kind of goes through how 14 the models do -- and then Clark has a new 15 paper coming out where he goes through all 16 the different approaches and then goes 17 through the economic literature on it. 18 And what it boils down to is that 19 it's a really tough problem. I mean, 20 particularly as you're trying to take on the 21 whole energy system, as EIA is, because 22 technological change comes in through so many BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 84 1 different pathways. Particularly as you're 2 starting to go out toward 2050, what you find 3 is that the spillover effects become 4 increasingly dominant; that is, if you sort 5 of look at some of the big things that have 6 happened in energy. You look at, for 7 example, in production of oil, this 8 combination of 3-D seismic imaging with 9 directional drilling, and they've really 10 expanded that. 11 And sort of it you trace that back 12 and that comes through medicine where you've 13 got these CAT scans and then that came over 14 into the production side, and that, in turn, 15 was made possible all this computational 16 science. 17 And so you get all of this very 18 complicated set of interactions that are 19 going on. And that's going to be really 20 tough to model explicitly, and you may -- you 21 know, I wouldn't try to package 22 everything -- the vogue is to package BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 85 1 everything up into a learning curve and to 2 say that this relationship between a 3 declining price and cumulative deployment is 4 the whole story, and that the causality was 5 cumulative deployment led to the lower price. 6 You know, there's another hypothesis that 7 lower prices lead to greater deployment. 8 It's probably a much more complicated story, 9 and I think it is a much more complicated 10 story. 11 So I guess my advice is, yes, and I 12 think you probably are on top of the 13 literature, but that literature is 14 sophisticated. And probably, in the final 15 analysis, going through and just thinking 16 through the what-ifs may be as important a 17 contribution as you can make. 18 Trying to mechanize it so that you 19 actually are going to predict technological 20 change, I don't know that I would necessarily 21 recommend that you go fully down that path. 22 You know, it's just my two cents. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 86 1 MR. KYDES: Just two points. I'm 2 essentially confirming your point. The 3 spillover effects, I mean, who would have 4 thought, for example, that the PC, it's 5 accelerated development in the '80s and '90s 6 would lead to basically such great abilities 7 to do analysis of data that you previously 8 had to spend enormous amounts of money for? 9 MR. EDMONDS: Sure. 10 MR. KYDES: So there's no effect 11 that maybe might have been hard to anticipate 12 in advance, and just a point I think that 13 confirms your point. 14 The other area that learning -- and 15 we've looked at, is learning through R&D, 16 which the Europeans are heavily into. The 17 fundamental problem as far as I can tell is 18 that you kind of have to assume the parameter 19 or the success of what are the parameters are 20 to get significance on the other. 21 So in order to get -- if you put 22 them both in the experience portion and then BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 87 1 the R&D portion simultaneously, you don't get 2 a good fit. But if you, on the other hand, 3 sort of fix the learning through experience 4 of cumulative builds, and then estimate the 5 R&D component, you got a significant 6 (inaudible). 7 MS. HOLTE: Let me just comment on 8 one thing Andy said very early on about the 9 different parts of the model being handled a 10 bit differently in terms of technology. One 11 of the suggestions that we received, I 12 believe from EERE, had to do with 13 standardizing consumer behavior across the 14 model. Well, we think that's exactly the 15 wrong thing to do. We don't think that the 16 residential homeowner uses the same process 17 in making some decision about energy-using 18 equipment as does the person who's making it 19 for a big hospital complex as does an 20 electric utility. So we think that 21 standardizing in that way is exactly the 22 wrong thing to do. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 88 1 MR. HENGARTNER: Jae? 2 MR. EDMONDS: Let me also just 3 reiterate what Cutler said. I think it's 4 really great that you're doing the sort of 5 stepping back and thinking about what is it 6 you're doing and where you're going and just 7 how does this all fit together. I really 8 think that's a wonderful thing that you're 9 doing. 10 I think in doing that, one of the 11 things that I didn't hear, and presumably 12 it's because you don't have enough time to go 13 through all this stuff, but what you 14 obviously want to do is go back and ask 15 yourself the questions of what is it that I'm 16 trying to do here. Who are my customers? 17 What are the questions that I really need to 18 answer? And what is it that this model is 19 really going to have to do? 20 Because I'm sure that you're going 21 to get requests from congressional districts, 22 I would like to know what in 2043 the gas BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382