UNITED STATES DEPARTMENT OF ENERGY MEETING OF THE AMERICAN STATISTICAL ASSOCIATION (ASA) COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION (EIA) Washington, D.C. Thursday, October 28, 2004 2 1 PARTICIPANTS: 2 F. JAY BREIDT 3 NICOLAS HENGARTNER 4 JOHNNY BLAIR 5 MARK BURTON 6 MOSHE FEDER 7 BARBARA FORSYTH 8 NEHA KHANA 9 NAGARAJ K. NEERCHAL 10 SUSAN M. SEREIKA 11 RANDY R. SITTER 12 HOWARD BRADSHER-FREDRICK 13 ROBERT RUTCHIK 14 NANCY KIRKENDALL 15 PRESTON McDOWNEY 16 GUY CARUSO 17 TOM BROENE 18 HENRY S. BROOKS 19 BRENDA COX 20 GRACE SUTHERLAND 21 SHAWNA WAUGH 22 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 3 1 PARTICIPANTS (CONT'D): 2 TOM LORENZ 3 PHILLIP TSENG 4 JOHN WOOD 5 HOWARD GRUENSPECHT 6 WILLIAM WEINIG 7 INDUJIT KUNDRA 8 JOE SEDRANSK 9 KAREN NORMAN 10 11 12 * * * * * 13 14 15 16 17 18 19 20 21 22 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 4 1 C O N T E N T S 2 AGENDA: PAGE 3 Greetings, Presentations and 5 Remarks 4 Updates for the Committee 34 5 Proposal for Analysis of Collected 62 6 Data 7 A Customer Evaluation of the Short 83 Term Energy Outlook (STEO) 8 Frames: Progress on and Lessons 126 9 Learned from Frames Evaluations 10 The EIA Short Term Regional Electricity 134 Model: Capabilities and Data 11 Requirements 12 Natural Gas Production, Frames, 219 Samples and Estimation 13 Methods for Assessing NEMS Solution 227 14 Data for Interpretive and Diagnostic Purposes 15 Introduction to Program Assessment 238 16 Rating Tool (PART) Program Evaluation 17 External Evaluations of Survey Programs 245 18 External Evaluations of Forecasting 306 and Models 19 How the ASA Energy Committee Might Help 317 20 in Program Evaluation under PART? 21 22 * * * * * BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 5 1 P R O C E E D I N G S 2 (8:30 a.m.) 3 CHAIRMAN BREIDT: Okay, I think 4 we're ready to begin. Welcome to the fall 5 meeting of the American Statistical 6 Association Committee on energy statistics 7 with the Energy Information Administration. 8 This meeting is being held under the 9 provision of the Federal Advisory Committee 10 Act. This is an ASA not an EIA committee, 11 which periodically provides advice to EIA. 12 The meeting is open to the public and public 13 comments are welcome. Time will be set aside 14 for comments at the end of each morning and 15 the afternoon sessions. Written comments are 16 welcome and may be sent to either ASA or EIA. 17 All attendees including guests and 18 EIA employees should sign the register in the 19 hall and should include their e-mail 20 addresses. Restrooms are at the end of the 21 hall towards the back of this room, back that 22 way, a fountain is in the same corridor on BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 6 1 the way, telephones in this room share a 2 single number. It is, if you're interested 3 202-586-3071. 4 Kathleen Wert with ASA meetings 5 department is here. She was out at the front 6 desk and is available for committee member 7 questions on expense reimbursements. Tara 8 Stull was the former committee meetings 9 liaison, and you may know that she has moved 10 to a new job in Boston where she'll be 11 working for the Boston Museum of Natural 12 History. And if you were up late last night 13 you may have seen pictures of her revelling. 14 So in commenting each participant is asked to 15 speak toward a microphone, we have a 16 transcriber to appreciate that and committee 17 members and speakers at the head table need 18 to speak clearly and into a microphone, Bill 19 always writes that for me, "clearly and into 20 a microphone". 21 These microphones are reasonably 22 sensitive so you may not need to really lean BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 7 1 in, they should be able to pick it up. And 2 speakers are asked to use a microphone, 3 which is available here, and Bill can help 4 out with that. So now I'd like each of us to 5 introduce ourselves. So, we'll begin with 6 the committee members and then we'll go out 7 into the audience. Please speak into a 8 microphone and give your name and your 9 affiliation. So my name is Jay Breidt, I am 10 with Colorado State University Department of 11 Physics and can we start here. 12 MR. McDOWNEY: Preston Mc Downey, 13 EIR, SMG. 14 DR. NEERCHAL: Nagaraj Neerchal, 15 UMBC, Math department. 16 MS. KHANNA: Neha Khanna, State 17 University of New York, Birghamton, Economics 18 and Environmental Studies. 19 MS. Forsyth: Burhara Forsyth from 20 Westate. 21 DR. FEDER: Meshe Feder, Research 22 Triangle Institute, Research and statistics BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 8 1 division. 2 DR. BURTON: Mark Burton, 3 University of Tennessee, Knoxville center for 4 Transportation Research. 5 DR. HENGARTNER: Nick Hengartner, 6 Los Alamo, National Laboratory Statistical 7 Science Group. 8 MR. CARUSO: Guy Caruso, 9 Administrator, Energy Information 10 Administration. 11 MS. KIRKENDALL: Nancy Kirkendall, 12 Statistics and Methods Group. 13 CHAIRMAN BREIDT: Okay we'll go 14 back to the audience and can we use the 15 microphone please? Yeah I'm pointing at you 16 -- yeah or maybe Bill, you want to do this. 17 MR. WEINIG: This'll be fine. 18 CHAIRMAN BREIDT: Ok I'm sorry. 19 MR. WEINIG: One last member of the 20 EIA, I'm Bill Weinig with the Energy 21 Information Administration. 22 MR. TSENG: I'm Phillip Tseng, EIA. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 9 1 MS. HOJJATI: Behjat Hojjati, 2 Energy Information Administration. 3 MS. WAUGH: Shawna Waugh, 4 Statistics and Methods Group. 5 MR. HOUGH: Richard Hough, US 6 Census Bureau. 7 MS. HAITOT: Vicki Haitot, US 8 Census Bureau. 9 MS. SNAPP: Eva Snapp, US Census 10 Bureau. 11 MS. LOFTIN: Lacey Loftin, US 12 Census Bureau. 13 MR. BOURNAZIAN: Jake Bournazian, 14 EIA. 15 MR. HSEN: Paul Hsen, Census 16 Bureau. 17 MR. LORENZ: Tom Lorenz, EIA. 18 MS. BLESSING: Colleen Blessing, 19 EIA. 20 MS. MILLER: Renee Miller, EIA. 21 MR. SEDRANSK: Joe Sedransk, CWRU 22 and EIA. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 10 1 MR. RUTCHIK: Bob Rutchik, EIA. 2 MR. WOOD: John Wood, EIA. 3 MR. WEINIG: Just pass the 4 microphone over. 5 MS. JENNINGS: Alethia Jennings, 6 EIA. 7 MR. FREEDMAN: Stan Freedman, EIA. 8 MS. TAYLOR: Yvonne Taylor, EIA. 9 MR. JOOSTEN: Jim Joosten, EIA. 10 MR. BRADSHER-FREDRICK: Howard 11 Bradsher-Fredrick, EIA. 12 MR. WOOD: John Wood, EIA, Reserve 13 and Production tables. 14 MS. SPENCER: Lynda Spencer, Kema 15 Incorporated. 16 MS. FRENCH: Carol French, EIA. 17 MR. ZHANG: Bin Zhang, EIA. 18 MR. MILLER: Herb Miller, EIA. 19 CHAIRMAN BREIDT: Okay. Bill is 20 checking the halls for anyone else. Okay, 21 it's now time for my favorite announcement, 22 for your information Nancy Kirkendall is the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 11 1 designated Federal officer for the advisory 2 committee. In this capacity Dr. Kirkendall 3 may chair, but must attend each meeting and 4 she is authorized to adjourn the meeting if 5 she determines this to be in the public 6 interest. She must approve all meeting of 7 the advisory committee and every agenda; also 8 she may designate or substitute in her 9 absence. 10 Okay, so we have a very interesting 11 agenda today, we have a number of breakout 12 sessions. This is one of the most technical 13 agendas I think we've seen in a long time. 14 So there's a lot of statistical and 15 econometric meet here, so this should be a 16 lot of fun. There are a total of 10 breakout 17 sessions and the first session this morning, 18 which is not a break out session, plenary 19 session. 20 We'll begin the briefing with EIA 21 Administrator Guy Caruso, on EIA's general 22 state of affairs and then Nancy Kirkendall BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 12 1 the director of EIAs Statistics and Methods 2 Group will tell us about EIA work since the 3 spring 2004 meeting and the advice we 4 provided. 5 Lunch for the committee, presenters 6 and invited guests will be on the first floor 7 at 12:15. It will be in the usual room. And 8 this evening we have reservation for dinner 9 at the Aroma Indian Restaurant 1919 "I" 10 Street Northwest. You can take the blue line 11 or the orange line to get there. It's pretty 12 close to the hotel GWU Inn that we're staying 13 at. Our reservations are at 6, so can I see 14 a show of hands for who will be attending the 15 dinner. 16 Okay, and tomorrow morning 17 breakfast for the committee will be here 18 again beginning at about 8 and we'll resume 19 at 8:30 tomorrow in this room. So I guess 20 that's it. Now it's my pleasure to recognize 21 Guy Caruso, Administrator of the Energy 22 Information Administration. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 13 1 MR. CARUSO: Thank you Jay and good 2 morning everyone and special welcome to ASA 3 committee members and I want to first start 4 by telling you how much we appreciate the 5 work that you have done and now my fifth 6 meeting since I've been here and I think 7 we've found this advice extremely valuable, 8 and as Jay said today's agenda I think shows 9 the kind of specific linkage to the work that 10 we've been doing that I'll talk a little bit 11 about and how important it can be and in the 12 work of EIA and in the I think the accuracy 13 and the timeliness of our information and the 14 analysis. 15 So thank you once again and I will 16 say a little bit more about Jay specifically, 17 at the end, because as you known this is his 18 last session as the chair and a so. Let me 19 start by talking a little but about what's 20 been going on at EIA since our last meeting 21 in spring. As you know every year EIA is 22 asked to do winter fuels outlook and this BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 14 1 year the attention on the outlook for this 2 winter was particularly acute with rising oil 3 and gas prices and it fairly is -- as we'll 4 talk about the interest not only in market 5 but on the hill and the public has been 6 higher than ever. 7 We're going to talk a bit about 8 confidentiality. I know this committee has 9 had a number of briefings on by Jay and Nancy 10 in the past and that's continuing to play out 11 with respect to the statistical agencies and 12 guidelines from OMB etcetera as well as more 13 general data access that we talked about in 14 this committee. 15 Budgeting is always an issue, of 16 course to be able to achieve the results we 17 hope to and I'll bring you up to date on 18 where we are and the fiscal year 2005 budget 19 and we've reached the committee last meeting 20 on our strategic plan which we came to 21 agreement on as organization and individual 22 offices since we last met and bring you up to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 15 1 date a little bit on the progress towards 2 that. 3 Winter fuels as I mentioned you 4 know, enormous interest not only because of 5 the type of markets both in oil and gas but 6 going into a political -- a presidential 7 election there's even more interest in what 8 we do than ever. As we do our monthly 9 forecasting and put out our weekly data these 10 are all areas, which this committee has 11 directly or indirectly had some influence 12 over and will continue to. 13 In today's meeting we're going to 14 talk about some of the surveys that feed into 15 this Short term energy outlook and 16 specifically some of the results and the 17 regional modeling of this short term energy 18 outlook and the customer survey of the short 19 term energy outlook which we'll be talking 20 about in fact in the next session of the 21 breakout session. 22 That just shows in this slide the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 16 1 specific sessions we'll be having today and 2 tomorrow that feed directly into the 3 methodology and results of our short term 4 energy outlook and so once again a very 5 topical very timely, very much appreciated 6 the advice we've gotten from you on our 7 survey's and on our frames and in particular 8 on the -- for the modeling of the short term 9 energy outlook which continues to be enhanced 10 particularly on the regionalization aspects 11 of that. Dave Costello will be here to talk 12 about that. I think that's later on today or 13 tomorrow. 14 Another one which really started 15 two years ago was our desire to more directly 16 collect monthly natural gas production data 17 and John Wood and his people in Dallas have 18 been working with this committee for, in fact 19 from the first meeting that I've been here -- 20 Five meetings now though and finally we have 21 -- we hope some good news and that is that 22 OMB has approved the natural gas production BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 17 1 survey forum 914 in Iowa. 2 There'll be a session later on 3 today that John will be talking to you about 4 it -- specifically about what we're doing on 5 that survey and what we hope to achieve and I 6 can assure you that the work that this 7 committee has done has directly led to the 8 development of that survey and we were hoping 9 for more timely and more accurate natural gas 10 production data, that's our goal and I know 11 that the winters of '02-'03 and '03-'04 are 12 when natural gas prices spike so much there 13 was an increase in political concern over 14 natural gas and this will be, we hope, an 15 important response from EIA. 16 On the budget side the congress 17 went on recess without having passed the 18 budget in our case, an interior budget, where 19 our budget is an interior committee. 20 So we're on a continuing resolution 21 for fiscal year '05 at the same spending 22 level of '04 which is always a challenge BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 18 1 because we were expecting a 5% increase going 2 from 81 million to 85 million this year. So 3 we're having to operate at a lower pace of 4 spending than we would've liked and that 5 directly affects some of the things including 6 the natural gas survey that I just mentioned 7 but, we will actually have a session led by 8 Howard Gruenspecht tomorrow morning to talk 9 about that and get your advice on operating 10 in this kind of tight budget scenario. 11 CIPSEA as you have been -- as this 12 has been on the agenda a number of times 13 here, continues to be in effect and we are 14 now working with OMB on the specific 15 guidelines for the agencies like our own, 16 like EIA, who are not directly, mentioned in 17 CIPSEA, but really come under it and we're 18 working very hard to tighten up security of 19 the data and we are doing things like 20 training and other areas of -- to basically 21 to improve the security of our data. One of 22 the areas we've talked about in this BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 19 1 committee is granting access to researchers 2 and we've started a inter- office team to 3 look into this issue, Nancy Kirkendall is 4 leading that and we've also have had meetings 5 with a number of other statistical agencies 6 and work with them on getting advice and 7 ideas on how they do it and some of the 8 agencies are represented here, that we work 9 with -- on that. We will continue to work -- 10 to come up with a way that we think can 11 satisfy the needs of researchers but also 12 meet the requirements under CIPSEA and other 13 confidentiality pledges. 14 The last meeting we talked a little 15 bit about how sensitive some of the EIA data 16 has become with respect to the financial 17 institutions and markets in general and it's 18 -- I think it's interest is heightened more 19 since then because several financial 20 instruments have been launched on NYMEX and 21 other for which EIA data is directly being 22 used for either derivatives or auctions and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 20 1 therefore when our weekly data for either oil 2 or natural gas comes out, Wednesday or 3 Thursday morning, there's considerable amount 4 of writing on it and therefore a lot more 5 interest there than there's ever been, by the 6 media, by the financial houses and the 7 public. 8 So this sort of feeds back to the 9 issue of confidentiality and need to even 10 have greater security of our data, which you 11 know I think EIA has not been subject to the 12 kind of scrutiny that other statistical 13 agencies that produce the principal economic 14 indicators such as Census BLA and BLS have 15 dealt with this for decades whereas it's kind 16 of new for EIA and we're, again working with 17 all of those agencies to work on data 18 safeguards and I think that this slide really 19 speaks to that. 20 Certainly we need to prevent the 21 unauthorized release of data and we're 22 finding that the scrutiny is when there are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 21 1 billions, or certainly hundreds of millions 2 of dollars riding on what that number is each 3 week. We're finding consequences that is 4 very, very close scrutiny by as I mentioned, 5 financial institutions but also the media, 6 which uses our data to inform their 7 customers. 8 One of the technical issues which 9 isn't directly an issue for this committee is 10 that our website is being utilized 11 intensively during those few moments just 12 when the data is to be released at 10:30 on 13 Wednesdays and Thursday mornings and many of 14 the consumers are using robots to -- the term 15 of it is pinging our system, hundreds and 16 thousands of times per second and that has 17 created a bandwidth problem for us in terms 18 of speed at which our customers can acquire 19 the data and so we're working on that as well 20 but that's really more of an IT issue but 21 clearly one of the consequences of this 22 heightened interest in EIAs data. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 22 1 The other thing, Jay Edmonds isn't 2 here, he's had a close interest in the 3 greenhouse gas collection -- data collection 4 issue, that is the guidelines for collecting 5 emissions data, voluntary reporting of 6 Greenhouse Gases has moved forward and is 7 close to being finalized at OMB for final 8 approval right now, but it will have a direct 9 impact on EIA because we will be required to 10 collect a great deal more of this greenhouse 11 gas emission data than under the current 12 system. Talked about the strategic plan 13 goals last time. 14 These are the 3 main goals and 15 there are a number of sub-elements under this 16 and perhaps the quality of data is one that 17 this committee has dealt mostly with and 18 that's -- the next slide talks about some of 19 the things that the -- this committee has 20 done on the survey quality assessments and 21 then again the issue of the quality of our 22 analysis and the sufficiency of the frames BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 23 1 which we'll talk about directly this morning. 2 And the 3rd goal, which is, more of 3 having to do with the motivation and the 4 leadership quality of EIA, motivation of the 5 staff and productivity. We have -- there 6 have been a couple of developments one, we 7 did a survey, an organizational assessments 8 survey here in EIA in May and June of this 9 year and Howard and I are very pleased with 10 the results which are shown in this slide, 11 comparatively we've improved in terms of the 12 staff's agreeing or strongly agreeing with 13 the statement that the management provides 14 quality and continues improvement by creating 15 and modeling in the organization of vision 16 values and quality. 17 I think you know -- this is fairly 18 very positive development and we know we can 19 still do better and we're looking for some of 20 the issues, that we're going to get on our 21 agenda -- today -- again to lead to further 22 improvements in those areas, in both of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 24 1 goals, one and goal three. The other thing 2 on goal one is we did a -- we participated in 3 American Customer Satisfaction Index surveys. 4 It's called ACSI on our website and for the 5 first time I believe. Is that accurate 6 Colleen? is that the first time we've ever 7 participated in ACSI? 8 MS. BLESSING: No we did it -- we 9 participated in 2000. 10 MR. CARUSO: 2000, okay. But we 11 hadn't done it for several years and we did 12 it again this year. And the ACSI score of 13 72, which we thought was very very pleased. 14 It was a bit higher than the Government 15 average and -- but I think as important were 16 some of the areas where we thought that we 17 could use to improve our website, which is 18 becoming increasingly used and in just the 2 19 years or so that I've been here. 20 And what we've done about improving 21 the website is to appoint a web champion, 22 John Weiner and John is taking that task on BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 25 1 and we're certainly looking at ways on 2 improving the navigation and other issues 3 that the ACSI survey pointed out and 4 certainly it will be coming to this committee 5 in future to ask for further assistance and 6 advice from you. 7 So that's a very quick run through 8 of some of the things that we've been doing 9 here at EIA and the events that have been 10 driving our agenda here in the last six 11 months and looking forward to again getting 12 the advice and guidance from this committee 13 during the next day and half. So thanks once 14 again Jay, I'd be happy to answer any 15 questions now or at any time during the next 16 two days. 17 CHAIRMAN BREIDT: Okay, thank you. 18 MR. CARUSO: Are there any 19 questions? 20 CHAIRMAN BREIDT: What is a web 21 champion, is that like a drug czar? 22 MR. CARUSO: Yeah, we have a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 26 1 committee within EIA, from each of the 2 offices that looks on their specific products 3 that goes on the web and in effect John is 4 going to be the coordinator of that group. 5 But it sort of, yeah it's kind of more of, 6 we're trying to give it a little more, let's 7 say imprimature, more like a czar you're 8 right about that. You know, if there aren't 9 any questions let me take this moment this 10 moment to more formally thank Jay Breidt who 11 has been the chairman since last year -- how 12 long has it been? 13 CHAIRMAN BREIDT: Two years. 14 MR. CARUSO: Two years. That went 15 by fast, and on the committee itself for six 16 years. I have a letter from Secretary 17 Abraham, which I'll read and then Nancy has a 18 certificate, which she'll present to you. 19 "Dear Dr. Breidt, Thank you for 20 your outstanding service as committee chair 21 and as a member of the American Statistical 22 Association Committee on energy statistics. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 27 1 You have contributed substantially to the 2 effectiveness of the committee during the 3 past six years. Your leadership in guiding 4 the committee and you're continuing examples 5 of vision, encouragement and sound judgment 6 have kept the committee productive and on 7 course. Your insightful comments and 8 technical discussions have helped the Energy 9 Information Administration in many technical 10 areas during the past 6 years. 11 In particular we appreciate your 12 advice on selecting estimation procedures 13 using relative standard errors, editing and 14 evaluating EIA survey testing and methods. 15 We thank you for your vision and help, and 16 hope that we may continue to call upon you 17 from time to time for advice and assistance. 18 Sincerely, Spencer Abraham, Secretary of 19 Energy." Thanks Jay. 20 MS. KIRKENDALL: And before you 21 disappear we also have a certificate and this 22 is from Guy and myself thanking you for your BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 28 1 work with the committee. Oh I guess I'm 2 supposed to present next. 3 CHAIRMAN BREIDT: So it's my 4 pleasure to introduce Nancy Kirkendall. 5 MS. KIRKENDALL: Okay well Guy went 6 through updates on a lot of the activities 7 that we had in the last meeting and I'm going 8 to sweep up with just a few additional ones. 9 There are 4 topics that I'll talk about. 10 Electric power, the revision policy and 11 survey testing methods and then estimation of 12 stock of other oils. So on electric power we 13 had a big session in the last meeting where 14 Doug talked about transmission papers, he'll 15 be here later, that paper is -- I think it's 16 been approved by Howard, it's about ready to 17 come out of the front office and they're 18 trying to collect names of people who would 19 like to have a hard copy. 20 It is going to be posted on the web 21 so you can always look at it there. But if 22 you're interested in a copy of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 29 1 publication let us know and we can make sure 2 that you get one. 3 We also talked about the focus 4 groups that Howard Bradsher-Fredrick and Phil 5 participated in, that also illuminated our 6 data needs on transmission. And the electric 7 power package, they actually did use the 8 information that we learned from the focus 9 groups and transmission paper. I think your 10 advice was that we needed to do these things 11 a little bit more quickly and coordinate them 12 a little bit better and of course, that's 13 good advice. 14 But it did make an impact on the 15 questionnaire and right now the electric 16 power package, it received a number of 17 questions from the industry, they're not sure 18 they would like to report transmission 19 outages to EIA. And so I guess EIA is in the 20 process of coming up with it's response back 21 to OMB. Of course that was one of the 22 recommendations from Doug's paper. I guess BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 30 1 that was all on that one. 2 All right revision policy -- we 3 talked about revision policy that was Alethia 4 Jennings and Renee Miller. Just to remind 5 you the goal of -- we have a standard that 6 says that we shouldn't revise our data too 7 much and I think that the committee wasn't 8 sure that -- they thought that was 9 particularly important. But we presented our 10 new policy on revising data that we had 11 previously called final, trying to make it 12 more accurate and the collaboration that goes 13 on within EIA to agree on changes like that. 14 And you saw the proposal was fine, 15 it is being implemented now and there've been 16 a number of changes that have been made that 17 way. Stan Freedman and Bob Rutchik talked 18 about their survey testing methods. We just 19 went through what we think are good 20 approaches on testing and we ask for your 21 assessment. Committee recommendations are on 22 the next slide and we're basically trying to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 31 1 follow your advice since then -- next slide 2 -- Stan has presented this paper in Amsterdam 3 and has gotten some good comments on it and 4 approval of the methods and he's also been 5 invited by Lynda Carlson who of course 6 started this whole cognitive testing approach 7 within EIA to conduct a workshop on our 8 methods at her office and she also wants us 9 to work with her going doing cognitive 10 testing and some of their surveys. 11 So there will be some cross- 12 fertilization. She also has some people who 13 have done cognitive testing, so we can learn 14 a little bit, they can learn a little bit, 15 that and might be a good collaboration. 16 We talk about -- the project was to 17 come up with a better way of estimating 18 weekly stocks of other oil. Basically this 19 is -- we have a weekly survey that gives us 20 the inventories of the major products and 21 other oils is sort of the rest, however, we 22 also like to publish it total. So we need an BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 32 1 estimate for the other, so that we can get a 2 good estimate in total. There was a problem 3 last year when stocks -- our estimate rather, 4 oil was pretty bad and so this has been a 5 project to try to do better. 6 The major improvement has been 7 caused by collecting propane data, so now 8 what we're estimating is much smaller than it 9 used to be and that was the thing that had 10 the biggest change during that problem time 11 period. We're still looking at ways of doing 12 a better job, estimating the residual -- next 13 slide -- I think you're recommendation was 14 that we needed to collaborate better because 15 we have many efforts in EIA that look at 16 estimating stocks of other oils including our 17 short term energy outlook folks as well as 18 the petroleum people. 19 So we're still working on this, we 20 have come up with a couple of our best 21 regression based methods and now we're doing 22 some comparisons and now we're working with BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 33 1 the other program officers to see what we can 2 come up with and that is basically what I 3 have as an update. Anybody have questions 4 about anything? Anything either Guy or I 5 forgot to talk about that you were 6 particularly remembering as an issue from 7 last time. Okay, thank you. 8 CHAIRMAN BREIDT: Thanks Nancy. 9 Okay we're now ready to move to break out 10 sessions and if the committee members take a 11 look at the break-out sessions. You might 12 notice that if you are not Barb then you 13 always go to the same place. So if you're 14 upstairs you always stay upstairs, if you go 15 downstairs you always stay downstairs, except 16 Barb who is the only exception -- 17 MS. FORSYTH: I need to work out. 18 CHAIRMAN BREIDT: Okay. Last time 19 we had some complicated fractional factorial 20 and people were -- complaining about that, so 21 this is a little simpler. So let's go ahead 22 and reconvene in breakout. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 34 1 (Recess) 2 MR. RUTCHIK: Good morning, I'm Bob 3 Rutchik. I'm of Statistics and Methods Group 4 of EIA. I want to welcome you to the third 5 breakout session. In the past year and a 6 half that we've had on EIA 920 combined heat 7 and power plant reports. The previous two 8 sessions were in April of last year and that 9 was on the EIA 920 pre-survey design visits. 10 In last October a year ago we had 11 another session on proposed respondent level 12 and aggregate level edits for the EIA-920. 13 The purpose of this session is to get your 14 feedback, -- the committee's feedback on 15 EIA's plan to evaluate the effectiveness of 16 the form EIA-920 which has been collecting 17 data since January of this year. We picked 18 the EIA-920 to apply this program to, that 19 I'll be discussing, because we at EIA have 20 extensive experience in the survey's design 21 and testing. We are very familiar with it 22 and very comfortable with it. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 35 1 This is the plan on which I'm going 2 to be asking your advice -- for which I'm 3 going to be asking the committees advice. 4 As you can see there are four 5 criteria, which we have proposed to use to 6 evaluate the EIA- 920. These criteria are 7 timeliness of response, ability to provide 8 data, accuracy of response and reduction in 9 processing time for the EIA. And shortly I 10 will go over measures to operationalize each 11 of these four criteria and I'll also be 12 asking your advice on the measures to 13 operationalize each. We also hope to use 14 this proposal as a template to evaluate other 15 surveys, which EIA has applied it's survey 16 design and testing program to do something 17 that Nancy talked about earlier. 18 So EIA will not just have a testing 19 and design program, we will have a testing, 20 design and evaluation program. Now before I 21 get to the measures for each of the criteria, 22 I just want to review the EIA-920 with you. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 36 1 The EIA-920 combines a heat and power plant 2 report, collects data from combined heat and 3 power plant facilities such as, papers mills, 4 refineries, steel mills and food processing 5 plants and so on. We collect data on four 6 variables. One is total fuel consumed by the 7 CHP facility, the second is fuel consumed to 8 generate electricity in that CHP facility, 9 third is electrical generation and fourth is 10 fossil fuel stock, coal and distillate oil. 11 The major focus of the survey is 12 fuel consumed to generate electricity. If 13 the respondent has understood the survey, the 14 EIA-920, the reported quantity for fuel 15 consumed to generate electricity should be 16 less than total fuel consumed because, 17 basically the CHP plant should be using it's 18 fuel for two purposes. One, to generate 19 electricity and the other for it's 20 manufacturing process, such as drying paper. 21 Two other things about the EIA-920 22 that I just forgot to mention is that it is a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 37 1 monthly form of a sample selected from all 2 CHP facilities and I think there are probably 3 about 600 -- and that could be wrong -- CHP 4 facilities in this country. There's also an 5 annual form that's completed by CHP 6 facilities that are not selected to respond 7 monthly. Okay, now into the variables that 8 we plan to use, or hope to use - - to 9 operationalize the four criteria that I 10 presented earlier. The first criteria, is 11 timeliness and under timeliness we want to 12 measure that or evaluate that by more 13 respondents reporting within the reporting 14 period. 15 What I mean by more respondents is 16 that before the EIA-920, EIA used to collect 17 data from CHP plants on the form EIA-906, 18 which is power plant reports, now that 19 EIA-906 collects from regulated utilities, so 20 add more respondents to the EIA-906, and two 21 fewer respondents getting reporting 22 extensions, EIA would get phone calls from BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 38 1 respondents saying, no I can't fill this up 2 by the 15th of the month can you give me an 3 extra 5 days or 10 days or so on. And also 4 I've been using the word "we" a lot here, I 5 think that's a good sign to mention that on 6 this project I'm working with my colleagues 7 Tom Broene, Stan Freedman, Guy Caruso and 8 Karen Norman, all of Statistics and Methods 9 Group. 10 The 2nd criteria that -- in our 11 plan is ability to provide data and we have a 12 number of measures that we plan to use to 13 evaluate that. One is response rate, and 14 again, the response rates -- you know, higher 15 than let's say then was on the EIA-906, 16 that's just one example. 17 Now there is reporting errors, 18 pardon me. Let's say that there's a problem 19 with a particular variable -- all right, in 20 the EIA-920, let's say that instead of 21 recording just fossil fuel stocks such as 22 coal -- you know, plants are recording BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 39 1 something in storage that is not a fossil 2 fuel. So there may be a problem with the 3 instructions or the survey itself not being 4 clear. The 3rd thing we have is analysis of 5 comments, you'll notice at the bottom of the 6 first page of the EIA-920 the section of 7 comments from respondents, so we're going to 8 analyze those respondents and also their 9 comments that from EIA staff, that is put in 10 the file for the EIA-920 so we're going to 11 look at that too, and also down the line 12 after we've -- you know, looked at response 13 rate, reporting errors and all sorts of 14 comments and other things we're going to talk 15 to the EIA-920 processing staff. 16 We're going to ask them a lot of 17 questions but main -- basically things like, 18 what you think is going right with this 19 survey, what do you think the problem area 20 is, what's the bulk of your pullbacks on, 21 what's the bulk of questions that you get 22 from respondents, what can we do to improve BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 40 1 the survey and so forth, a lot of things. 2 And then, after that probably one of the last 3 things we're going to do is, we're going to 4 follow-up on respondents and we're going to 5 ask them question as basic as you think this 6 survey is an improvement over the survey 7 EIA-906, do you understand what EIA wants in 8 question 2, that's a key question there, fuel 9 used to generate electricity and several 10 other questions. 11 Okay we have, two other criteria 12 and the 3rd one is accuracy of response. One 13 thing we're going to do statistically is 14 compare data on the EIA-920 to data collected 15 on the EIA-906, the power plant report. This 16 is a little bit of a pitfall if it's here, 17 because on the EIA-906, EIA collected data 18 from CHP plants for something called useful 19 thermal output, which is a fuel used for 20 manufacturing and other processes and combine 21 heat and power plants. 22 From useful thermal EIA then BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 41 1 calculated fuel used to generate electricity, 2 as the difference between useful thermal 3 output and total fuel consumed. It was EIAs 4 wariness about the accuracy of use of thermal 5 output in the calculated fuel used to 6 generate electricity now but that led to the 7 design and implementation of EIA-920, so the 8 comparison there is a little bit fraught as 9 one of my colleagues suggested yesterday. 10 Two, we're going to try to do this 11 also this act by comparison with sub groups 12 in the EIA-920 by prime mover steam turbine 13 configuration. There's different type of 14 turbines, there's steam turbines, gas 15 turbines and other types of setups. And also 16 by industry is there a difference between 17 paper mills versus food processors, or 18 plastic manufacturers versus refineries and 19 so forth. And finally, our final criteria in 20 the measures we've have for that, is 21 reduction in processing time for EIA. 22 We're going to look at -- you know, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 42 1 has there been a decrease in respondent call 2 backs and in other words that would be kind 3 of an indication if there's been a decrease 4 in call backs that respondents are having -- 5 are understanding the survey and in other 6 words they're filling in more items than, 7 let's say the EIA-906 and this will also 8 indicate that maybe the EIA is saving 9 resources on the EIA-920. 10 MR. RUTCHIK: So those are 4 11 criteria and on these criteria we just listed 12 out, we have measures we want to use 13 operationalize that criterion. We've only 14 done research so far on ability to provide 15 data and as I said there is preliminary 16 analysis and that's a very good phrase. We 17 looked at January to May data. We have 18 looked at about 140 respondents of the 19 approximately 300 responses that EIA got for 20 that period. We looked at respondent in the 21 EIA comments, then we compared the question. 22 One, total fuel consumption data to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 43 1 question two data, that's data -- fuel use to 2 generate electricity data. The comments 3 provided very little systematic information. 4 There is nothing in the comments that just 5 leaked out at you to say, this is going right 6 with the survey, this is going wrong, and 7 kind of all over the board and illuminated 8 very little. 9 The question -- two to question one 10 comparison though did yield to this breakdown 11 of the 140 or so responses that we looked 12 for. Seventy four respondents recorded a 13 fuel use to generate electricity at the CHP 14 plant was less then total consumption, in 15 other words, these people seem to have got 16 the basic concept of the survey. There were 17 six to eight respondents that did not seem to 18 get the concept of the survey. For most of 19 these, they report their fuel use to generate 20 electricity was equally to their total fuel 21 consumption. 22 There were some that did not report BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 44 1 consumption at all just before -- no data at 2 all, just had some other problems with their 3 responses. There is also some other 4 findings. I don't want to put findings in 5 quote. From our analysis a look at these 140 6 responses. 7 One is on waste heat fuel 8 consumption not reported. Let me explain 9 waste heat is. In combined heat in power 10 plants, some of them have what's called 11 combined cycle technology. And you are 12 getting the full brunt of my engineering 13 knowledge with this. Combined cycle 14 technology, in that fuel is, let's take coal, 15 is put into in a boiler churns(?) out turbine 16 electricity is generated. Then a process of 17 generating electricity, steam is produced, at 18 that time the plant CHP facility can do one 19 or two things with that steam, they can admit 20 it into the atmosphere. 21 Hence the phrase "waste heat," they 22 don't need it, they are not going to use it, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 45 1 it's waste, or they can recycle the steam and 2 use it, let's say as part of their 3 manufacturing processes, you know, to 4 manufacture the candy or again use my 5 favorite one, drive the paper in the paper 6 mill. Some are designed as this cognitive of 7 any views, we knew that the plants would have 8 problems reporting fuel consumption for waste 9 heat. 10 These are waste heat, we really do 11 not consume fuel for waste heating because 12 the production has already been done at the 13 first part of the cycle, so therefore they 14 would report zero fuel consumption for waste 15 heat, and that's what happened when we look 16 at the responses for the combined cycle 17 points in the 140 or so that we looked at, I 18 think we did about maybe about a dozen 19 combined cycle facilities of the 140 and none 20 of them I think reported any fuel consumed 21 with a waste heat. 22 The second kind of finding was a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 46 1 larger respondents appeared to report 2 correctly, what I mean by this is unless (?) 3 we have, let's say a large food processing 4 company or a large paper company and they 5 have eight facilities reporting eight or nine 6 -- 8 separate responses and as I looked at 7 them and Stan looked at them we see that 8 let's say six reported correctly, they 9 reported their fuel use to generate 10 electricity with less then total fuel. 11 Two, total fuel was equal to fuel 12 use to generate electricity. So three 13 quarters of the people in that company saying 14 that they got the idea of the survey, and it 15 also shows that this decentralized reporting 16 you know, the reports are coming to EIA or 17 EIA-920, it is not coming from the EIA from 18 the central headquarters, but from the 19 individual facilities. 20 And the final two things is that 21 our pre-survey design respondents reported 22 correctly in the 140 or so, there were about BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 47 1 four pre-survey design people that we have 2 visited and talked to and our education 3 efforts seemed to work, they seemed to get 4 the hang of the survey. There were about 6 5 or 7 cognitive respondents in this group, I 6 think was half and half on whether they got 7 the idea of survey or did not. 8 Now I can come up with a facile of 9 rationale for why it was this way with the 10 cognitive respondents versus the pre-survey 11 design respondents and I will. In the 12 cognitive respondents, we let's say visited 13 you know, just once facility out of let's say 14 3 or 4 of that company and that one facility 15 filled it out correctly but the others that 16 we didn't visit would not fill it out 17 correctly, so again direct education may 18 work. 19 Okay, so that's the only research 20 we've done on this so far. And these are the 21 questions I have for the committee. Are the 22 criteria, the full criteria that EAI has BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 48 1 proposed for the evaluation at the ENI-20, 2 are these fully valid? Are the other 3 criteria that EIA can use? And finally, are 4 the measures to operationalize each of the 5 four criteria -- are there criteria -- are 6 there measures other than the ones we 7 proposed that we can use still? 8 In other words, is there something 9 under ability to provide data that we have 10 not listed that we can use. Again to sum up 11 you know, the thrust of the question, is EIA 12 on the right track to assessing survey 13 effectiveness? Thank you very much and open 14 it up for the committee. Nicolas? 15 DR. HENGARTNER: I have a question 16 that is not addressing any of these issues 17 but I'm really thankful for you to have 18 provided that test form because the first 19 time I see what you were talking about, and 20 that I'm reading at the top and the sentence 21 says, report actual value. 22 MR. RUTCHIK: Very good. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 49 1 DR. HENGARTNER: These are not 2 available, report estimates the value, now if 3 I go down on the phone I see nowhere an 4 indication that the value you are collecting 5 are either real or estimate. That is an 6 issue. So that goes to the accuracy and the 7 ability to answer because if we don't know if 8 the numbers that we are getting are real or 9 estimate, then we have a problem. And there 10 is also no ability to correct as the fact 11 that you have an estimate as to make the 12 value in January and you get the real numbers 13 in February do you resubmit the form? What's 14 happening? 15 MR. RUTCHIK: Okay, this may not 16 fully answer your question, and that's 17 something I left out. When we did a 18 pre-survey design as it is in, they were not 19 cognitive interviews, the respondents told us 20 that it could measure directly fuel used to 21 generate electricity. It goes into the right 22 nature of the configuration of the CHP BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 50 1 plants. 2 But he said, through I'm backing 3 out of steam pressures and other things, that 4 they could provide us with an estimation. I 5 know this direct -- addressed it directly, 6 only question two but that's what they told 7 that stuff. What we were getting most of the 8 time in question two, if you used to generate 9 electricity are, you know, estimates, 10 evaluations done by the facility themselves 11 and as a matter of fact this was the just as 12 the talk of the first session we had on this 13 in April 2003. So I don't know that really 14 answered your question but these are some 15 questions -- calculations are right. 16 MR. BLAIR: May I follow up -- 17 MR. RUTCHIK: Yes, sure. 18 MR. BLAIR: To the next question 19 and that is, do you know from talking with 20 the respondents sort of what procedures they 21 use for estimation and do you agree with how 22 they are coming up with their estimates? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 51 1 MR. RUTCHIK: No, not at this 2 point, I don't know. But that would be a 3 good idea to use if, you know, we do 4 follow-ups. 5 MR. BLAIR: I think I mean just 6 about the second letter. If you are going to 7 continue these visits in what you described 8 as education that if you know sort of how 9 people are coming up with these estimates 10 that that could be some part of what you 11 continue to collect the information on and 12 also maybe build into part of your education 13 networks. 14 MR. RUTCHIK: But it is fun, I 15 think it's more basic one that you want to 16 know sort of which way it is estimated -- 17 DR. HENGARTNER: And the danger now 18 is that using question two, those estimated 19 value or how sophisticated, they already 20 estimated that as - - evaluated through a 21 survey by saying, well, hold the consumption 22 and the electrical estimate and consumption BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 52 1 producing -- well, not asking the price, they 2 could be the same. Well, you have to be a 3 little careful using those values to evaluate 4 how well the respondents are understanding 5 the form. As you are measuring out the 6 specification of the respondents, we need to 7 have as you say, calculating the pressure and 8 we are getting a specific way beyond 9 cognitive. 10 MR. RUTCHIK: Yeah. Anybody else 11 has anything they want to ask? John, I'm 12 sorry, John. 13 MR. BLAIR: I was just going to 14 come back to the criteria number, response 15 rate which is obviously important but 16 describe sort of separately and I wonder if 17 you are putting together sort of the unit 18 response rate and the item response rate, so 19 that for each of the variable each of the 20 questions that you can compute an individual 21 rate which is the unit response rate times 22 the item response rate. And that is what BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 53 1 really tells you I think what you have for 2 each of your data for instance. 3 MR. RUTCHIK: Good idea. I think 4 you should -- 5 MS. WAUGH: I'm looking at -- is 6 this on? 7 MR. RUTCHIK: Yeah. 8 SPEAKER: Can you tell me who you 9 are? 10 MS. WAUGH: Shawna Waugh from EAI, 11 I'm looking at some of the theory if you will 12 about survey talks about three factors, 13 timing, accuracy and cost. And it appears 14 that the EIA processing time might be related 15 to cost but I didn't actually see any 16 variable that would evaluate cost 17 effectiveness and that might be something to 18 also attempt operation on. 19 MR. RUTCHIK: Thank you. Okay is 20 there anybody else have any comments, 21 suggestions because all we got so far have 22 been very, very helpful. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 54 1 MR. BLAIR: Yes, maybe if you go 2 back for a bit to -- you talked about the 3 comments that you receive -- 4 MR. RUTCHIK: Yeah. 5 MR. BLAIR: And they were kind of 6 all over the place and there was no pattern. 7 I guess sort of two things and one is if I 8 get comments all over the place and the 9 comments seem sensible that worries me a bit 10 because, it sound like then that are 11 problems. 12 MR. RUTCHIK: Well, what I meant by 13 that I probably wasn't very clear, there 14 didn't seem to be any great mass or number of 15 comments on any one thing. I think I did 16 something where I did like a frequency chart, 17 a bar chart of frequency. You know, when I 18 did various categories and have two or three 19 on this, two or three on that, three of four 20 on this, three or four on that, there wasn't 21 something like let say 140 respondents and 22 not all those 140 -- well actually I looked BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 55 1 at the whole 300 actually, not everybody 2 makes a comment, so those who made comments, 3 there wasn't somehow but say 15 or 20 or 25 4 saying that we don't understand question two. 5 MR. BLAIR: Right. 6 MR. RUTCHIK: That's what I meant. 7 MR. BLAIR: Sure, but you also 8 suggest that you did some sort at least 9 informal coding of the comments in order to 10 -- 11 MR. RUTCHIK: Yeah, informal would 12 be a good word here. 13 MR. BLAIR: Okay, if now that you 14 are getting that was it 68 out 140 or 15 something, 142 that provided comments? 16 MR. RUTCHIK: No, I don't know how 17 many provided comments, we didn't count that 18 up. 19 MR. BLAIR: Oh that was different 20 -- 21 MR. RUTCHIK: That was a -- 22 MR. BLAIR: People who didn't get BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 56 1 it. 2 MR. RUTCHIK: Yeah, there was a 3 question one versus two comparison. We 4 didn't really count up how many people, of 5 responses -- respondents provided comments. 6 MR. BLAIR: Oh, okay, well and this 7 is kind of stupid(?) and you certainly want 8 to count that and depending on how many that 9 we have may be able to do some more formal 10 coding and track over time you know if there 11 are some differences but, how useful that is 12 depends on sort of how many comments you got 13 and if you got five comments you know what 14 can you do, but if you had 50 you know that's 15 something, something else. So if you have 16 some substantial numbers then I would you 17 know, look at developing some sort of a 18 coding for it. 19 MR. HOUGH: My name is Rick Hough 20 from Census Bureau, you guys investigated the 21 use of electronic reporting for your response 22 related issues? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 57 1 MR. RUTCHIK: I wish somebody here 2 from the program office was here, but I think 3 the survey is reported electronically. It's 4 not a mail in survey, respondents go to a web 5 site, they access the survey, they fill it 6 out and send it back electronically, I mean 7 that's a very welcome and a clear way of 8 explaining it but is it on an electronic 9 reporting system. 10 MR. HOUGH: Okay. 11 MR. BLAIR: I don't want to drag 12 this out, but, a question on the expert 13 review. You mentioned that the survey staff 14 with the people and I wonder if there might 15 be a larger, of course I'm actually not sure 16 what the survey says but includes and that 17 includes the people that are responsible for 18 EEO, the processing or people analysis with 19 the data you know for the users but you might 20 consider sort of to know exactly who is that 21 that are appropriate for this expert 22 revealing, maybe something part of you should BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 58 1 make it different information from the 2 processing staff, that you get from the data 3 collecting staff. 4 MR. RUTCHIK: Anybody else, 5 anything you wish to add. I want to thank 6 you first for participating and listening and 7 if you don't have anymore comments and if you 8 do please come forward I just want to thank 9 you because of the comments we got were very, 10 very clear in substance. Thank you. 11 SPEAKER: Good job. 12 CHAIRMAN BREIDT: Okay, I guess we 13 will get started with the summary of the 14 break out session. So first Johnny Blair 15 will summarize the proposal for announces of 16 collected data break out issues. 17 MR. BLAIR: EIA has begun 18 evaluation of redesign form 920 and in our 19 session we are basically commenting on the 20 evaluation plan and essentially comments were 21 very positive. There were three questions 22 that were posed to the committee, the first BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 59 1 one was, are the evaluation criteria the 2 right ones? There were four general criteria 3 which are timeliness of response to the 4 survey, ability to provide the data that is 5 requested, the accuracy of response, the 6 accuracy of those data and reduction in 7 processing time for the EIA. 8 And in general we thought yes, 9 those were appropriate criteria but we had 10 some comments, which I will come to later on 11 how to apply them. The second question to 12 the committee was, are there other criteria 13 that EIA can use and there were two points 14 that were raised here, the first which was 15 raised actually by someone in the audience 16 was that cost is not explicitly measured in 17 the criteria. There are some indirect 18 measures such as processing time, but we felt 19 that more could be done to track cost 20 directly in the track changing cost due to 21 the new form. 22 The second is that the form has on BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 60 1 page 2 interesting instruction, which is 2 report actual values if these are not 3 available, we put estimated values. And a 4 comment on this was that it would be useful 5 to know item by item for each respondent 6 which items are actual and which items are 7 estimated and so the suggestion was to make a 8 minor modification in the forms so that 9 respondents can indicate when they are 10 putting in estimated values and more over in 11 the follow-up with respondents which is also 12 a part of the evaluation plan to look at how 13 those estimates are arrived at, how are the 14 estimates being constructed and whether EIA 15 agrees in general with the estimation 16 procedures and if not what suggestion that 17 they can make to improve them and as part of 18 their education, ongoing education of 19 respondents, what can they do to improve 20 those estimates. 21 The third question to the committee 22 was what, what measures can be use to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 61 1 operationalize the criteria and just a couple 2 of comments here, one connected to -- related 3 to the expert review which is one procedure 4 that used right now. There is an expert 5 review by the survey staff and we suggested 6 that that might be expanded to include other 7 people such as the processing staff, data 8 analysts and more broadly other users to 9 participate in that expert review since 10 expert reviews are generally the low cost and 11 quick way to get response. 12 There should be really marginal 13 additional effort to expand the pool of 14 experts, they do their review. And finally 15 on that response rate which is obviously an 16 important criterion, really unit response 17 rate was focused on and some mention of item 18 response rate but the suggestion was that 19 item by item is question by question of that 20 the product that goes is really what one 21 should be looking at, the product of unit and 22 item response rate for each question to see BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 62 1 sort of what you're really getting at the 2 question level. 3 And those are the notes that I have 4 which actually probably took as long as our 5 session did, which very quickly but if any of 6 the other committee members want to correct 7 or add anything to what I've said, now is the 8 time. 9 CHAIRMAN BREIDT: Okay any comments 10 or question for Johnny? Okay thanks, Johnny. 11 And now I will summarize the secondary break 12 out session which is a customary evaluation 13 of the Short Term Energy Outlook STEO, this 14 is based on a random sample of 500 email 15 addresses from list of about 4,000 email 16 addresses. These are self-selected, it's a 17 list -- so these are people who have chosen 18 to repeat new updates monthly and these are 19 thought to be fairly representative of the 20 core users of the STEO so. 21 From that email list, you have a 22 logistically straightforward means of BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 63 1 sampling and there is a lot of automation 2 that can be built into it which is a nice 3 feature. The run for about 25% unit response 4 rate and one of the questions had to do with 5 response rate. Howard Bradsher-Fredrick 6 described the results of the survey, 7 generally showing pretty much satisfaction 8 with the STEO, it's fairly consistent for 9 results for other products particularly the 10 Annual Energy Outlook and the International 11 Energy Outlook and you might recall from the 12 April meeting those were reported on the 13 basis of this sample, the satisfaction 14 survey, in that case was a survey of NEMS 15 conference register, is that right. 16 So let's see, one of the questions 17 on the survey that was emailed to people was 18 whether or not that respondent would agree to 19 follow up for more detailed questions, so 20 again there is stage of self-selection. You 21 have the self-selected list of -- you have 22 those who agree to participate in the survey BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 64 1 and then further you have further self- 2 selection on those who might be willing to 3 follow it up. And still with all this 4 self-selection going on, you get something 5 that is not necessarily representative of all 6 STEO users uses or even necessarily of all 7 STEO core users but probably something useful 8 in terms of customer satisfaction. You get a 9 useful first look. 10 The questions for the committee 11 were, how often should customer surveys of 12 this type be conducted? Would once a year be 13 too frequent or not and is the response rate 14 adequate and then questions on other 15 suggestions. So on the first question, how 16 often should customer surveys of this type be 17 administered? The committee I think was in 18 general agreement, you didn't want to do it 19 every year sending out to everyone that would 20 be unduly annoying, so it was thought that 21 one thing you could do is to implement some 22 kind of rotating panel where you might BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 65 1 revisit some people so that you can track 2 change over time if there were changes that 3 were of interest, you do want to revisit some 4 of the same folks and a possible selection 5 for part of that core panel could be some of 6 these people who have self selected to hear 7 the survey again. 8 Beyond that you can have a rotating 9 component and since there is 4,000 names on 10 the list and that gets somewhat refreshed 11 over time as people add and drop, you can 12 rotate through the email addresses for the 13 sample as a 500 only over eight years so 14 that's not really unduly burdensome. The 15 second question has to do with the response 16 rate, it was 25.9% and would that be adequate 17 and again spent some what you mean by 18 adequate, this really isn't going to be a 19 representative sample of all STEO users again 20 but it should be a useful sample in terms of 21 trying to get some information on customer 22 satisfaction. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 66 1 So the list service is self 2 selected, the respondents are self selected 3 after four ways of email follow- up and still 4 the thinking is that you should be able to 5 get some useful information from those who 6 want to be heard, those who are actually 7 taking the time to respond to the survey are 8 those who really have an opinion and this 9 should give you useful first look at the kind 10 of those core users. 11 But you get something that's sort 12 of actionable where you might be able to work 13 on the STEO. The performance of STEO you 14 want to go back to those core uses who added 15 strong opinion and actually find out exactly 16 what is it that they have questions or 17 concerns about. And it seems that they do 18 have a number of mechanisms for giving 19 feedback to EIA on this product but through 20 the survey mechanism they do have this thing 21 where they can self select to be followed up 22 and nothing has been done with that yet but BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 67 1 it could be, so that can be very useful to 2 jot down with that follow-up to get some 3 discussion and specifics like model 4 components and things like that. 5 As far as increasing response rate, 6 there are things that you can do but with 7 multiple ways of email follow-up it's not 8 going to do the trick. At some point you 9 again have to change the mode of data 10 collection if you really want to increase the 11 response rate and the other possibility is 12 that, eventually there is going to be these 13 non respondents that are going to respond no 14 matter how many times you contact them to 15 this kind of conventional mode follow-up, you 16 might sub sample them and then make some 17 efforts to get out responses from them, may 18 with a simplified questionnaire. 19 So and maybe giving them some 20 incentive to respond, I don't know. Then the 21 other question was other suggestions and the 22 discussion kind of centered around other BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 68 1 modes of entry, sort of other entry points 2 for data collection and these included some 3 discussion of the American Consumer 4 Satisfaction Index ACSI which is a pop under, 5 must be careful not to say pup up. But pop 6 under methodology that is sitting there on 7 the web page after you close you browsers so 8 it has -- this has caused technical problems 9 where people have a dozen of these open or 10 something and they are irritating and the 11 other concern is that there is a sense of 12 frame bias because the more technological 13 savvy users have blocked all their pop ups 14 and pop unders and everything else. 15 So it's maybe not as useful as 16 other means of getting that data but it's 17 good to have for customer satisfaction not 18 necessarily the case that you need 19 representation, you need to give people the 20 opportunity of have their voices heard in 21 different mechanism, two different 22 mechanisms. So let's see and one of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 69 1 other possible mechanisms discussed was sort 2 of an invitation on the web page where you 3 were invited to participate in survey. Again 4 self-selection but again potentially giving 5 you useful information that could be followed 6 up on. So that was kind of the being focus 7 of our discussion I think. Is there any 8 comments from people who are there to follow 9 up on that question. 10 DR. FEDER: Jay when you said 11 rotating panel, are those panels going to be 12 interviewed more than once? 13 CHAIRMAN BREIDT: Yes -- 14 DR. FEDER: Okay. 15 CHAIRMAN BREIDT: I think of a core 16 panel that would be interviewed year after 17 year and the others would sort of rotate out. 18 DR. FEDER: I have one small 19 question, is that, what about new users is, 20 there any attention going to be paid to new 21 users? 22 CHAIRMAN BREIDT: Well, the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 70 1 interest through the new users who are core 2 users who self select and get in on the list 3 there you know, can get in there but I guess 4 it would be a good idea to try to get those 5 into the core as well, refresh that panel, 6 that didn't really come off -- there is a 7 concern about new users to kind of get a 8 lower level being more frustrated with 9 products and things like that, so it's a 10 definitely interesting sub group. Any other? 11 MS. KHANNA: Yeah, I think on the 12 clarification point. One of the things that 13 we talked about is the (?)N. We are talking 14 about listing this up under -- approach was 15 that -- even that would not get rid of self 16 selection or even the bias in the selection 17 because people who bookmark the pages would 18 not necessarily receive pop under or the 19 invitation, the letter that we sort of talked 20 about -- 21 CHAIRMAN BREIDT: Right. 22 MS. KHANNA: Having invitation but BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 71 1 even there there was going to be some issues 2 of bias even the potential sample that you 3 got -- 4 CHAIRMAN BREIDT: Right. 5 MS. KHANNA: Apart from 6 self-selection. 7 CHAIRMAN BREIDT: And that just 8 reminds me something that -- I think that 9 statistics is something like 10,000 hits on 10 the STEO page and 4,000 users on the lists 11 there and those are different population, 12 they overlap to some extent but this 13 particular survey is really aimed at those 14 core uses who have self selected to 15 participate in this email collection. 16 MR. BLAIR: Yeah, just a comment on 17 the official response rate and customer 18 satisfaction survey that there is some -- 19 there is some evidence in the literature in 20 fact that indicates that that non 21 respondents, excuse me, the respondents to 22 customer satisfaction surveys tend to have a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 72 1 more positive rating than non respondents 2 which seems somewhat kind of imperative but, 3 in fact the explanation or at least that's 4 been positive, is that respondents, our 5 sample members look at doing the survey as 6 kind of a favor for the agency doing the 7 survey, so there is a kind of issue of 8 reciprocity, so if they have a positive 9 feeling towards the agency or whomever is 10 doing the survey, there is someone more 11 likely to respond and those who are not so -- 12 this is something I think that you know, 13 maybe useful to keep in mind when you're 14 dealing with this low response surveys. 15 CHAIRMAN BREIDT: One other thing 16 we came is that -- the earlier respondents 17 were grumpier, so as they went along -- Are 18 there other questions or comments? 19 DR. HENGARTNER: Do you also 20 compile complaints? I mean customer 21 satisfaction you know, you don't want to get 22 people mad. You don't want them to complain, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 73 1 I mean as long as they are grumpy that's all 2 right but as long as they don't complain. 3 MS. KIRKENDALL: There is a comment 4 box on the website, that is people are 5 invited, they send in a comment if they want 6 to. 7 DR. HENGARTNER: And is it 8 reviewed? 9 MS. KIRKENDALL: Colleen wasn't 10 reviewing it every month and sends them out 11 to whoever -- those who have seen it. But I 12 think she does you know I think she said 13 that, that's what she does with it. 14 CHAIRMAN BREIDT: Yes. 15 DR. HENGARTNER: There are always 16 going to keep complaining. The idea is to 17 keep the rate as low as possible or to avoid 18 -- notice most things fight in time and 19 that's one way one could actually monitor the 20 ongoing quality. 21 MS. FORSYTH: For expecting contact 22 as well. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 74 1 CHAIRMAN BREIDT: Yes, because you 2 know, people are going to grumble, grumble, 3 grumble -- and then it goes up and that's 4 when you have to be worried about. So let 5 them have the monitoring of the time, which 6 seems to be not too hard in conception -- 7 MS. KIRKENDALL: Well, it's sort of 8 interesting because I think by telling 9 business every month but I don't know that 10 I've seen that summary that she puts out 11 monthly or anything that she would -- and 12 distributes a number of complaints that she 13 gets over time related to the same. 14 MS. FORSYTH: And it might mean 15 that some complaints that are specific to a 16 particular product or a particular, I don't 17 know -- a page or whatever that hearing about 18 them for another page people who manage other 19 contracts or other pages might be able to 20 manage the content -- the distribution of 21 that at least some kind of content might also 22 be -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 75 1 MS. KIRKENDALL: We should have our 2 new web champion to look at it. 3 MS. FORSYTH: Yeah. 4 CHAIRMAN BREIDT: Nagaraj? 5 DR. NEERCHAL: Well, another point 6 he is going to bring up is that you know, one 7 aspect of customer satisfaction is to letting 8 all the customers know that you are improved 9 -- you are trying to improve and I think one 10 thinks that might be useful is based on the 11 survey or even as of your normal way of doing 12 things, you come up with improvement, send a 13 general email to the list saying that you 14 have improved the service. 15 You know, so because many people 16 who are not responding, they may not even 17 have noticed the problem but the fact that 18 you saw a problem and you have improved it, I 19 think that is already kind of improves your 20 scoring in their book. 21 CHAIRMAN BREIDT: Okay, so we are 22 on time or pretty much on time to go to next BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 76 1 break up session. Can we reconvene there? 2 (Recess) 3 MS. SUTHERLAND: Good morning I'm 4 Grace Sutherland welcome to the session on 5 assessing EIA frames and update on EIA wide 6 projects. This will be an update from the 7 spring meeting as to what we've been doing 8 and where we are headed. For a brief 9 background, in the spring EIA presented to 10 the committee the activity this has been 11 involved in up to that point, it included 12 checking respondent list, comparing aggregate 13 data, examining supply to sufficient balances 14 and comparing price data volumes. 15 In response the committee suggested 16 that EIA ask known establishments to identify 17 others within the same market to calculate 18 propensity scores by post stratifying 19 potentially using census data. To apply the 20 principles of dual system estimation to 21 available frame data, to obtain as much 22 information from census without disclosing BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 77 1 sensitive data in order to identify missing 2 respondents. 3 That EIA might forego using 4 balancing items for measuring coverage -- the 5 balancing items with least favorable message 6 of assessing frames. In response to the 7 committee's suggestion with regard to asking 8 establishments to identify others within 9 their same market, EIA had tried that and 10 respondents have been reluctant to provide 11 EIA with their customer's information. 12 However we have had success in obtaining 13 names of the buyers or competitors. For 14 instance our electric power surveys has used 15 this technique. The EIA is not currently 16 preferring the sampling approach but may 17 consider it again in the future. 18 With regards to fine principles of 19 dual system estimation to available frame 20 data, EIA is in favor of this whenever 21 possible and have applied this principle, for 22 example EIA compared the EEIA electricity BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 78 1 renewable frame with the National Renewable 2 Energy Laboratory frame. We also now have an 3 EIA wide interoffice team with evaluating 4 frame sufficiency. Because of the difficulty 5 in coming up with quantitative assessments 6 for frame sufficiency for all surveys, EI is 7 pursuing a qualitative approach. 8 The team is using the information 9 gathered previously on it's survey and is not 10 only gathering additional information and 11 will look at frame stability over a longer 12 period of time. We currently have just the 13 past year or ongoing frame, and we don't know 14 whether or not surveys are using comparable 15 lists and if they are not, why? Whether 16 there reason are such constrains or legal 17 issues? 18 And at the spring meeting in 2005, 19 you will be hearing an update from that 20 team's effort. Up next you will hear Howard 21 Bradsher-Fredrick of the EIA who will 22 summarize the dual system estimation study. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 79 1 The EIA electricity renewable frame and the 2 National Renewable Energy Laboratory Frame. 3 And then Shawana Waugh of the EIA will 4 describe the collaboration between the Census 5 Bureau and EIA regarding the evaluating of 6 several EIA frames and manufacturing sector. 7 And last you will hear from Tom Lorenz of EIA 8 who will describe what he has learnt by using 9 data from EIA's petroleum surveys both 10 monthly and annual to edit MECS data. MECS 11 stands for Manufacturing Energy Consumption 12 Survey. Although Tom's analysis was an 13 editing project, the frames he will be able 14 to use, his analysis in effecting the frames. 15 And at the end of Tom's presentation, the EIA 16 would like a feedback from committee. But 17 first we will hear from Howard 18 Bradsher-Fredrick. 19 MR. BRADSHER-FREDRICK: Okay I 20 think my slides are coming up. Okay, I'm 21 going to discuss the application of the dual 22 system principles to compute a hypothetical BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 80 1 estimate of the number of frame elements 2 missing from both the National Renewable 3 Energy Lab data frame and I mean I will refer 4 to this as NREL and the EIA renewable frame. 5 As Grace mentioned this analysis was 6 requested by the committee at the last 7 meeting, last spring. 8 NREL collects data from secondary 9 sources and generates a nameplate capacity 10 from renewable energy generating plants, the 11 NREL list is publicly available and EIA 12 collects data on renewable plants but it was 13 a 1 megawatt cut off. So we have eliminated 14 the NREL facility below the cut off from the 15 study. This exercise was intended to show 16 how the dual system principles might be 17 applied in a real setting if these 18 assumptions were to hold, I won't get to the 19 assumptions until the end. 20 First I'll go through some of the 21 computations. So excuse me, we are going to 22 proceed to estimate the number of facilities BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 81 1 missing from both frames if the assumptions 2 were to hold. So we look at the total data, 3 you notice that the NREL is just 5,146 4 facilities, the EIA frame had 4,960 and if we 5 break this down into the components, we have 6 in the NREL list and in the EIA frame we 7 4,932 in common and we have in the EIA frame 8 28 that are not in the NREL list and we had 9 214 in the NREL list that were not in the EIA 10 frame. 11 So we had these three sales 12 together. We get a total of 5,174 known 13 facilities. So we want to try to figure out, 14 well, how many are we likely to have missed 15 in both. We can sell for X with this by 16 making a few assumptions. So we assume 17 independence, we have the probability of NREL 18 error times the probability of an EIA error 19 times the total number of facilities, we do 20 the mathematics here, we get 1.2 missing 21 elements. 22 We can also apply that principle to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 82 1 the various fuel strata that also have 2 information on. In this case in fact we 3 looked at all the fuels but all of them came 4 out to be zero except in the case of 5 agricultural residuals. In that case NREL 6 showed 27, EIA showed 33, and you will notice 7 here we have a higher error rate because we 8 have not -- there is a percentage not as many 9 as common, 19 in both, 14 that are not in the 10 NREL, but in the EIA frame, we have 8 that 11 are in NREL and not in the EIA frame and we 12 have X being those that we are trying to 13 figure out which are those that are not in 14 both is an estimate. 15 Again we can do the mathematics 16 here and we get 2.7 missing. Again applying 17 the same assumptions and those assumptions 18 that were essentially using here that there 19 are two necessary ones and I was really 20 seeing that neither of these hold. The 21 frames need to be independently assembled and 22 at times that seems not to be the case BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 83 1 because NREL uses all secondary sources and 2 we know they also use the EIA as a source. 3 And the missing frame information 4 should be random rather than systematic and 5 this appears not to be the case, I mean at 6 least in terms of timber residues. We have a 7 140, other 214 of the missing elements in the 8 EIA frame. The 65% being in all one fuel, so 9 this appears to be system error. 10 MS. SUTHERLAND: Okay, up next it 11 is going to be Shawana Waugh who is going to 12 talk about the collaboration between census 13 and the EIA. 14 MS. WAUGH: Good morning. I will 15 be talking about the five frames that are 16 being evaluated by the Census Bureau and they 17 are all in the manufacturing sector and I'd 18 like to start actually by introducing Rick 19 Hough from Census Bureau. He is the next 20 survey manager and he was actually -- if you 21 like to go ahead and stand up Rick. He is 22 the person who will be working on this BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 84 1 analysis. 2 I'll be covering the purpose of the 3 frames evaluation. I'll be reintroducing you 4 to the five frame surveys here, EIA(?) that 5 is being evaluated and then talking about the 6 methodology that we propose to use as well as 7 applying it to both EIA 5 which is COKE 8 plants and the EIA 63A which is solar thermal 9 panels and then I'll talk a little bit about 10 this schedule in terms of what is next. 11 The purpose of the frame evaluation 12 is basically to look at the coverage and to 13 do this in a quantitative way. Also the 14 surveys from the coal, nuclear, electric and 15 alternative fuel program office, and again 16 there are five of them. And also as part of 17 the analysis we're going to be looking at 18 differences between the EIA and the census 19 frame, as well as the forms that will affect 20 coverage and the way in which we present the 21 coverage. 22 Because you are looking at volume BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 85 1 and so there may be some differences in the 2 question wording on the forms that would 3 impact volumes that are published separately 4 by EIA and census. And also looking at 5 characteristics of establishments that are 6 missing from our census bureau frame. There 7 are two surveys in the coal area, two on the 8 renewable and one in electric power. 9 The two coal surveys, the EIA3 is 10 -- reports consumption of coal by 11 manufacturing plants and the EIA5 reports 12 consumption of coal as well as production of 13 coal, coke by coke plants. And the renewable 14 energy surveys include the 63A, which collect 15 information from manufacturers or photo 16 thermal collectors and the 63B, which 17 collects information from manufacturers who 18 produce photo voltaic cells and modules. And 19 the electric power survey, which actually, 20 they discussed at the previous session, will 21 include the sub set respondent to a report 22 for combined heat and power plants and that's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 86 1 on the aerial frame of the EIA6. 2 This displays the 5 frames again, 3 which are in the left column as well as the 4 census frames that we will be evaluating them 5 with. And it will either be the economic 6 census and the census of manufacturers or it 7 will be the current industrial report and 8 then in each case we are going to be looking 9 at either an input or an output and today I 10 will be talking specifically these solar 11 collectors where we are going to be looking 12 at the dollar value for value of shipments as 13 an output to determine coverage by volume. 14 And again there will be other 15 variables used depending on the key variables 16 of the survey and I've also displayed here 17 the numbers that are on the frame and three 18 of that we are currently working on are 562A 19 and 63B, all have a small population around 20 20 and 21 intake of the coke plant. The 21 manufacturing plants, there are eastward 2002 22 507 and the electric power plants there are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 87 1 actually more on the frame than what I've 2 shown here but we are going to be looking at 3 the 625 combined heat and power plants and 4 then also looking at the independent power 5 producers which are more like an electric 6 utility but may also be manufacturing. 7 There are a number of steps and 8 methodologies. The first one is to actually 9 conduct the matching and then that will be 10 followed by the analysis. And during the 11 matching we've identified the establishment 12 from the EIA frame and also the threshold. 13 So in some cases we may have a smaller 14 population than what the census bureau has 15 because, for example on the electric power 16 surveys we only have people who are producing 17 1 megawatt of electricity to report. Then 18 those establishments will be matched to the 19 economic census bureaus business register. 20 And again we are taking a ve