AMERICAN STATISTICAL ASSOCIATION COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION DAY 2 OF 2 WASHINGTON, D.C. FRIDAY, APRIL 20, 2007 2 1 PARTICIPANTS: 2 NAGARAJ K. NEERCHAL, Ph.D., Chair 3 DEREK BINGHAM, Simon Frasier University 4 EDWARD A. BLAIR, University of Houston 5 JAE EDMONDS, Ph.D., Pacific Northwest Laboratory 6 MOSHE FEDER, D.Sc., Research Triangle Institute 7 BARBARA FORSYTH, Westat 8 WALTER W. HILL, St. Mary's College of Maryland 9 NEHA KHANNA, Binghamton University 10 EDWARD KOKKELENBERG, Ithaca, New York 11 12 13 * * * * * 14 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 3 1 P R O C E E D I N G S 2 (8:40 a.m.) 3 DR. NEERCHAL: What I'd like to do 4 -- I think you start with -- I will ask any 5 Committee member who is not here and -- but 6 here today to go ahead and introduce 7 themselves and also the audience members, 8 anybody in the audience who is -- who was not 9 here, but who is here today to go to the mike 10 and introduce themselves. 11 And also I want to remind you that 12 please register and put your e-mail down when 13 you're coming in because of -- on the 14 registration table there. 15 Any new persons here? I do. 16 MS. CARLSON: Mary Carlson. 17 DR. NEERCHAL: You have to come to 18 the mike. 19 MS. CARLSON: All right. 20 SPEAKER: You have like those 21 foreign national fingerprint IDs to get into 22 the building. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 4 1 MR. BURNS: Hello. I'm Eugene 2 Burns. I'm with the Office of Energy Markets 3 and End Use in the Energy Consumption 4 Division. 5 MR. CONRAD: I'm Lenny Conrad with 6 SMG. 7 MS. BATTLES: I'm Stephanie 8 Battles. I'm Director of the Energy 9 Consumption Division. 10 MS. CARLSON: Mary Carlson, Office 11 of Oil and Gas, Natural Gas Division. 12 MS. MICHAELS: Joelle Michaels, 13 Energy Consumption Division. I'm the survey 14 manager for the Commercial Buildings Energy 15 Consumption Survey. 16 MS. O'BRIEN: Eileen O'Brien, 17 Survey Manager of the Residential Energy 18 Survey in the Energy Consumption Division, 19 EMEU. 20 MR. SWENSON: Alan Swenson, Energy 21 Consumption Division. 22 DR. NEERCHAL: Before we start off BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 5 1 with (off mike) today, I think we should take 2 a moment and remember the people who are 3 affected by the Virginia Tech tragedy. Today 4 is a national day of mourning, and, you know, 5 we all feel for them and that our colleagues, 6 and, you know, fellow citizens of Virginia 7 there. So maybe we should observe a moment 8 of silence and then. 9 (Moment of silence.) 10 DR. NEERCHAL: Thank you. Today's 11 first -- we have two plenary sessions today, 12 and the first plenary session speaker is 13 Jacob Bournazian? 14 DR. BOURNAZIAN: Yes. 15 DR. NEERCHAL: And he's going to 16 talk about the microdata. 17 DR. BOURNAZIAN: Yes. 18 DR. NEERCHAL: Okay. 19 DR. BOURNAZIAN: Well, Members of 20 the Committee on Energy Statistics, Fellow 21 EIA staff and guests, I'm happy to be here 22 this morning because this topic is an area BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 6 1 that's developing and it's on a fast pace 2 with the federal statistical agencies. 3 It's also an area that's been 4 growing with the federal statistical 5 agencies, and it's a long time since we've 6 been able to talk about this here at EIA, and 7 so my presentation today really comes at a 8 timely moment. We're talking about expanding 9 research or access to microdata or EIA's 10 microdata. 11 All right, Larry. EIA is 12 considering three options for expanding 13 research or access. There's a program that 14 Ruey-Pyng spoke about yesterday afternoon 15 with the National Institute on Statistical 16 Sciences. How many Committee members heard 17 Ruey-Pyng's presentation? 18 One, two, three, four. Okay. A 19 majority of you. All right. So I'll be 20 going each one of these options in detail, 21 but the NIST-EIA student research program, a 22 nice program on site here at the Forrestal BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 7 1 Building. I'll be going over that again. 2 A second option, though, is to 3 place our residential energy consumption 4 survey data and our commercial building 5 energy consumption survey at the Census 6 Bureau's RDC facilities. 7 There's nine facilities and the 8 Census Bureau conducts the Manufacturing 9 Energy Consumption Survey, MECS, which is 10 already housed at the Census Bureau's RDC. 11 So by placing these two surveys with the 12 Census Bureau, you'd have the whole sweep of 13 consumption surveys in one RDC facilities set 14 up. 15 There's a third option, and that's 16 to place the same two surveys at the National 17 Opinion Research Council's data enclave 18 that's going live May 1st, in a few weeks. 19 The big feature of NORC's data 20 enclave program is remote access, where the 21 researcher stays right there in their office, 22 right from their PC, dials in and can do the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 8 1 research. 2 It's a private VPN network. No 3 cutting and pasting allowed, no downloading, 4 no printing. You just -- you can do your 5 research and connect to those company-level 6 data files. 7 So we'll be talking about these 8 three options and at the end of my 9 presentation, I'm going to be asking you for 10 some input to help us evaluate which is most 11 appropriate and be effective for EIA. 12 Now, EIA has never engaged in this 13 activity to any significant extent as far as 14 allowing researcher access to the 15 company-level files. But there are benefits 16 and many federal, if not all of the federal 17 statistical agencies have moved in this 18 direction for this very reason. 19 There are substantial benefits by 20 allowing researcher access. First, you're 21 fostering an open research environment. 22 Basically, this benefit here is replication BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 9 1 of your results, the transparency of the 2 data, and of the information you're putting 3 out. 4 However, in every single agency's 5 case we've talked with that has allowed 6 researcher access, they found another 7 by-product of benefit here: It improves data 8 quality. Well, if you heard Ruey-Pyng's 9 presentation, he mentioned a few selected 10 topics dealing with statistical methodology. 11 Agencies learn about their data by 12 allowing researcher access. They evaluate 13 their statistical methods, but they also 14 learn a great deal about their data. In 15 fact, talking with RDC Administrator, a lot 16 of outliers, misreporting of problems with 17 the data get identified. Not that you can do 18 anything about it five years after the fact, 19 but you learn about your data and it has -- 20 and it impacts you on your analysis. 21 Now, in the old days at EIA, we 22 used this, okay? We went down through BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 10 1 output, company by company, and we analyzed 2 what those companies reported. 3 We don't do that anymore. Resource 4 constraints. We have automated edits. We're 5 going to sort the data, look for companies 6 are sufficient distance away from the crowd. 7 We're going to look at what companies make an 8 impact on a cell. We don't have time to call 9 three -- make 300 phone calls to validate our 10 data, and the automated edits work well. 11 But what does a researcher do? 12 Researcher wanders right into the forest, 13 looks at every tree, every branch, leaf on 14 the, you know, branch. Right, he's going to 15 look at a small company's reporting history 16 regardless if it falls in the geographic area 17 they're interested in. 18 So if every other agency has 19 realized these benefits, no doubt EIA will 20 benefit by the same way. 21 Strongest reason down here at the 22 bottom: Microdata they permit in-depth BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 11 1 analysis for many policy-relevant issues that 2 the agency just is not able to analyze. We 3 don't have the resources to do that. As 4 Ruey-Pyng pointed out on his research 5 projects, there's a lot of methodology 6 issues. We just don't have the time. 7 They're wonderful research projects, but 8 staffing and time is limited. 9 However, if you heard Guy Caruso's 10 opening remarks yesterday, he talked about 11 service reports. Well, we get a lot of 12 questions, besides questions from Congress. 13 We get questions on whether electric power 14 producers from Montana are leveraging 15 residential electricity consumers in Idaho. 16 You got questions why gasoline prices in 17 Georgia are more expensive than Florida when 18 Georgia's tax is only seven cents and 19 Florida's is 20. 20 Okay. We don't have time to get 21 into that real local state, not local, but 22 state analysis. In fact, what Guy said was BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 12 1 he has to prioritize. He can't do every 2 single service request. And what is he 3 doing? 4 He's regionalizing, nationalizing 5 it, and if there are other requests that 6 match up, because we're going to focus on the 7 national supply picture or regional. 8 Who is going to be able to analyze 9 the difference between Arkansas and 10 Tennessee? Maybe a professor from the 11 University of Tennessee would be interested 12 in that, but certainly EIA doesn't have the 13 analysis resources to go after every single 14 analysis question. So we would benefit a 15 great deal by using our data in this fashion. 16 Okay. Now I've made this point in 17 my paper, but I have to admit I probably 18 didn't do it effectively because my 19 discussant approached me at lunch time 20 yesterday and he said, you know, what do you 21 mean by this statement expanding researcher 22 access is going to increase EIA's investment BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 13 1 in the data collection. In fact, I used Ray 2 to return to my paper. All right? 3 Well, I promised that. I said, I'm 4 going to make it up to the whole Committee 5 this morning with this one slide. And I may 6 have failed in my paper; we'll come back with 7 a picture. 8 Well, we looked at where EIA spends 9 its resources. Now, Guy Caruso mentioned 10 yesterday that it was a strong wind blowing 11 through Congress yesterday or this year and 12 we got $90 million in the budget. 13 We didn't always have that strong 14 wind. Okay. Budgetary financing was more of 15 the '80s. Let's round down to $80 million. 16 And what's two- thirds of $80 million? I 17 think it's a little over $50 million, but I'm 18 going to round down to $50 million. And Guy 19 also mentioned that we've been around for 20 what? Thirty years? What's 30 times $50 21 million? $1.5 billion, to be exact. Where 22 are our resources and money going? It's BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 14 1 going to collect data. We processed the data 2 and now we have a bunch of master files with 3 company-level data on there sitting on the 4 shelves. Nobody is using it. We've got the 5 aggregate tabular data that's been released 6 and we're real good about that. We've got 7 online query systems that get you that 8 aggregate tabular data right from the web. 9 But nothing, nothing is going on with our 10 company-level data. It sits there, year 11 after year, decade after decade. I don't 12 even know if we could link up with our 13 company-level files from the '80s to even do, 14 you know, some long-term analysis with the 15 files in the '90s or even in this decade. 16 But we just keep on going out, 17 collecting, compiling, and stacking on 18 shelves, and it sits there. It sits there, 19 and so I'm going to point this out, because 20 I'm going to be talking about some options 21 that are cost money, tens of thousands of 22 dollars. I'm going about spending tens of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 15 1 thousands of dollars, talking about 2 increasing our data utility. 3 And when you compare the tens of 4 thousands of dollars I'm going to ask this 5 agency to spend, compared to the opportunity 6 cost of having over a billion dollars of 7 resources sit there, the benefits should be 8 there and the money as well worth spending. 9 Now, I hoped that addressed on what 10 I mean by rate of return, because Ed's a New 11 Yorker. Right? So what does he say in the 12 lunch line? Are you looking for seven 13 percent, eight percent -- the typical New 14 York question. And I thought about it on my 15 second pass through the linguini line is that 16 there's plenty of metrics. We could look at 17 users' accessions to the files. We could 18 look at research proposals. We could look at 19 products produced. You could even look at 20 citations on the research. There's many 21 metrics we could develop to measure the 22 usefulness and utility of this activity. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 16 1 Okay. Well, then why haven't we 2 just run out there and done this? I mean if 3 this is such a great activity and every other 4 federal agency is doing it, why aren't we out 5 there with the pack? 6 Well, you got some risks. And so 7 let's talk about those risks 'cause you want 8 to -- that's one reason, main reasons, why we 9 have not engaged in expanding researcher 10 activity. 11 Well, this is a general policy that 12 there's concerns over confidentiality. 13 Across federal statistical agencies, 14 confidentiality impacts response rates and 15 refusal rates, well documented in the 16 research. One thing that's unique about EIA, 17 we've got mandatory data collection 18 authority. You have very high response 19 rates, and so even though there can be 20 confidentiality problems, our response rates 21 would never be sucking wind as bad as what we 22 see in like Census or BLS that are down to BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 17 1 the 40 percent range on voluntary surveys. 2 Mandatory surveys in general always have a 3 higher rate. 4 So do we win automatically? Can I 5 dismiss this? No, not really? Cause lack of 6 trust negatively affects data quality. 7 Actually, lack of trust negatively affects 8 communication and even in a personal level; 9 right? 10 When trust is high, easy to 11 communicate with one another. When trust is 12 low, it's very difficult. The same in a 13 business setting. Okay. So what happens to 14 EIA if we cause a breach in confidentiality 15 or engage in a program that diminishes and 16 denigrates our pledge? 17 Well, I'll tell you one thing: 18 Volume data is going to get rounded down and 19 cost data is going to get rounded up; right? 20 That's what we get. EIA is going 21 to force me to report, you'll get your file 22 and report on time. Is it going to be the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 18 1 right data? Maybe not. Is that going to 2 impact us? Yeah. That's real important. We 3 have a good reputation for accuracy. 4 There's another common risk. The 5 perceived risk of a breach can be just as 6 damaging as an actual breach. Again, it's 7 dynamics of human relationships; the same in 8 a business setting. 9 However, there's one caveat on the 10 perceived risk. It's how you handle it. 11 It's how you handle the perceived risk versus 12 an actual breach. An actual breach you're 13 stuck. You're burned on that one. But on a 14 perceived -- how do you respond could 15 boomerang and be a huge deposit with your 16 trust and goodwill with your respondents. 17 Getting cast in a false light and then 18 correcting it with the truth always comes 19 back for strengthening that bond. 20 However, it's very important to 21 handle on your -- we have certain rules now 22 and notification of breaches and then BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 19 1 policies and procedures. 2 Here's a huge, another big risk: 3 EIA may get blamed for bad research. Now you 4 researchers are a suspect group. Why? 5 Because you're intermittently playing around 6 with the data. You come in and you want to 7 do energy studies. Then you want to do a 8 labor study. And then you come back and do 9 another energy study. Whereas hard-core 10 staffers, consultants, well, they're always 11 in the energy industry; right? 12 Are you guys going to get it right 13 this time? Wow, what if you get it wrong and 14 EIA gets blamed 'cause, after all, you used 15 EIA data. Mm. 16 Well, federal agencies handle it 17 two ways: Disclaimers and or extensive 18 disclosure -- product review. And you can 19 pick any flavor or a combination of the two. 20 A very interesting observation. 21 Actually this one here has led some program 22 officers to altogether pull out and say we're BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 20 1 not going to let any researchers touch the 2 data because bad research -- the ironic 3 thing, though, is on the aggregate data, and 4 I've talked about three core cases in the 5 Midwest where professors were called in as 6 expert witnesses, testified on jobbers 7 getting squeezed by refiners in the upstream 8 market; used EIA data, petroleum data and 9 won. The jobbers won. 10 The researchers did a great job, a 11 professional job of explaining that to the 12 courts and they used it correctly. 13 So somehow, maybe it's 14 intrinsically that they're able to be bright 15 and clever and on point using aggregate data, 16 but when you come to microdata, you guys just 17 mess it up every time; right? 18 You know, just like you can handle 19 magnitude, but not frequency. But it's 20 there. 21 The cost of providing access -- 22 normally that should not be a risk, except BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 21 1 for when you work at EIA. All right. This 2 is not always going to be a strong wind 3 blowing through Congress. All right? 4 And if you're going to commit the 5 agency to an annual expenditure of fifty 6 grand, ten grand, a hundred thousand dollars, 7 okay, that is a constraint. 8 And so it's a risk -- are you going 9 to be able to continue with these programs? 10 We have to look at cost. In fact, cost is 11 going to be the number one bullet item when 12 we go through these three options. 13 But I do want to put in context, 14 'cause I been talking about what other 15 federal agencies have been doing, and it's 16 pretty exciting. Remember when I opened, I 17 said this is a fast-developing area. 18 Well, if I gave this speech five 19 years ago, I'd have four agencies up here on 20 the top. But it keeps on growing. Now, it's 21 not just that little light bulbs are going on 22 and more and more agencies are getting into BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 22 1 this game and engaging in this activity. You 2 see -- all right. It's a weak laser pointer, 3 so good thing I had Chinese food last night. 4 Chopsticks seems to be in the morning. 5 Agency for Health Research and 6 Quality, they have a nice researcher access 7 program and what do they do? They host their 8 data at the National Center for Health 9 Statistics. They're -- they have an RDC in 10 Hyattsville. They pay an annual fee. Sort 11 of similar to the option I'm going to be 12 explaining for what we're considering with 13 the Census Bureau. The Bureau of Economic 14 Analysis, the Bureau of Labor Statistics 15 on-site researcher programs. 16 The Census Bureau, (off mike) they 17 have the nine research data center 18 facilities. National Agriculture Statistics 19 Service. They have a data lab right down the 20 street, and also 40 regional offices that are 21 connected on a secure LAN for researchers to 22 come in and access microdata. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 23 1 The National Center for Education 2 Statistics. Well, one of the leaders in it. 3 It started in the late '80s, mainly using 4 licensing, but now moving on. 5 The National Center for Health 6 Statistics and National Science Foundation. 7 Now, even the agencies that have 8 been in this activity for a while are going 9 through their second and third generation of 10 evolving; okay? It's not like they went and 11 got engaged in it; they're staying with it. 12 They're moving well ahead forward. 13 For example, NASS has a data lab, 14 but now NASS has also signed a contract with 15 NORC to place their data in their data 16 enclave. Why? They're looking for that 17 remote access feature. Okay. 18 Also, NCHS, they've got a research 19 data center. They signed a contract with 20 Census Bureau and placed their data at the 21 RDCs in Census. What are they doing? Just 22 increased their RDCs by nine- fold. Okay? BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 24 1 Now, besides all this, too many CDs 2 are probably ending up in researchers' 3 attics, but I don't know why, but NCES, the 4 Census Bureau, NASS, NCHS moving to online 5 query systems. You got CDC Wonder, in which 6 you can query a database and get, you know, a 7 company, household respondent-level records. 8 The Census Bureau American Fact 9 Finder query level you can do data restricted 10 queries, but you can go and access -- do your 11 research on company-level data. NASS has the 12 Agricultural Research Methods Survey. NCES 13 just started this in the last year. Besides 14 licensing, now they are also putting up, and 15 they have on their Web site their 16 company-level files. Again, buffered access 17 for approved researchers. 18 So you can see these agencies are 19 moving forward in developing multiple modes 20 for researcher access, not just one. 21 Okay. So who's been left behind? 22 The Bureau of Transportation Statistics, a BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 25 1 very small agency about one-third the size of 2 EIA. It's been around for about 10 years. 3 Internal Revenue Service and that's 4 administrative data. I don't think it's a 5 fair comparison, except they have one unit, 6 Statistics of Income. 7 The Social Security Administration, 8 once again administrative data. Now look at 9 this little agency right here. I'm not 10 saying we should do this because everybody 11 else is doing it. I'm saying everybody else 12 is doing it because there's real benefits and 13 they're growing in the area and if EIA is not 14 going to engage in it, EIA just gets left 15 behind. And that's been the story of the 16 last 10 years. 17 Okay. Can we do this? Maybe 18 that's why we've never done it. Well, yes, 19 we can. We do have the legal authority to 20 engage in it, because if we didn't, end of 21 the presentation. 22 This is what we're relying on in BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 26 1 our negotiation with the Census Bureau. They 2 require legal authority. 3 All right. Also, expanding 4 researcher access promotes EIA's mission. 5 Now EIA's mission statement is kind of 6 boring, but if you step back and you look at 7 it as a trilogy within a trilogy, it's a 8 really a great mission statement, because why 9 are we here as an agency? Well, to provide 10 the data, the forecasting work, and analysis. 11 But so why do we provide that? 12 Well, we want to promote sound policy making, 13 operating efficient markets, and a public 14 understanding. Public understanding of what? 15 Well, first of all energy. We want to 16 promote an understanding of energy -- 17 energy's interaction with the economy; 18 energy's interaction with the environment. 19 Now, when I see words like 20 interaction with the economy, interaction 21 with the environment, I'm thinking data 22 linkage. Right? That's kind of the analysis BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 27 1 that provides for that kind of work. You 2 want to link the files and really get some 3 powerful research done. 4 Now how's that going to benefit 5 this program, researcher access? Right here. 6 Promote sound policy making and public 7 understanding, two big areas we can get a 8 boost from. 9 Got some outside support for this 10 activity, too. The External Study Team 11 issued the report last year, 2006, and 12 laudable goal. They recommended that EIA 13 increase its level of interaction with the 14 research community to a level that is on par 15 with EIA's level of interaction with the 16 energy industry. Good luck. Right. Right. 17 I agree, because it's wonderful we have these 18 energy conferences. It's the only time 19 really researchers come here on site and we 20 have this kind of candid interaction. 21 The only other interaction you guys 22 doing is presentations at JSM in August, a BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 28 1 minor showing at American Public data users 2 conference, but this really is a core 3 activity for us. Compare it to EIA's 4 interaction at energy conferences. Okay. So 5 it's going to be a long time, and I don't 6 think researcher access is going to balance 7 the equation, but it's a step in the right 8 direction. 9 But this quote I felt was really 10 useful. Later on in the report, they write, 11 ideas to promote this goal, right up above, 12 include providing a program of visiting 13 scholars to bring academic researchers to 14 EIA. 15 Well, that's what Ruey-Pyng was 16 talking about yesterday. And expanding 17 researcher access to microdata made under 18 appropriate safeguards to protect the 19 confidentiality of the reporting entities. 20 No one's looking to undermine our data 21 protection pledges. No one's looking to 22 throw that away, and no one's looking to BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 29 1 undermine the goodwill and trust we have 2 built up over the last 30 years. 3 We have the resources to maintain 4 those data safeguards and yet permit this 5 activity, and they were highlighting it. 6 Yes, we recognize the importance of 7 preserving that trust and goodwill with our 8 respondent community. 9 Okay. So let' go to the three 10 options and wrap up. 11 The first option that research data 12 center at Census Bureau. A hundred thousand 13 dollars to start up; $75,000 annually. Well, 14 I just want to brag, because a (inaudible). 15 Well, we started off the negotiations with 16 $250,000 up front and a $125,000 a year, so 17 we did the best we could, but that's where 18 the Census Bureau has drawn the line in the 19 sand for offering their services. 20 It's off site. It's a full service 21 operation here. They do everything from, you 22 know, receive the proposals, monitor the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 30 1 employee when they're working there, do the 2 disclosure review, do the product review. 3 Okay. 4 As I mentioned, we already have the 5 Manufacturing Energy Consumption Survey data 6 there at the same location. Now, I was 7 talking with Dan Weinberg (ph.), who is the 8 Director of the Center for Economic Studies, 9 and he is in charge of the RDC program at 10 Census Bureau. And I said, you know, can't 11 you come down below a hundred grand because 12 quite frankly you got competition. You got 13 competition, buddy. You could lose our 14 business if you can't come down below a 15 hundred. 16 And so Dan reminds me, he says, 17 Jake, I serve on the Board of Directors for 18 NORC, and I'm helping them build their data 19 enclave. We do not view this competition at 20 all. In fact, I want to help my colleagues 21 develop their program. Okay. 22 You know why? Right here. Right BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 31 1 here. He said, Jake, Census Bureau is still 2 and will always probably have the largest 3 register of business data available. 4 So if you're a researcher and 5 you've got some interesting topic, most 6 likely this is the place that you're going to 7 need to go. So Dan's not worried about 8 competition. 9 Now we got some downsides. When he 10 was (off mike) working at RDC, you got to 11 submit a proposal to the Census Bureau. It 12 goes to their Disclosure Review Board; a copy 13 would come over to us. You got about a month 14 or two, probably two months time lag there. 15 Once you get the proposal approved, then you 16 get -- go in to getting special sworn status. 17 They're not going to waste time doing a 18 background check if your proposal is not 19 accepted. Right? This takes six months, the 20 FBI background check. 21 What are we talking about? It 22 could take (off mike) away, just like eight BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 32 1 months, eight months right down there, and 2 it's not for free. Okay. We're paying a 3 hundred grand. Researchers got pay $3,750 a 4 month. That's about the average cost of a 5 researcher. On top of it is $18,500. The 6 National Center for Health Statistics has a 7 similar fee, about $18,000 per researcher to 8 use that facility. 9 So -- it's plus or minus in there. 10 Okay. Ruey-Pyng talked about this from the 11 National Institute of Statistical Sciences -- 12 $10,000. Great program. Nice deep 13 affiliations than this has with its other 14 universities. They've got -- it would be 15 done here on site, so EIA needs to provide a 16 work area, dedicated PC, some staff to (off 17 mike). Selected research topics, so it may 18 involve other survey data besides RECS and 19 CBECS data. That's pretty appealing. Also 20 you got a list of selected research topics 21 that, you know, we can channel out for (off 22 mike). But it goes to a selected group of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 33 1 researchers, a selected group of researchers. 2 How many people in this room got 3 their Ph.D. after the year 2000? That's what 4 I'm talking about. See. That's right. See, 5 you know, if you woke up this morning, and 6 said, you know, I'm getting kind of old to do 7 a lot of things in life. When you came here 8 to this meeting, I bet you never thought 9 you're too obsolete to do any research on 10 company-level data at EIA, but, in fact, 11 you're all discriminated against. You're 12 out. You're axed. You're off the list. 13 All right. This goes to grad 14 students; right? That's why we have selected 15 topics, because that's what grad students are 16 for. They, you know, they like to do what 17 they're told. Give them a topic, and it may 18 work. But it's a great program, and it's a 19 nice little module that adds on. 20 Okay. How do we control that? The 21 researcher is an agent of EIA. It's 22 important. I'll be pointing this out in each BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 34 1 one of my options, because you have to look 2 at what control mechanisms are in place to 3 make sure there is no diminishment of the 4 confidentiality pledge. What safeguards are 5 in place? What rules, remedies, you know, 6 regs do you have? 7 DR. NEERCHAL: Jacob, just a 8 clarification. 9 DR. BOURNAZIAN: Mm hmm. 10 DR. NEERCHAL: The (off mike) what 11 this option means or include what you're 12 going to (off mike), at that time (off mike) 13 just a list. 14 DR. KIRKENDALL: (off mike). 15 DR. BOURNAZIAN: Right. How much 16 do grad students make; right? I'm just 17 joking. No. 18 DR. NEERCHAL: (off mike) 19 DR. BOURNAZIAN: No, that works 20 through NISS. Right. There is a payment, 21 right, that goes with that. 22 DR. NEERCHAL: (off mike) BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 35 1 DR. BOURNAZIAN: Right. 2 DR. NEERCHAL: (off mike) 3 DR. BOURNAZIAN: Good point, 4 Nagaraj. In fact, you might as well say $10K 5 plus, $10K plus. 6 Last option is from the National 7 Opinion Research Council. Annual fees 50,000 8 plus. Why? Because they're offering a 9 portfolio approach. They're saying, okay, we 10 got this remote access feature in which 11 researchers can use it, but what other 12 services do you need as an agency? You need 13 disclosure protection training. You need 14 file work -- coordinating with the 15 researcher, working with him on that. We can 16 give them training. How about archival 17 services? You got files from 10 years ago 18 and your documentation for that and link them 19 up. All right. Is there any archival work 20 that needs to be done? 21 Typical private sector consultant. 22 They do it cost plus; right? But it allows BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 36 1 you to choose your boutique selection of 2 services you want. But the baseline $50,000. 3 The researcher is also going to get 4 charged. I think it was about, (off mike), 5 but it was somewhere around $100 a week. 6 They were going to do it in weekly units of 7 measurement, being that when you offer it for 8 free to researcher, it's not valued as much 9 if they maybe have to pay a de minimis fee. 10 So they wanted something applied (off mike) 11 to them. 12 Oh, they do have two off-site 13 facilities -- Chicago and D.C. 14 Now, we've got to apply those data 15 safeguards, right, because we don't want to 16 change the playing field. We keep the level 17 of risk the same. 18 And so you have monitored -- (off 19 mike) argues buffered remote access. 20 We're going to apply some filters 21 to the queries and have some queries 22 restricted. Everything else at the default BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 37 1 is let the query go to the data. 2 Okay. Now, a lot of authentic -- 3 you have two-way authentication for the IP 4 addresses. Not only is the researcher 5 approved by NORC to log into the system 6 through that VPN network, there's an 7 authentication data safeguard for the IP 8 address. 9 And like I said, they have a 10 features (off mike), you can't download (off 11 mike) through the system. It's like your own 12 private VPN network (off mike) research. 13 The researcher now becomes a 14 subcontractor or agent. (off mike) the 15 contractor. Now, oh, also NASS, ERS has 16 signed up and they're putting their data 17 there. They're using a -- working as a 18 subcontractor, but EIA can go either way -- 19 subcontractor or agent. 20 Okay. I showed you some cost data. 21 That was the first bullet on every slide. I 22 just want to summarize that. NISS is coming BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 38 1 in at $10,000; NORC $10,000 plus. 2 Everybody's plus. NORC $50,000 plus except 3 the Census Bureau. I have it at $75,000, but 4 I didn't tell you it's a $100,000 in the 5 first year. They had some start-up costs 6 getting a -- purchase additional equipment 7 and service to host our data, but you got to 8 say if we engage in this activity, it's not 9 for a year or two. It's a commitment to this 10 activity. And so this really would be the 11 recurring annual cost to the agency who did 12 that. 13 You got -- (off mike) do some work. 14 It's not something we can just jump into 15 outside of the NISS program, which, like I 16 said, a nice contained module (off mike) 17 organization. 18 We got to document those files. A 19 lot of our file are (off mike) on shelves 20 (off mike) the documentation is pretty 21 sparse. If you want a researcher to work 22 with your files, they got to have the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 39 1 documentation to tell them what the file 2 structure is. 3 Okay. Researcher training has to 4 go along with this or else the research is 5 going to get bogged down real quickly. 6 Disclosure protection training can 7 be modules, a training module, but that has 8 to be addressed. You got to let the 9 researcher know how you treat confidentiality 10 at your agency, how it applies to you. 11 Cyberinfrastructure work. I've 12 told you how NORC is establishing their data 13 enclave, but, you know, every time you think 14 you got something set up and your IT people 15 have a solution for you, there's some hacker 16 out there waiting to unlock the key. It just 17 happens. It's the nature of this business 18 and in cybersecurity. EIA has received -- it 19 receives a lot of cyberattacks on a daily 20 basis (off mike) 300 in the first three 21 months of this year. But we're also getting 22 what they call this root matrices where the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 40 1 hacker comes in and places a false IP address 2 on an employee's PC. They don't know it, and 3 then can either steal data or try to corrupt 4 the database and through that the employee's 5 PC. We've had about 10 attacks on that. 6 So maybe that dual IP 7 authentication that NORC is proposing may not 8 work. What do they have? 9 They have solutions to that, too, 10 but once again more cyberinfrastructure would 11 probably be needed to put in place the remote 12 access researcher program. 13 Resource constraints. You got to 14 be looking at money and where the funding is 15 going to come from to do that. 16 Okay. So let's summarize and turn 17 it over to you Committee members (off mike) 18 want to get some input. We put on some 19 questions. 20 On these three options that I've 21 just been talking about this morning, which 22 modes of researcher access should EIA provide BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 41 1 and be providing or getting into developing? 2 Which mode of access works best with the 3 research community? (off mike) not only the 4 ability to do the research, but the type of 5 research. You remember that trilogy within 6 our mission statement regarding energy, 7 energy's interaction with the economy, 8 energy's interaction with the environment. 9 What kind of research and not only the 10 substantive areas, but also statistical 11 methodology. 12 What kind of research would people 13 be interested in, and if they had access to 14 company- level files here at EIA. 15 And then what I'd like what I'd 16 like to hear from -- if anyone is engaged in 17 accessing microdata from federal statistical 18 agency, a nice war story, your anecdote that 19 sets, laid out what you found useful or 20 beneficial or what you found as an obstacle 21 or a difficulty so that maybe EIA wouldn't 22 have to repeat that same mistake and then BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 42 1 develop once again an effective researcher 2 access program. 3 So in summarizing, I turn it back 4 to you Nagaraj, our Chair, for comments. 5 DR. NEERCHAL: Thank you. Thank 6 you, Jacob. 7 I will go to Moshe for the first 8 (off mike) discussion. 9 MR. FEDER: First of all, I wanted 10 to say great presentation. In fact, we have 11 two great ideas the last two days. Ruey-Pyng 12 spoke about the N-I-S-S plan, and Jake spoke 13 about -- and I think both are great ideas and 14 I'm looking forward to hearing more about 15 (off mike). 16 I just had one question before you 17 went to summary. Since NISS and the option 18 is relatively very cheap, why not do that and 19 consider yet another option? Is there 20 anything in the contract that would be signed 21 that would preclude the other from being 22 done? BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 43 1 DR. BOURNAZIAN: No. No, in fact, 2 that is one of the options. We can do 3 multiple modes. 4 MR. FEDER: So even if I let's say 5 go with the data enclave, why not do that at 6 the NISS as well? I mean I don't -- I don't. 7 DR. BOURNAZIAN: They're not 8 mutually exclusive. 9 MR. FEDER: (off mike) Why is it so 10 cheap and -- 11 DR. KIRKENDALL: It's not -- no 12 matter -- it's actually not a competition 13 with NISS. We are going to pursue that one. 14 MR. FEDER: Yeah. 15 DR. KIRKENDALL: Because it's no 16 particular additional cost. 17 MR. FEDER: No, no, no. I meant 18 the NISS data arrangement. 19 DR. KIRKENDALL: That's part of the 20 same thing. 21 MR. FEDER: A part of it. Okay. 22 Great. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 44 1 DR. KIRKENDALL: So it's a -- we 2 are doing that and I think on that one, this 3 came from a little fellowship program we had 4 through ASA, and we actually did have a few 5 researchers to come through that program that 6 worked here at EIA on site. They 7 particularly accessed the Electric Power 8 data, and they thought that it worked really 9 well for them. And, you know, agreeable to 10 do some work. 11 Electric power thought it was good, 12 too. 13 MR. FEDER: I remember seeing Joe 14 Guspid (ph.) here. Was he working (off 15 mike)? 16 DR. KIRKENDALL: No, Joe wasn't. 17 But anyhow, that's the kind of program we 18 were thinking about where the program 19 officers review the proposals and (off mike) 20 to, and because they're the ones who have to 21 provide the access and make sure the data are 22 protected according to their specifications. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 45 1 MR. FEDER: Now, Jake, you 2 mentioned this is a developing area. Indeed, 3 it is, and about a week and a half ago, I saw 4 an announcement for a new journal on data 5 confidentiality. I know one of the editors 6 is Christina, and I can send you -- I think I 7 saw Paula here. She's also an expert in this 8 area, so I'll send you guys information on 9 it. In fact, I would consider publishing a 10 paper in that journal when you make your 11 choice of the considerations that might be 12 useful, either that or (off mike), but 13 because it's a very good piece of information 14 for other agencies and government agencies 15 across the world to know what you're doing, 16 because you were very right about all the 17 benefits to EIA and to the research community 18 of giving people access to the data that 19 cannot be used otherwise because of 20 constraints, and the constraints are not 21 necessarily because your budget is not (off 22 mike) enough. It's because there's so much BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 46 1 data. Even if you had billions of dollars, 2 you would still be able to get a lot of 3 things out of the data. 4 So I think it's worthwhile to 5 consider publishing once you've (off mike) on 6 this -- on that. 7 Let's see. Now, my experience has 8 been mostly with data collected from human 9 subjects, which has particular constraints. 10 We have to get every piece of data collection 11 approved by the IRB. That's the 12 Institutional Review Board. And everything 13 that's collected has to be -- any use of that 14 data has to be approved. So if we cannot 15 even get someone to be our agent, so to 16 speak, and use the data, if that has not been 17 approved, and usually it has to be done prior 18 to collection. 19 Now, in your case, it's mostly 20 business data, but I do believe you have some 21 household-level data, so you have to check on 22 the legalities and if you can iron this out, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 47 1 again that would be worth publishing so 2 people will know what is -- what can and 3 cannot be done, I think that's one thing to 4 consider. 5 I really liked your comments about 6 the perceived risk of breaching of the 7 confidentiality pledge. It's something 8 that's been a lot on my mind because I think 9 all it takes is one case in the Washington 10 Post, let's say about an agency that released 11 information that was collected under pledge 12 of confidentiality with response rates, 13 accuracy of reporting and so on would be in 14 jeopardy. 15 So it's a good point, and I think 16 has to be kept in line. 17 So in summary, I liked -- Ed will 18 probably talk more about more substantive 19 aspects of use of the data, but I liked the 20 idea of leveraging researches, you know, 21 resource (off mike) students, whatever they 22 have to make more of the data. I think the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 48 1 first NISS option should be done regardless 2 of which other options you chose, because 3 it's cheap and probably very effective, and, 4 again, I think when you develop your approach 5 and make your own choices, I would encourage 6 you to consider publishing it in the 7 disclosure literature. 8 Yeah, that's about everything I 9 wanted to say. 10 DR. NEERCHAL: Thank you, Moshe. 11 DR. KOKKELENBERG: You know, a lot 12 of different observations, but let me give 13 you as background. I was one of the first 14 people over the falls with the longitudinal 15 establishment of data files at Census, and 16 (off mike) how many years ago that was. 17 Somebody asked me how working on that file 18 was, and after a year of working on it, I 19 said did you ever read Thomas Mann's Der 20 Zauberberg, the Magic Mountain, where time 21 all of a sudden goes extremely slowly and 22 nothing happens. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 49 1 Well, that was the Bureau of the 2 Census. I did want to underscore one of the 3 things that Moshe said right at the end about 4 the Washington Post, not that I have it in 5 for the Washington Post, but last fall, if 6 you will recall, there were some misplaced 7 laptops in various agencies, and there was a 8 great, great media uprising on that, but not 9 quite as bad as some we've seen elsewhere and 10 other times, but a lost laptop in an agency 11 that has eve 50 or 60 computers is not an 12 uncommon thing; right? 13 And so if that is going to be the 14 kind of reaction that happens, there is a 15 catastrophic risk if somebody does somehow 16 breach the confidentiality rules. So that 17 really has to be a concern. 18 Since I did this work, there was a 19 lot -- there's been a lot of developments and 20 basically because of these off-site centers 21 like the Census off-site centers, the 22 researcher needs to come to these problems BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 50 1 with an awful lot of background. I didn't 2 have it, and I learned that I had to have it, 3 and I got some of it. 4 First of all, the researcher really 5 needs to know what the underlying -- 6 understand underlying data generating process 7 is and what the questions were that were 8 asked to get this data. 9 Without that, you just got numbers. 10 And there's an awful lot of economists who 11 will take any dataset and run it through 12 their algorithm and go and say, see, I showed 13 the efficiency of or the cost function is or 14 the utility function is, and most of it is 15 not read very much deservedly. 16 This agency is really good on 17 statistical reliability. It's probably one 18 of the leaders in the area, so I think that 19 when you hand out data for whatever purpose, 20 emphasizing your understanding of the 21 reliability of the data is useful. 22 One of the big things that datasets BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 51 1 have in them at the micro level are an awful 2 lot of imputations, and they generally are 3 flagged if the agency is doing the work 4 properly. And the researcher has to 5 understand that particular aspect of things. 6 Now, the agency unfortunately is 7 going to have to have virtually a consultancy 8 to be able to help the researcher use the 9 data. 10 The researchers come in and they 11 don't know what the data -- I mean they have 12 an idea, and they spend a lot of time working 13 on a research plan. They submit it. It gets 14 approved. It has a basic understanding of 15 the problem, but it really doesn't understand 16 what data is there and how it will be used 17 and how to use it. 18 And so there's a lot of backing and 19 forthing. You can't say submit your request 20 for the variables you want or the 15 firms 21 you want for the time period that you want, 22 and we'll give you the data, and then expect BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 52 1 the person to go away fully satisfied. 2 They're going to be back saying don't you 3 have something and (off mike) so yes, and 4 they don't know what the -- the rules of 5 engagement that you lay out are going to have 6 -- be able to have them put a request in 7 advance knowing that the data exists and that 8 means they have to have access to the data 9 before they ask the question. 10 So one thing that has been proposed 11 is synthetic data sets. Okay. Are you 12 familiar with those? 13 DR. BOURNAZIAN: Yes. 14 DR. KOKKELENBERG: Okay. That 15 would be the way to go I think. If you gave 16 a synthetic dataset, which is all hokey 17 numbers, and the person -- but it's complete. 18 It shows all the variables that you really do 19 have and in the loving detail you have, and 20 then the researcher takes the synthetic 21 dataset and works on the problem, understands 22 what variables they need, what the strengths BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 53 1 and weaknesses of the models they have are, 2 but then gets access to the real live data. 3 And it's a one- shot or maybe two-shot access 4 to the data. But they're not -- 5 DR. BOURNAZIAN: So they first do 6 research on the synthetic data? 7 DR. KOKKELENBERG: On the synthetic 8 data to try to understand what the 9 limitations of the data are and what are the 10 variables that are available and their 11 strengths and weaknesses of each variable and 12 the time pass that they can look at. 13 So I understand some of the 14 research data center stuff is going to be 15 that way out at Census. 16 The other thing you're going to 17 need is you're going to need to be able merge 18 this data with other datasets, and that means 19 that this cut and paste prohibition for -- 20 might be legitimate. You've got to work your 21 way around that. But you want to match it up 22 with Census data or Ag data that's available, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 54 1 maybe also microdata, but without being able 2 to match it up, you're going to have problems 3 with that, or limitations. 4 Now, I would agree with Moshe when 5 he said that you should try multiple methods 6 of researcher access. The best way to 7 control confidentiality is only the access 8 through the agencies with computers that are 9 not connectable to the Internet and don't 10 have USB ports on them, you know, that sort 11 of thing. 12 DR. BOURNAZIAN: Mm hmm. 13 DR. KOKKELENBERG: And can't burn a 14 disk. The next best step is probably these 15 RDCs that the Census Bureau has, because 16 they've had a lot of experience with this 17 problem. 18 I don't know anything about the 19 National Opinion -- 20 DR. BOURNAZIAN: Research Council? 21 DR. KOKKELENBERG: Yeah, that -- 22 what their centers are going to be like so I BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 55 1 couldn't say anything about them. 2 DR. KIRKENDALL: That's relatively 3 new. 4 DR. KOKKELENBERG: Right. But even 5 the Research Data Centers take years to set 6 up. I know Cornell has been trying to get a 7 Research Data Center, and the best they could 8 do was get one I think in Buffalo and in one 9 in New York City, where they have access. In 10 fact, they teach a whole course on how to use 11 these Research Data Centers in the graduate 12 level. It's called Information 747, and John 13 Abowd teaches it, and I think he's a person 14 you should really strongly talk to, because 15 he's had a lot of experience working with the 16 data centers. He was Arnold Zelmer's 17 student. 18 DR. BOURNAZIAN: I know John. 19 DR. KOKKELENBERG: You do know him? 20 DR. BOURNAZIAN: Yep. 21 DR. KOKKELENBERG: Okay. He's an 22 expert in the -- he knows what BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 56 1 confidentiality laws apply and so forth, and 2 so he's the kind of guy you can (off mike) to 3 bring into have as a discussant when you get 4 to the point where you really want to make 5 some decisions on these (off mike), and get 6 some better advice. 7 Let's see. I think that's about 8 (off mike). Well, one other point: The 9 Census have one man, Jim Monahan was his name 10 who knew the dataset that they were making 11 available. He had spent years working it, so 12 it -- this takes a heavy- duty investment of 13 resources on the part of the agency to have 14 somebody who can shepherd the research or 15 into accessing the data and what's out there 16 and what isn't out there. 17 DR. BOURNAZIAN: Right. 18 DR. KOKKELENBERG: That's all. 19 DR. NEERCHAL: Neha? 20 DR. KHANNA: I have a question, one 21 question and another comment. My first 22 question is before the options that you BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 57 1 considered the three different options, does 2 it -- for any of those options does it matter 3 whether the researcher is a foreign national 4 and located in a U.S. university? 5 DR. BOURNAZIAN: Yes. If they're a 6 foreign national, they're going to have a 7 very difficult time coming on site here, and 8 then if it's Census Bureau RDC that FBI 9 background check could take a year. 10 DR. KHANNA: That's what I thought, 11 because that's a big disadvantage for an 12 onsite kind of arrangement because you're 13 losing more than half your research community 14 right there. 15 DR. KOKKELENBERG: But then you'll 16 lose it even for off-site -- 17 DR. BOURNAZIAN: Yeah. 18 DR. KOKKELENBERG: -- if they have 19 to go through this background check. 20 DR. KHANNA: Yes, but if you're 21 going -- the background check has to be done 22 anyway, whether you're coming on-site or BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 58 1 off-site, but on-site is -- and it's (off 2 mike) today, because I'm a foreign national 3 and they didn't like where I worked before. 4 I don't know. And I just -- it makes it more 5 difficult to me whether the load I think some 6 good fraction (off mike) because a foreign 7 national grad students from Georgia (off 8 mike) foreign national. That's just 9 something I wanted to put on the table 10 because not everyone in this room has to deal 11 with it as much as maybe I have to. 12 Another question that I -- another 13 question or comment that I had was I think 14 Ed's point about being able to match the data 15 with other publicly available data is really 16 important. An incubator by itself may be 17 nice, but it may not be very useful to do the 18 research project. 19 And there are two ways of doing 20 that. I mean if you went through let's say 21 the off-site VPN access route, because of the 22 fact that NORC might have let's say other BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 59 1 Census data also available that matching 2 issue may not actually be as great; whereas, 3 if you have it on-site and come and work on 4 our computer her, then I kind (off mike) the 5 Census data, and I have to -- 6 DR. BOURNAZIAN: Right. 7 DR. KHANNA: And I have to be 8 physically -- 9 DR. KIRKENDALL: I'm not sure 10 you'll be able to access Census data from 11 NORC either. 12 DR. KHANNA: No? 13 DR. FORSYTH: But you can from 14 Census. 15 DR. KIRKENDALL: But you certainly 16 could at Census. 17 DR. KHANNA: No, but if I have an 18 off- site, if I have the VPN, let's -- if I 19 have, you know, approval and I can use the 20 VPN from my computer at home or in my office, 21 and through the (off mike) can I access both 22 the Census and the EIA data? BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 60 1 DR. BOURNAZIAN: No. Right now, 2 NASS and ERS are putting their agriculture 3 data with NORC in a -- but, no, there's -- 4 they don't have the depth of the business 5 registries at the Census RDC has. 6 DR. KHANNA: But then another 7 question is: If a researcher, let's say, has 8 some publicly available Census data, let's 9 say at the Census track level or block group 10 level, something that you can just go to the 11 Census Web site and download, would it be 12 possible for the researcher to bring those 13 data to the NORC Web site? 14 DR. BOURNAZIAN: Yes. NORC 15 provides those portfolio services to a 16 researcher can bring in exogenous or 17 administrative data. 18 DR. KHANNA: Okay. 19 DR. BOURNAZIAN: And match to 20 whatever they're accessing. 21 DR. KHANNA: Okay. So that -- 22 DR. BOURNAZIAN: Enhance it. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 61 1 DR. KHANNA: Enhance it a little 2 bit. But okay. 3 DR. NEERCHAL: Moshe. 4 DR. FEDER: I wanted to comment on 5 the synthetic data. It's an appealing idea, 6 but it has a number of limitations. Number 7 one is the cost. To do it, you really need 8 to spend way more than a hundred thousand 9 dollars to develop it, but the main problem 10 is how do you do that? You hypothesize a 11 certain model. You feed the model parameters 12 from your data and then you generate data 13 from that model. 14 And the problem is if the 15 researchers were interested in validating 16 that same very model that was used to 17 generate the data, they cannot, because the 18 data were using that model. And that has 19 been raised many times. 20 So synthetic data has this inherent 21 limitation that cannot be overcome, well, 22 because maybe use some semi-parametric then BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 62 1 approach, then you would really develop some 2 methods that haven't been created yet. 3 I don't think we have been working 4 on it, but it really -- real data is real are 5 -- sorry -- real data and simulated data 6 costly and not as good. So that's why I 7 still like the idea of direct access to the 8 data, plus what Neha was talking about, being 9 able to link it with Census data, which you 10 would not be able to do with synthetic data, 11 because you need some -- 12 DR. KOKKELENBERG: The Census data 13 is only used for you to develop your research 14 proposal pertinent to the points of where you 15 know what the real data is, but that's 16 available and then use it in that sense, not 17 to fit any models necessarily but to try to 18 understand data availability. 19 DR. FEDER: I would use Census data 20 to get company variables that are not 21 available in the EIA data. 22 DR. KOKKELENBERG: Oh, yeah, well, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 63 1 that's a good point. 2 DR. FEDER: Yeah. 3 DR. KOKKELENBERG: You know one way 4 to that I would have thought of doing this is 5 let's say you have the electricity generation 6 by generator. So if you take this vector of 7 just add to it or multiply it or take it to a 8 power of a random number, and there's your 9 synthetic data. 10 DR. NEERCHAL: I think that one 11 quick point on that one I think knowing the 12 data structure, even if I don't know what the 13 numbers are in the beginning really help a 14 researcher focus their data request a lot. 15 DR. BOURNAZIAN: Absolutely. 16 DR. NEERCHAL: I think in terms of 17 that you can just basically, you're going to 18 give me a blank matrix with all the data -- 19 you know, the variables in there and what the 20 components are and so on. That will be 21 extremely useful. 22 DR. FEDER: Well, you could make BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 64 1 the code book available. 2 DR. NEERCHAL: No, no. On the 3 other hand, I think Moshe is absolutely 4 right. I think really doing -- putting 5 synthetic data out there, which is kind of, 6 you know, unbeatable is a very difficult 7 task. I think that that's a -- but I do want 8 to make one comment, you know. 9 From the presentations, the way I 10 understood is that, you know, you have two 11 important objectives. One is to address all 12 these questions, many of these questions that 13 come in and you don't have time to answer it. 14 You just say, you know, that information is 15 over there; you know, help yourself. That 16 kind of, you know, you don't need to spend 17 too much time on this. 18 The other objective is to promote 19 research on your data. I think the -- maybe 20 the first one should be the, you know, the 21 first objective. I think that maybe you want 22 to kind of prioritize it and say how can we BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 65 1 really, like, cut down some of this work that 2 we need to do periodically and not -- don't 3 really enjoy doing it. At the same time, we 4 don't want to say no. I think this is the 5 negotiation that Guy was talking about 6 yesterday. 7 I think, you know, come with 8 something that really helps you to say oh, it 9 will be very hard, we have been thinking 10 about what you would ask and we put this 11 information out there. Help yourself. So, 12 but, you know, that's about it. 13 On the other hand, promoting 14 researchers (off mike) certainly it's not 15 possible for us to sit down here and think 16 about all the things a researcher might ever 17 ask for. No matter what you put out there, 18 somebody is going to come and say, I need 19 more. Okay. I think that that's something 20 -- that is why what promoting research, you 21 really want to go the "proposal" route. If 22 someone wants data, let them write down what BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 66 1 they want. You do whatever you can to the 2 best of your available resources. 3 You may want to kind of bifurcate 4 your approach like that. 5 DR. KIRKENDALL: Actually, all of 6 these proposals involve us responding to a 7 proposal for research by a researcher. A 8 person would have to send in a proposal 9 either to NISS or to NORC or to the Census 10 Bureau and that would proposal would have to 11 be reviewed any number of different places 12 and accepted in all of those places before 13 they would actually get the access. 14 So everybody would have to look at 15 it and make the sure the research was okay, 16 according to the plan and then the people who 17 owned the data would have to be sure it was a 18 good idea, too. 19 DR. NEERCHAL: And I think we need 20 to -- we may have to (off mike) better. 21 Barbara, who invested (off mike)? 22 DR. BINGHAM: No, go ahead, because BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 67 1 I think she wanted to -- 2 DR. FORSYTH: It was, but I think 3 it might be less important (off mike) what 4 you just said, Nancy. 5 What I was going to suggest is, you 6 know, the synthetic data and giving -- 7 helping people to understand the structure of 8 the data. I liked your idea of a course for 9 users, teaching (off mike) of the structure 10 of the microdata, and you have a ready 11 outlet. You know, JPSM, for example, could 12 help you put together like a short course, a 13 two-day course, and possibly in the context 14 of that course, depending on how you 15 structure it, maybe give people some kind of 16 access to some kind of subset or limited set; 17 at least give them some experience, 18 opportunities to experience the structure of 19 the microdata. So that's what I was going to 20 suggest. 21 DR. KIRKENDALL: Actually, both of 22 these surveys already have public use BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 68 1 datasets, so we have -- it's just that 2 there's been some top coating and maybe not 3 all the variables are there -- 4 DR. FORSYTH: Right. Right. 5 DR. KIRKENDALL: But other methods 6 to (off mike) of it. That's at least a way 7 you could get started understanding the data. 8 DR. FORSYTH: Right. The 9 microdata, though, would have a somewhat 10 different structure, because it would be more 11 of it; right? 12 DR. KIRKENDALL: Yeah. It would -- 13 SPEAKER: The actual -- 14 DR. FORSYTH: Right. And the 15 matching across datasets. 16 DR. BINGHAM: So I'm curious (off 17 mike) change the subject. I'm curious about 18 what the -- I'm curious what the -- how easy 19 is a breach of confidentiality insofar as 20 when you're looking -- so what is 21 confidentiality in this case? I mean that's 22 the first question and does that mean I'm BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 69 1 able to identify a respondent's exact 2 information and lots of detailed information 3 about them and is that bad? 4 DR. KIRKENDALL: Well, we tell them 5 we're not going to give that data out, and 6 that is the kind of identification we want to 7 protect against, and we really don't really 8 want somebody to say that they've identified 9 it, even if they're wrong. 10 DR. BINGHAM: Right. (Laughter.) 11 DR. BINGHAM: Okay. So -- 12 DR. KIRKENDALL: And that's a 13 little harder to protect against. 14 DR. BINGHAM: So I know that you -- 15 the temptation is always there to do that; 16 right? If you're working with that level of 17 data. Okay. So this happens -- so there's a 18 lot of work that goes on with confidentiality 19 in surveys, for instance, where 20 health-related things and they collect things 21 all the way down to the county level and 22 suppose you're looking at HIV counts and BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 70 1 there are two people who live in the county; 2 right? And so the other person knows that 3 they don't have HIV, so they know who does; 4 right? 5 DR. KIRKENDALL: Mm hmm. 6 DR. BINGHAM: And so there are lots 7 of efforts to either aggregate to another 8 level or randomly switch around counties and 9 things like that and my question is any of 10 this taking place at -- you know, on this 11 microdata when you're going to send it out, 12 you know, because you know whether it's a 13 graduate student or it's another company, 14 just a guy in a company who says, you know 15 what, I know enough about this. I should be 16 able to identify these individual records 17 here, and is this fiddling? Is it pos -- 18 what's being done? 19 DR. BOURNAZIAN: Well, the -- 20 there's a lot in that area. The pledge is 21 that you're not -- there's not going to be 22 unauthorized disclosure of reported value of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 71 1 a respondent by EIA. 2 That's why in a lot of the slides 3 you're saying that this researcher is going 4 to be an agent or a subcontractor. We all 5 look at company-level data. In fact, 70 6 percent of Americans can be identified just 7 by zip code and birth date. So you've got 8 these survey records that have a lot of 9 direct identifiers, even though you can strip 10 off the name and company ID. 11 The researcher, he's interested in 12 some kind of statistical analysis, so he 13 might be in a small geographic area. He 14 knows this is a certain type of company. He 15 might not know the name, and then, yes, you 16 know, if he's well, you know, involved in 17 this research, he knows this market and he 18 knows the players in the market, and so he 19 knows really who he's looking at when he's 20 looking at the record. 21 The point is when he finishes his 22 research and he writes up his paper, is he BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 72 1 going to be talking about, you know, X, Y, Z 2 power generator, you know, or is he going to 3 be talking about this, the power generating 4 industry in this area. 5 So it still has to be statistical 6 analysis. That's what Nancy was saying. 7 Those proposals get reviewed quite 8 rigorously. What's the purpose for allowing 9 access? What do they want to study? 10 And if it has a nice statistical 11 purpose, yes, it can go forward. Researcher 12 is going to have access to it, but he can't 13 take it out. He can't take it home. And he 14 can't be publishing, you know, a 15 company-level responses in his paper. 16 But the access is there. 17 DR. KIRKENDALL: And he's signed an 18 agreement that he won't reveal the data. So 19 there has to be an agreement with the 20 researcher that they, you know, I'm sure -- 21 even though they may have looked at something 22 and they have figured out that it's company BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 73 1 X, Y, Z, he can't -- he's not supposed to 2 make use of that information. 3 DR. BOURNAZIAN: Right. 4 DR. KIRKENDALL: And he's not 5 supposed to reveal it, and, in fact, if it's 6 CIPSEA data, then he's subject to big 7 penalties if he does. 8 DR. BINGHAM: Okay. So I guess the 9 only thing -- so in that case, so, like I 10 think I said, I suspect with somebody 11 fiddling will figure something out. 12 You can also guess that if 13 somebody's doing a detailed enough micro 14 micro analysis on something via, I guess 15 somebody has to vet this in order to make 16 sure that something inadvertently doesn't get 17 published. 18 You know, I don't know to what 19 level people are using the information, what, 20 you know, like how fine down something that's 21 going to appear in a publication, how far 22 down somebody will have dug, and, if they BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 74 1 will have -- I guess it's the only thing that 2 I would -- the only question I have is okay, 3 can something -- okay, can confidentiality be 4 inadvertently breached, and) I guess it seems 5 like from the spirit of what you're saying, 6 the answer is no. But -- 7 DR. KIRKENDALL: We hope the answer 8 is no, and there are safeguards. You know, 9 the papers are reviewed -- they'd be reviewed 10 either by the Census Bureau Research Data 11 Center or-and or by the NORC folks. It would 12 also be reviewed by EIA people, and so you 13 hope that you haven't allowed something that 14 breaches confidentiality to get out. 15 And that's what the level of review 16 is about. 17 DR. BOURNAZIAN: And this has been 18 going on for many years. 19 DR. KIRKENDALL: But that doesn't 20 guarantee either, of course. 21 DR. BOURNAZIAN: But it's been 22 going on for almost 20 years, and we have not BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 75 1 found any confidentiality breaches from 2 researcher access among the federal 3 statistical agencies. There's been breaches, 4 but it hasn't come from this activity where a 5 researcher has gone in and done something 6 intentional or even foolish or deliberate. 7 They're usually -- well, they're 8 all dedicated in good faith and in those 9 reviews is your final check. And so you got 10 to say, look at the track record. Zero 11 breaches across eight, nine agencies engaging 12 in this activity over the last 20 years. 13 DR. KIRKENDALL: Well, it's bound 14 to happen like tomorrow. 15 DR. BOURNAZIAN: Right. You can 16 never say it will never happen. Right. 17 Well, we're building up to it. 18 DR. NEERCHAL: Ed, you had your -- 19 DR. KOKKELENBERG: Well, much of 20 what I was going to respond to, Derek brought 21 up and it was responded to by Nancy and Jake. 22 One thing I did want to add, though, is that BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 76 1 you do know, as a researcher, you know that 2 company's name? 3 DR. KIRKENDALL: Mm hmm. 4 DR. KOKKELENBERG: And you -- I 5 don't care whether you strip the name out or 6 not, but in the Census data, it was not -- at 7 the establishment level, I knew that company, 8 and I knew its quarterly labor bills and its 9 production, every -- you know, the details. 10 So that's available to me. All 11 these other safeguards are there, plus that 12 there are a lot of penalties under the laws 13 that enable the Census, which, you know, 14 could put you in prison if you really wanted 15 to go out and breach the confidentiality 16 arrangement. 17 And the final review of your paper 18 is before -- even with Moshe raised the issue 19 about confidential data in human-or for 20 individuals. 21 I know I've been working with 22 student- level records, and I get that paper BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 77 1 vetted by the FERPA enforcer, I guess I even 2 called her, at SUNY Binghamton before I can 3 let the paper go out the door or what have 4 you. 5 So it's the same with Census. But 6 you really want to go look over those papers 7 very carefully to make sure that I wasn't 8 revealing anything to anybody. That's all. 9 You just have to stand on the, you 10 know, trust the ethical responsibilities of 11 the individual researcher that requires the 12 -- perhaps this long vetting of the 13 researcher to do that; that the FBI does for 14 me to commit to do it. 15 DR. NEERCHAL: Thank you. Thank 16 you and the presenters, discussants and (off 17 mike). I think we break and get back here 18 around 10 maybe? We're a little bit behind, 19 but that should be okay. We need a little 20 break, I think. 21 DR. KIRKENDALL: Okay. Yeah. 22 (Recess.) BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 78 1 DR. NEERCHAL: So we are onto our 2 last plenary session. Can I just go and ask 3 any audience member who was not here before 4 to go to the mike and mention your name and 5 affiliation, please. 6 SPEAKER: You may even need to turn 7 your switch on. 8 MR. SIMBALSKI: John Simbalski. I 9 work for EIA, and I'm the NEMS residential 10 (off mike), and I'm here to listen to how 11 George is going to pick apart our work. 12 (Laughter.) 13 MS. BOEDECKER: Erin Boedecker. I 14 work on the commercial module of the NEMS 15 system. 16 MR. McDOWNEY: Preston McDowney. 17 DIA, Systems and Methods Group. 18 MR. RADICH: Tony Radich. I work 19 on the Petroleum Market Model in NEMS. 20 MR. SKELLY: Dan Skelly, EIA. 21 DR. NEERCHAL: Thank you. So the 22 last session speaker is George Lady. It's on BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 79 1 Forecast Evaluation Methodology. 2 DR. LADY: Thank you very much. I 3 very much appreciate a chance to talk about 4 my project and John is wrong. In fact, 5 without his help, a good deal of what I'm 6 going to report on today couldn't have been 7 done. So. 8 In fact, I should say in advance, 9 just to be clear, OIF has more and more taken 10 an interest and supported what I am doing and 11 by the time we're done today, you'll see my 12 point of view in terms of what I recommend is 13 going to depend very much on that. 14 The project that I'm reporting on 15 is at this point taking thought on what would 16 be a good way to evaluate the accuracy of 17 NEMS forecasts. 18 The truth is that EIA has published 19 for some years comparisons of the NEMS 20 projections with the eventual values of the 21 energy statistics, and I think they reported 22 on something like 17 or 18 of the major BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 80 1 series that they're forecasting. 2 There's a literature on this, and 3 part of the literature found that, although 4 the forecasts would often look quite 5 accurate, if you disaggregated them, for 6 example, if you took the consumption forecast 7 and re-expressed it in terms of residential, 8 commercial, transportation, and industrial 9 sector consumption that the errors would be 10 somewhat larger and then offsetting when 11 added together in the total. 12 My idea was to pursue sort of that 13 point of view and to look at the forecasts 14 and try to understand what were the major 15 influences that would lead to the differences 16 between what was forecast and what eventually 17 happened. 18 And here on the slide is the list I 19 came up with of the sorts of things that 20 would explain why, in the end, the projection 21 would not be accurate. 22 From the standpoint of what I can BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 81 1 do and have been trying to develop a point of 2 view about, I can analyze the weather. I'm 3 trying to develop a method where I can 4 organize major explanatory variables for 5 supply and demand, like price and gross 6 domestic product. And I think in the end, I 7 can evaluate structural change by comparing 8 different versions of NEMS and can detect 9 whether or not there has been a change in the 10 model logic. 11 What has been pointed to me is that 12 there are a number of other things that can 13 make a big difference, too, perhaps the most 14 important one is the fact that the data 15 definitions can change over time, and this 16 can make the forecasts appear inaccurate in 17 terms of historical values, but really the 18 world is changing underneath of the model. 19 And I think that's very important, 20 and I'm hoping really that this project 21 starts an enterprise of worrying about all of 22 the possible sources of forecast error, but BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 82 1 some of them, such as changes in data, I 2 can't speak to. I hope that a tradition 3 starts where that particular issue is taken 4 care of. 5 As far as what I'm doing, the 6 methods at stake really are three: 7 The point is to understand at the 8 level of explanation why the supply and 9 demand forecasts in NEMS are not accurate, 10 and the idea is to take the errors that are 11 found and to try to partition them across 12 such as price forecasts which are inaccurate, 13 driver forecasts, like GDP, that are 14 inaccurate, innocent things like the weather, 15 which can be different, in fact, than was 16 assumed for NEMS; and try to parse out all of 17 these influences. 18 The ideal -- and I received some 19 very nice written comments from the Committee 20 earlier in the week -- the ideal would be to 21 save each version of NEMS, to archive it, and 22 when the time comes to rerun the model using BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 83 1 actual values for assumed values and maybe 2 even reconfiguring the model logic to account 3 for changes in laws or other structural 4 changes in the energy system. 5 In fact, just to show you how old I 6 am, back in 1980, I actually, when I worked 7 here, initiated a program to archive the 8 versions of all of the models that were used. 9 At that time, one of the more, one of the 10 things that was in mind was that when a 11 forecast was published, the archive model 12 could be given to a third party, who could 13 then inspect what was the basis for the 14 agency's projections. 15 But strictly speaking, it could be 16 used in this project. The problem is that 17 it's very hard to do, and it's been well over 18 a decade I think since there has been any 19 routine archive program. It's very expensive 20 and as far as I know, there hasn't been any 21 initiative to use archives versions of the 22 model to speak to the kinds of matters that I BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 84 1 think I would like to organize. 2 But it obviously would be the best 3 thing, and that's what the reviewers say, and 4 I agree with that. But I fear that it is too 5 hard to do. 6 What I will propose today is that 7 there is a good second best that we can do; 8 that I think is a lot easier as far as 9 forecast evaluation compared to archiving the 10 model, and subject to the errors of linear 11 approximation will do probably as nearly a 12 good job. 13 And in particular it's my item two 14 here. I'm advocating now that each major 15 version of NEMS, the AEO versions would be 16 the first to come to mind, have a structure 17 of runs made which are designed to pick out 18 major influences. 19 OIF has already done this several 20 times with respect to prices, and John's 21 group, very nicely, gave me over 70 runs that 22 they did a little over a year ago where they BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 85 1 changed each fuel price one by one and just 2 in terms of the demand models were able 3 through the solution to see what difference 4 the price changes made. 5 Well, I believe this can be done in 6 general, and that what I would advocate in 7 the end is that there be some routine 8 protocol developed, some experimental design 9 agreed to, where some number of solutions are 10 created, designed in a comparative static 11 sense to pick out influences and rather than 12 archive the models, we'll archive the 13 solutions. Much, much simpler. And then a 14 statistical analysis of the solutions can be 15 used to pick out the sources of error in a 16 fashion that I'm proposing. 17 For what I did, not having, except 18 in the one case, such an inventory of 19 solutions was to use the solutions that are 20 published for the Annual Energy Outlook, 21 which I have back through 1998. 22 The reviewers quite rightly in BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 86 1 considering the statistical approach I 2 brought to those solutions worried that there 3 is what in economics is called an 4 identification problem. In other words, when 5 I was picking out the influence of price, the 6 truth is that price is an endogenous variable 7 in the model. And as a result, in general, 8 if you took market data, you wouldn't be able 9 to just run a simple least squares and 10 isolate the price effect. 11 And here is the graphic that the 12 reviewers provided demonstrating that. 13 So here is supply and demand 14 shifting around and in terms of the 15 elasticity that you want to estimate, there's 16 no way from these solutions to identify the 17 slope here, which is what you need. 18 But since NEMS is not the world, we 19 can control it more. And the truth of the 20 matter is that the principal scenarios for 21 the AEO were designed by me, in fact, in 1977 22 to show you how old I am, to speak to the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 87 1 identification problem. So specifically if 2 you consider the oil price variation cases, 3 what they are doing in principle is freezing 4 the demand configuration and then shifting 5 supply. 6 So the elasticity estimates that 7 I'll show you before we're done came from 8 these solutions pooled together. And 9 although the reviewers were quite right, 10 there's a simultaneity problem in general, I 11 believe it is very successfully spoken to by 12 the data that I pooled. 13 But my proposal number two I think 14 is even better. If in the end, which has 15 been done, we generate solution sets where 16 the variables are changed one by one in terms 17 of the supply and demand components 18 individually, then when you pool those data, 19 you will have completely avoided the 20 simultaneity problem at least in terms of the 21 first) order subject to the errors of a 22 linear approximation. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 88 1 So in terms of method, what I am 2 proposing in the end is that we -- I -- 3 somebody agree to an experimental design and 4 EIA devote the resources so that each model 5 version can be run through this drill and the 6 solution saved to then be used in later years 7 to assess the accuracy of the forecast. 8 Now for the rest of my talk, I want 9 to just go through the sorts of things that I 10 was able to do in the several months that 11 I've had a chance to work on this. All of 12 this is with -- is prospective. In other 13 words, my words my purpose was to investigate 14 how this might be done. Nothing that I will 15 provide you was intended to be, and whether 16 it was or not is certainly not definitive in 17 terms of the way in which the evaluation 18 should be conducted, but rather it's my 19 effort to try to get all of the methods in 20 hand and to show examples of how they might 21 be used. 22 Certainly, for adjusting forecasts BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 89 1 with respect to differences in the historical 2 values, weather would be an obvious influence 3 on energy consumption, so I looked at the 4 weather data that EIA keeps and found some 5 things I think probably should be attended 6 to. 7 The Annual Energy Review publishes 8 heating degree day and cooling degree day 9 time series, which are the major weather 10 variables in terms of understanding what 11 weather might have to do with e