Is Nate Silver ‘too influential’ yet?

SM: Can you explain why Pecota was seen as revolutionary?

NS: Baseball is the world’s best dataset — there’s a very rich database. The goal was to capture the full range of forecasts, which is difficult to do. Things are framed in terms of probabilities, not certainties. It does show people the work about how you get to your predictions, which I think has more value than a typical black box approach. The journey is often more interesting than the predictions.

SM: Did your parents think you were insane for leaving your consulting job?

NS: Most of my money was coming from poker at the time. In 2003, Chris Moneymaker won the poker world series. He was an ok poker player who got lucky. Poker is really easy to play if you know what hand the other person has. I would stay up all night and play $500 and then catch a cab to work and sleep at my desk. I didn’t care anymore. I wish I’d played more poker, because now the games have become very tough.

SM: So you can’t play anymore because naive players have been weeded out?

NS: Two things: the US government made it very hard to get money into online poker sites. Once the bad players (we call them “fish”) were bust out, there was no more supply of them for the predators, so to speak. You probably will see a revival of online poker because some states have liberalized their rules.

SM: You brought Pecota to Baseball Prospectus and joined the site (2003-2008). Were you writing or developing tools?

NS: I was doing that and managing the business, doing three or four things at once. It was ahead of the curve in some ways – we had a paywall early on that was successful and self-sustaining, although the competition got a lot better very quickly.
I wouldn’t find it all that attractive to work for a team, because I like to analyse data where you’re your own boss and share your findings with the public. I’ve done consulting projects where you share the data with a room of 12 people, and it’s much less satisfying than being able to disclose it to the general public.

SM: How did 538 start?

NS: In part because of the law Congress passed to ban online poker, I followed that and the representatives who took the lead on it. That got me more engaged. I was in Chicago at the time, and Barack Obama was a name around a lot at that time. It was exciting to have a candidate in the same circle. It was an exciting

In baseball the revolution happened from the outside in. Bill James was doing this stuff 20 or 30 years ago.

In politics there were campaigns who were sophisticated about this – Bush / Rove and Obama in 2004. For some reason both McCain and Romney didn’t seem to learn from that. Obama did, of course, and you also had the Dean campaign.

NS: I started writing at Daily Kos, anonymously under the name hablano (because I liked Mexican food). I was known for writing about baseball, and baseball and politics don’t intersect that well.

SM: Why?

NS: I was partly playing hooky, and I didn’t want people to reflect on my baseball perspectives.

One problem that a lot of political journalists have is that you have the same amount of column space to fill every way, but the news doesn’t flow that way. There are months and months where nothing important happens at all. But you can’t leave it blank.

SM: Why did you go public?

NS: I had people from Newsweek and others that wanted to talk to me, and I felt that if I wanted to capitalize on the interest, I had to out myself.

SM: You were coming out of a field that was fairly set in its ways – what was the reaction of political journalists to your arrival?

NS: It was friendlier at first than it became later on, before I became threatening in some way. The flipside of having no news but you need a story is the time when you have so much demand. 2008 was a year like that, a one in 20 year kind of election. It was a good time to be in that space.

The tone of the blog was more cheeky and confrontational in 2008, but I realised as I went along that I had philosophical opposition to the way that most campaigns are covered by the mainstream press. Sometimes this blogs contains explicit media criticism.

SM: In 2008 you got all 35 Senate races and all but one Congressional race (you missed Indiana). Was it clear that this was going to be a full-time job?

NS: By 2008 it had become my full-time job, although if McCain had won, it would not have been a good career choice.

People forget how successful the NYT is in digital content. I spent several months to make the decision, against several news organizations that were seen as more digitally focused. There were lots of attractive offers on the table.

SM: Who else asked you?

NS: I can’t say.

SM: Had you made the decision to join a news organization by that point?

NS: I think so. It’s hard to be both a content producer and a business person managing the brand. If you latch on to someone else, you have fewer business decisions to make. The NYT is a great way to distribute your product. A mistake that a lot of businesses make is that a developer or content producer becomes a manager, and sometimes you’re not utilizing that person as effectively as you could be.

SM: When you joined, did your reach or influence change immediately?

NS: It wasn’t an order of magnitude shift. We got 50% increase in traffic. The peaks are a lot higher, such as when something ends up on the front page.

SM: What kind of pageviews

NS: We got something like 3M uniques on election day, 2008. It ramped up – it started out with about 500 views when I began on Daily Kos, 3,000 after Newsweek; 5,000 when Palin got elected; 100k at the conventions. It has steady but not worldbeating traffic, but it goes off the charts during the election cycle.

SM: Lots of people seemed to want to pick fight with you. Places like Politico were writing pieces asking whether you were going to be a one-term celebrity. Why do you think that is?

NS: I had one Politico reporter who said ‘now that you’re hired by the Times, we can’t quote you.’ It was deliberate competition. In 2008 it was fairly clear that Obama was going to win. There were some exceptions: Martha Crowley from Fox said she thought McCain would win by one point.

We were never saying Obama is going to win for sure, but that the probability that it’s a toss up isn’t high. I think part of what made the attacks personal was that there wasn’t ambiguity, you couldn’t debate the details.

SM: There’s a difference between debating process and results. You have a good process.

NS: You can talk about people being too results oriented – and they are too results oriented. In business you want your product to be successful. I couldn’t understand why it wasn’t seen as acceptable to bet your business on an outcome. In poker it’s honorable to bet on things.

SM: So last time, did you get 50/50?

(Laughs)

NS: I’m not sure that 50/50 matters as much as getting the election. We had Obama winning the popular vote by 2%, and he won by 4%. It’s almost as big an error as saying Romney won by a quarter point. What was fortunate for me was that you had a lot of states barely leaning to Obama. So if Obama beat his polls, which he did, he would pick up a lot of states. The same wasn’t true for Romney.

SM: If Romney had won, would you doubt your system?

NS: Not really. We had a way to account for the economy – there are lots of ways to do that. By the time you get to the end of the campaign, all the model based predictions converge. It becomes an exercise in how much uncertainty there is. Earlier in the race it’s a more difficult modelling problem. I would have felt bad, but I wouldn’t think I would have to go and tear the model up.

SM: You would have felt bad because you’re an Obama supporter.

NS: The fact that Obama won the election didn’t occur me until a couple of days afterwards. I was working 100 hour weeks and you become so involved in the psychodrama – as well as doing blog posts and a book tour – so you become abstracted from the reality of the campaign. The bubble is a real thing. Journalists think that the way a column is interpreted can radically shift opinion. You want to avoid that kind of thing.

SM: Reads Jill Abramson – “half the people coming to the Times search for Nate.” She said you have your own tower. You signed a three year contract which comes up this summer, so what’s next?

NS: I don’t have any news to break. I’m in active discussions with the Times. It’s a great fit, I think Jill is a great editor. Anything can happen in negotiation but I’m happy working there. The best and worst things about it are self evident: they still have a great brand and they let their people have a strong voice of their own. The negative is that you get a level of scrutiny that is deserved in some cases, but everyone comes after the times. It’s the Yankees, and everyone hates the Yankees. The less obvious downside of that is that it’s hard to be casual at the Times. In a blog, you can say ‘here’s an observation I’ve had, I haven’t thought about this very deeply.’ It’s hard to do that at the Times. Although, for people like myself or David Carr, it is a space that cultivates having many voices.

SM: You mentioned your methodology is not as transparent as it might be in academia. What is it that has made you so popular? The voice of the blog? Is it the fact that you’re not putting these numbers out there, but doing it with voice and narrative?

NS: I spend a lot of time in my writing. I read and re-read my posts 4 or 5 times. I spend a lot of time on the graphic presentation. The Times’ graphics people are the best in the world at what they do. What we’re talking about is not that complicated. Sometimes I get into a back and forth with academics about things. How do you distill the gist of something without dumbing it down? It’s hard to do.

SM: Do you think that we are moving towards the type of predictions you do, applying those predictions to a broader subset of issues. Could that become a scientific discipline?

NS: There’s more interest in data science now. I’m suspicious of claims that there’s an inflection point in technological growth. The constraint is not the amount of information you have, it’s our ability to learn from that data. Our human constraints are a much bigger factor than the growth of data. You have problems like ‘overfitting’ where you’re mining data from the past and claiming it’s predictive, but it may just be descriptive. Is big data producing advances in prediction? Yes in some cases, but it’s a lot slower than you might think.

SM: You talk a lot about Bayesian methods.

NS: What qualifies as Bayesian is a sticky subject. I don’t mean it in the strictly academic sense. I use it to mean you do have a set of biases – the consensus up to that time. So if you’re testing a null hypothesis, for instance, if you find something that’s statistically significant but doesn’t make sense in the context of other learning, you should be suspicious. I think the idea of updating your beliefs as you move forward it Bayesian.

SM: It’s a Bayesian philosophy.

NS: Yes, there’s a frequentist technique you can apply in a Bayesian way.

SM: You had strong words for the punditocracy. What I find frustrating is columnists who assert things with no factual basis and then claim if you criticise it, they’re just exercising their right to voice an opinion.

NS: The thing about prediction, is that you’re putting your opinion on the line. I’m not just being cheeky.

Polls do fairly well under certain assumptions. They’re bad at telling us why people do things. People are bad at predicting their behaviour far in the future.

SM: The fact that Peggy Noonan is still at the WSJ. Is that the power of narrative?

NS: She’s a brilliant writer. The story I tell in my first book is that our brains are wired to build stories around essentially random data. People consume news in two different ways (according to party). I’d like to know which polls were cited by the media and how often. That would tell you something about media bias.

Q&A

Ian Condry, CMS: Is there a danger of too much accuracy and predictive power? One thing that’s frustrating is oversimplification of red and blue, focus on swing states and those recurring narratives. Is there a concern that 538 leads to a decline in participation because it’s already decided?

NS: If I feel like 538’s forecasts are influencing voting behaviour, I think that’s a very dangerous precedent. Media coverage does influence voting behaviour, e.g. Gary Thomas was essentially voted out during the primaries because the media put its finger on the scales. I’d rather have accurate information affecting public sentiment than inaccurate information. In general it’s usually the opposite error – people tend to overrate how predictable elections are.

SM: you said that if you saw 538 influencing election you might tep away form it. How would you see it influencing elections?

NS: I think it’s more of a risk in individual House campaigns. One place where it’s a risk  is when you have candidates competing for scarce resources, e.g. in the primaries. Tim Pawlenty’s people quit because they felt they were far enough in the polls that they couldn’t get donations, so it became self-fulfilling. If we saw donors trying to decide who to give money to and going to 538 to make a decision, that might be an indicator of that type of influence.

If you build a forecast model, the metric you should use is not solely about how accurate you are, but how much value you add. There are models that take what markets are saying and tweak that. But that’s essentially free-riding off existing knowledge. Trying to avoid those feedback loops is important.

John Hawkinson, EECS: What are the chief impediments to open sourcing your model?

NS: It’s not that hard to reverse engineer it. I don’t have a tenure track position, and if you open source something that value is diminish. I wouldn’t rule out doing it. If I get bored of this I could publish the source code. I think we could reach a better level of disclosure. It’s a for-profit model and that influences things.

Q: In order to forecast you have to understand the system. For congressional and parliamentary systems, do you have an opinion on better ways these systems could work, e.g. AV?

NS: In some ways the senate is odder. You’re getting more of a skew in the House of Representatives because you have Democrats concentrated in big cities. That has interesting implications, because this urban-rural divide – the setup tends to give more of a voice to rural areas. Get used to the current gridlock, I suppose.

Rachel, student at MIT: What thoughts do you have on the media’s ability to fix the problem of punditry you talked about?

NS: It’s a difficult problem. It’s fun to critique individuals, but their self interest is often in providing sensationalistic coverage. I’m not at the Times and have enough money to be magnanimous and seek out the truth. There are other people who are hyping things up because they have to meet a traffic quota. I don’t have a good answer. I do think that sometimes news orgs think too much about short term pageviews and not enough to what it says about your brand. It’s easy to compete with stupid. How can you be differentiating, is the question I would ask.

SM: Do you think that news consumers will start to play a more active role? My sense is that as consumers, we’re still functioning as we were 20 years ago – assuming that there are big media conglomerates working on similar stories.

NS: In many cases politico, for example, treat the reader as being an idiot. They assume the reader can’t see the forest for the trees. After the attacks in Benghazi, there was an arbitrary consensus that Romney’s reaction would hurt him in the campaign, but it didn’t move the polls hardly at all. The media tends to have an overly simplistic view of polls.

SM: But Politico, from a financial standpoint, is a journalistic success story.

NS: I don’t know much about their finances. One secret of being a rag targeted at beltway insiders is that you can make a lot of money from lobby organizations. You’re reaching people on the Hill who are peddling influence in lots of different ways. I think Roll Call and the National Journal do a somewhat better job with their reporting. I think the demand for data-driven coverage is also a market niche. It’s something that 20% of consumers want, and represents 5% of the coverage, so people like me are still in demand. In baseball, the quantitative market is quite saturated.

Charles, MIT undergraduate: Whatever your personal opinions about the electoral college versus the popular vote, isn’t it in your interest that the college remains because it makes your model more interesting?

The electoral college is a difficult modelling problem. You don’t assume that each state is independent from the next one. How do you strike the balance by assuming that the outcomes are statistically correlated but independent? It’s definitely more interesting having the college, it adds a layer of intrigue, yes.

SM: What if more states join?

Someone already registered 539, so I’m really against Puerto Rico getting registered.

Michael Stepner, MIT: It seems like you have the presidential prediction model down. IT seems like you enjoy building models. What is your future in technical innovations?

NS: I get bored of doing things every four years. I did a project for New York magazine rating different neighborhoods. Urban data, for lack of a better term, is another case of a rich dataset that’s not being used all that well. It’s fun to be more abstract, but I think at some point I’ll want to delve into a new datatest. That modelling on a fresh dataset is fun.

Sonny Sidhu, CMS, MIT: Could we apply your models to governance and policy? I think there’s some debates that could be richer without misleading and reductive statistics. Do we need a bureau of advanced labor statistics?

NS: Some of the people in the Treasury are very advanced in the analysis they’re doing, but trying to communicate that to the public is a different story. Ezra Klein is to some extent a model of how to do that. Polling is a simple set of facts, but those questions are much more complicated. That would be very valuable, but I think it’s a little optimistic to assume that’s what people are interested in. In the policy arena the set of facts are more ambiguous.

Ethan Wolfman, USA Today: You said people love to build stories from scant data. How do we make sure politicians don’t do this? Do you have any interest in politics?

NS: All politicians bullshit from scant evidence. The professionalisation of politics means the average tenure of a senator is 18 years or something, on average. So you get people who are more sophisticated at manipulating public opinion. To some extent politics has become too efficient for its own good, and long term problems are getting neglected.

SM: You don’t place a lot of stock in independent campaigns for president?

NS: To have someone who is that popular could be a fair overlap, and although he’s in the New York / New Jersey area he probably would need outside backing. It’s always the journalist’s dream of having an independent candidate. If he decided he wanted to run a serious campaign in 2016, he’d have a fairly good shot.

Jack Libersen, Financial Economics grad student: As more people have cell phones and not landlines, how well is traditional phone polling adjusting? Is online polling going to supplant it?

NS: Most of the credible mainstream polls do include cell phones. Rasmussen and some others don’t, and they haven’t done too well.

SM: Do you weight cell phone polls?

NS: Yes. We calculate pollster house effects (what do certain pollsters often say). The question is what do you mean by average – how well is the pollster doing lately, for instance. You had some polls conducted only online who did better than anyone else in a number of cases. You look at the graph of internet penetration vs landline penetration, those lines crossed a number of years ago. Phone polling will get more and more expensive as they have to call more cell phones.

Nick, ?: A lot of the innovations of Pecota are being used elsewhere. Do you worry about a similar thing happening to 538? Will every outlet have their own Nate Silver?

NS: Sure. There’s no doubt you’ll have people catching up. There already are. There were some people who had 98% of the same DNA of the 538 model. They had a similar model and methodology. You do encounter diminishing returns as you make your model more complex. You’re talking about fairly marginal returns. If I’m competing against everyone else’s similar models, 538 is not going to do that much better over the long term. But it’s very easy to do this really badly.

One thing I worry about is that my voice might be seen as too authoritative. It’s hard to whisper something quietly – things get blown up in a way that’s not intended. It’s hard to tiptoe into things in a way I used to.

John Truelove, MIT spouse and big data geek: The story you told is that you have hobbies that you identified as being low hanging fruit for rigorous analysis. But there are some common characteristics that these areas have. Can you give more explanation, and examples?

NS: I’m looking for areas with a medium level of predictability and quality. Cases where the culture has been against using metrics. Education might be another area. I don’t know if that would be productive.

Michael Cheung, MIT freshman: As you moved into election forecasting, what are the most surprising differences between that and baseball?

NS: Having a lot of respect for the reader. If you’re not communicating something to the reader, it’s often your fault. I’ll give a pass to quantum mechanics, but if you’re having trouble articulating an argument, maybe it’s not a very good argument.

Chris Tan: What innovations do you see and would like to see that separate a newspaper from its competitors?

NS: First, the visual presentation of materials at the NYT is outstanding. It’s super important. The reason why they insist on calling them journalists is that you’re thinking of how to convey information; you’re telling a story with data. Secondly, I think they’re very good at taking a digital first mentality about breaking news. When the pope resigned, they put the Rome correspondent’s Twitter feed as their lead – not waiting for a full story to be written. People have their individual brands within the Times, and that’s important. People follow individual writers more than the umbrella of the brand. They allow their writers more on Twitter more than organizations do.

Tom Pounds, Sloan alumnus: We could get concerned about voter fraud or suppression. How much sleep do you lose over that issue?