Friday at MIT CSAIL, Lada Adamic gave a talk on “To friend and to trust: eliciting truthful and useful ratings online”. Lada is an associate professor at the University of Michigan School of Information & Center for the Study of Complex Systems. I met her in 2009 at the SIGWEB Hypertext conference, and we did a small collaboration (together with her student Jiang Yang) when I was a software engineer at KGB. It was great to hear Lada again; she always brings fascinating examples, unexpected insights and humour into what are serious in-depth quantitative research projects.
CouchSurfing activity across the globe, from “Reputation and Reciprocity on CouchSurfing.com” by Lauterbach, Truong, Shah, and Adamic
Lada led out her talk by asking, “What happens when people rate other people?”, “What *is* a couchsurfing friend?”, “Can trust and friendship be quantified?”, and “Can friendship be interpreted as trust?” Pursuing these questions has led Lada to qualitative research for the very first time, asking people how they felt about the ratings, and how accurately they represented their sense of the actual relationship. Up to now, her work has applied quantitative methods to the ties within online networks, so it was really interesting to see how Lada complicated a lot of the assumptions we bring to analysis of networks.
Lada’s story starts during the early web, when people would link to each others’ homepages. When she was a student at Stanford in 1999, she wondered how many people made these links and what the graph looked like. She also wondered if other people were likely to make reciprocal links. At this point, Lada started to think about her friend Orkut, a person who threw lots of parties and even introduced Lada and her husband to each other. In the research she did at that time, it was clear that Orkut was at the center of social networks on the web as well as in the physical world (Orkut went on to found the social network Orkut).
Together with Eytan Adar, Lada asked what terms were leading people to link to each other at MIT and Stanford (PDF here), correlating friendships with participation in student organisations. Lada’s work often goes further than analysis to try to predict what connections are likely; she and Aytan wanted to predict who at these universities were likely to become friends . At MIT, there’s a strong dorm culture, so the fraternities were important predictors of who would become friends. At Stanford, it was often the extracurricular societies which led people to connect to each other.
Further research on Orkut’s online social network Club Nexus looked at people’s self perceptions, suggesting insights that political science students think they’re attractive, history students stay at home, and that English majors think they’re sexier than everyone else. Club Nexus allowed you to rate your friends on how trusty, nice, cool, or sexy they were. Adamic, Orkut, and colleagues then compared people’s self image to the ratings of their peers. They found that friends were able to guess someone’s qualities even if they didn’t know what qualities people used to describe themselves.
Lada is particularly interested in notions of trust. For example, her work with Jun Zhang and Mark Ackerman on Java Forum looked at “expertise networks in online communities,” and how to promote the comments of the most trusted contributors to the Java Forum (ACM DL link). More recently, Lada has been looking at the relationship between online and offline relationships and considering the impact of anonymity on trust. These issues matter to social scientists trying to think about how the Internet creates social cohesion. They also are incredibly important for companies like CouchSurfing and AirBNB, which mix online and offline social interaction.
Lada showed us a fascinating geographical graph of trust on CouchSurfing.org, showing which parts of the world are more likely to sleep on the couches of which parts of the world. She then talked about the path which leads someone to offer hospitality or take it up. Quite often people stay with someone else before they offer their own place to Internet guests. After the couchsurfing experience, both the host and the guest are invited to rate the other in terms of friendship, trust, and the outcome of the stay itself. Lada and her colleagues tried to see what it took to predict a case of couch surfing by analysing friendship, trust, and hosting graphs. They colleagues decided that “If you host me, it’s not necessarily that I hosted you, but maybe that I hosted someone who hosted someone who hosted you.”
Lada set the context of for this finding by giving a fascinating view of ratings on Amazon and Epinions, where people rate products as well as rating the reviews of other people. One of the surprising outcomes from Amazon is that there seems to be no statistical difference between the ratings of people who used their real names and people who use a pen name. People who used their real names did tend to write more posts, longer posts, and attract more followers. On Epinions, the picture was very different. Anonymous ratings on Epinions tend to be harsher and potentially more honest than non-anonymous ratings. Adamic guessed that this might be because people tend towards reciprocal ratings (just like her friends linking homepages at Stanford in 1999)– giving positive ratings to people who give you positive ratings. Epinions also allows people to “trust” each other, notifying you when someone trusts you, but hiding from you the list of people who distrust you.
Couchsurfing has a more complex system of ratings than these other systems, allowing users to specify friendship levels, specifying (privately) how much you trust someone, vouch for people, and leave positive, neutral, or negative references. Viewed from the ratings, people appear to love couchsurfing. They leave a positive reference for 87.7% of people they host, and 90.1% of those who host them. The ratio of positive to negative references is 2500 to 1!
Lada and her colleagues then asked couchsurfing participants, “why leave those references? is couchsurfing just really really fun?” It seems that many people who had a bad experience simply don’t provide a rating. When both groups do leave references, they often play a game of reference chicken, waiting for the other’s reference to calibrate the level of praise they give. Several people were particularly worried about what would happen if they left a bad reference: perhaps the other person would retaliate. Lada points out that despite these trends “it’s not all about tit for tat.” Sometimes we’re concerned for other people’s reputation and don’t want to ruin a third party’s experience just because our own encounter didn’t go well.
The difference between public and private ratings seems important. Public friendship ratings are more highly correlated than private trust ratings. There might be two explanations: perhaps public friendship ratings are aligned because they’re public. Or perhaps friendship is simply different than trust. When they interviewed participants, some people did mention an awareness of how public the ratings are. Despite this, many people hadn’t even bothered to see how the other person rated them.
Lada asked Couchsurfers a fascinating question: which takes more time to cultivate? Trust or friendship? Interviewees suggested that friendship is much more dependent on the duration of a relationship than trust. This actually was borne out in the data; friendship ratings were much more correlated to duration of the relationship than trust ratings.
Another idea in Couchsurfing is the idea of “vouching” for other people. Adamic and her researchers wondered if it would be possible to use the friendship, trust, and vouch information to predict who was most likely to vouch for other people. They found that the vouch data by itself was a bad predictor for vouches. Calculations on how many friends people had in common also couldn’t predict a vouch. Friendships were the best predictor of vouches, but it was “couchsurfing friendship” rather than friendships based on actual meetings.
Multiplicity of rating and trust types, and the differences of behaviour across those ratings have led Lada to question the solid notions we tend to bring to network graph data. In her talk, she asked, “are even truthful ratings reliable?” If people are rating honestly, what other demographic factors might tilt the ratings? They found that men on CouchSurfing will rate men and women equally. Women however tend to rate other women highly. This may result from women being more concerned about safety, but other studies tend to suggest that women tend to focus attention to other women within social networks. They also found that both friendship and trust for one’s countrymen is higher than for foreigners. They also found rating differences across cultures. Brazilians, for example, tend to rate each other less highly than Americans would rate other Americans. Across countries, cultural differences may not be as significant. In interviews, Europeans pointed their fingers at Americans and complained that Americans were more overtly friendly than Europeans. Despite this, the data didn’t show much difference between Europeans and Americans in actual trust ratings, although Americans tended to have more friends. Between cultures, there is also an issue of software localisation. Poor translations often led non-English speakers to be uncertain about the differences between friendship options.
To further complicate things, Lada described this fascinating practice of giving someone a good numerical rating and positive comments while encoding subtle messages to warn other users. For example, saying someone is warm, hospititable, and a good host can often be a coded message to others that they’re boring.
Since I come from the humanities, I was fascinated by Lada’s turn to qualitative research and her movement towards complicating the notions that we bring to our data analysis. I also loved the way that her talk brought out subtle differences that arise from the labels we apply to design features, as well as differences in the translation of those labels. I’m not sure she’s planning to go further down that road, but I think that further qualitative/quantitative collaborations on the nature of online ties offer possibilities for very powerful research.
These subtle design issues are incredibly important to the nature of public discourse in society. As CouchSurfing demonstrates, they can also shape how we share our property and relate to people across the world. Later on Friday, Lada and I snuck out for dim sum. In our wide-ranging conversation, we found ourselves drawn to discussions about social networking, civil society, and politics. We talked about PolicyMic, an American policy debate site which Chris Altcheck showed at the Center for Civic Media on Thursday. PolicyMic is a social network for political debate which allows people to rate the quality of comments as well as label people as “allies” and “foes.” One of the big questions for PolicyMic is whether its users are going to change their minds about politics, whether they will discover new political possibilities, or whether they’re going to become more entrenched in existing beliefs– and how adding connections like “foes” shapes those possibilities. As new forms of political discourse continue to grow online, it’s clear that analysis like Lada’s work on CouchSurfing will be an important part of understanding how our design choices shape societal possibilities.
Update: 20 Nov. Lada has sent me her slides from the talk. Here they are: