This is a liveblog account contributed to by Christina Wilson, Becky Michelson, and Sarah Zaidan at the Boston Civic Media Metrics & Methods Conference.
(Crossposted to the Engagement Lab blog)
What are the benefits of quantitative versus qualitative research? Are there effective mixed-methods approaches for evaluation? What standards and level of rigor are appropriate for applied research? What are the methods used to demonstrate value? This session will feature case studies from scholars who practice a broad range of research styles.
Moderator: Catherine D’Ignazio
Presenters: Catherine D’Ignazio, Kate Krontiris, Sarah Wiliams, Matthew Battles, Cris Magliozzi, Dietmar Offenhuber, Justeen Hyde, Don Blair
Introduction
Catherine welcomes the group and introduces the panelists and herself. Boston’s vibrant tech sector, civic environment, scholarly community, and array of community organizers make this a valuable opportunity to bring together various fields, traditions and methods of inquiry about the design of materials for inquiry and collective civic benefit. Each of these groups has different methods for describing and understanding the world, and different audiences that they address. The tension in this panel is to showcase the variety of civic methods among the panelist:participatory art , ethnography , qualitative data , and citizen science. How can we start to coordinate our various efforts in a deliberate and intentional ways to make our cities better: from implementation, to evaluation, to reporting to the public? This conference is the kick-off to the Boston Civic Media Consortium that will begin to give us the chance in Boston to keep exploring these questions over the next three years.
Given the proliferation of methods, 1) what should our goals for civic engagement be? (Catherine’s emphasis is on the biggest tent possible. But then, is that too fuzzy?) And then, 2) who is our research for?
Catherine D’Ignazio
Participatory Art + Civic Goals – Art can be a research method for activating civic imagination and mentions examples like Steve Lambert’s work. To explore this, Catherine introduces theInstitute for Infinitely Small Things (IIST), and a project they did in Cambridge to explore overlooked or ignored aspects of everyday life. “The City Formerly Known as Cambridge” was a particular project, looking at the names of public places in Cambridge and the dominance of Anglo-Saxon names commemorated on public squares, streets and monuments around Harvard Square. (Portuguese Alliance lobbied for a very long time to get Cardinal Medeiros Street named – then misspelled when the sign was put up!) What would the city look like if the present-day community could rename the city with names that mattered to them? IIST hosted “Renaming Parties” in partnership with co-current events around Cambridge to engage the public in conversation about why things were named as they were, and then to submit a new name. Much of the project was about the very construction of the artifice in public to have these conversations. Each person gave their input, story, and address to connect after the new map was created. At the end of two years, an artifact was produced, “A Map of the City Formerly Known as Cambridge.” Some renamed locations quite personally – for their dog, a family member – but others submitted named related to how spaces are currently used, perhaps to claim space, e.g., Cambridge Common (Freaks), a school that used a grassy common for grazing cows, and then some referenced historical events / business. Some organized groups were in the process of officially changing the name of certain streets, as well. Everyone who submitted a card, submitted a story, and each of these were recorded as references points on the map. This is just one example of how art can be used to activate civic imagination. Even though these names did not officially change, it nevertheless allowed participants to feel a sense of empowerment in collective visioning.
- A Map of the City Formerly Known as Cambridge by The Institute for Infinitely Small Things
Kate Krontiris
Kate identifies herself outside of formal academia, in the intersection of researcher, strategist, and facilitator, exemplified in the work she has done as an applied researcher with a background in public policy and business. Her most recent project, carried out with Google’s Civic Innovation team, is interested in looking at the people in the middle“interested by-standers”who are interested in civic life, but not taking action. Her team looked at six cities across the US (San Francisco, Chicago, Atlanta, Phoenix, Chicago, Boston), conducting in depth interviews paired with conjoint quantitative surveys done in those locations. Findings at: googlepolitics.blogspot.com. Interesting paradoxes were uncovered about the way this group views their power and engagement choices in civic life.
When thinking about how to share the findings from the research: 1) inform products and services at Google related to civic life and 2) share with the broader civic tech eco-system to inform their work to activate this interested bystander segment of the population. The data is meant to help others working in this field better understand the needs and perceptions of the people who would be using the civic tech tools they create. There is a rigorous internal review process at Google when communicating research publicly, different from the academic review process and reliant on feed-back from the broader community. The work was submitted to EPIC Conference for presentation and feedback. The feedback they got was jarring, stating that the research was interesting, but not enough informed by a theoretical framework. This critique initially came as a blow to to the Kate and her research team at Google, but then forced them to ask what their original goals were: making a contribution to the applied work going on at this moment in time, not reshaping the theoretical conversation among academics.
Mixed methods approach – qualitative and quantitative – the quantitive data from surveys surveys were used to validate signals picked up in the in-depth conversations (qualitative data) and note points of divergence. Currently, she is trying to think of ways to circle back to the qualitative side to bring stories to bear on some of the quantitative findings. Anyone interested in pursuing this kind of work should reach out to Kate.
Sarah Williams
As the director of the Civic Data Design Lab Sarah’s work is informed by the question: How can we use data to effect civic policy change? She tells the story of the garment district and rezoning for current use needs. CDDL used data from tracking fashion workers to show where they were moving in the district and the areas they were most using, in order to shape the rezoning guidelines.
Another case in Nairobi looked at the data on matatus (small minibus taxis), main public transport in the city, low cost (.30 USD/ride), set by drivers. City Council data set was incomplete about where these matatus moved. CDDL leveraged the wide-spread use of cell phones to build out the dataset, using the public (GTFS). There was a concerted engagement process throughout the data collection process with the local technology community, government, academia, the Matatu Owners Association, drivers and NGOs – giving them all a sense of co-ownership in the data, so that it would be used by them. In addition to digital tools, CDDL developed a paper map that everyday citizens could use. They held a hackathon, out of which emerged the app, FlashCast Sonar, and MA3Route which relays traffic accidents to help drivers. Subsequently, there was a movement to have these maps adopted as the official maps of the city, working with local government leaders. When others leverage data created to effect their own policy change, this is what CDDL considers success. Other cities have adopted the method and to use in other cities, and other apps that use the data related to city planning, local citizens use the data and UN Habitat have been using the data and graphics in their work. Success is when people take on the data of applied research projects and continue to use in in new interventions and products.
Matthew Battles and Cris Magliozzi from MetaLab
What kinds of engagement might be possible, Matthew asks. What are the kinds of methods that drive research interests in the humanities: exploring and making accessible collections / archives; impact of technologies on these collections; looking historically at these collections at libraries. Cold Storage project has made media (short films, interactive website) and technologies to uncover the invisible infrastructure at the Harvard Library, the depository system which handles 10 million volumes off-campus, not open to the public. HarvardDirect is the digital tool meant to eventually erase the physical repository space. The depository location in the real world is 30 miles away from campus. First step was to explore the physical space of the depository with architects and designers, and the specific role of this place: temperature controlled designed to preserve books for 500 years. Film documents the activities of the librarians at the depository who do not directly participate in life at Harvard in Cambridge. The Cold Storage film was a kind of ethnography designed to bring to light these stories. Subsequently, students in the design seminar have engaged with the Cold Storage research to embark on their own projects with the Harvard depository and its staffE.g., librarybeyondthebook.org /coldstorage
Dietmar Offenhuber
Beyond Truth and Bias: a design-oriented approach to data quantification. We often take the concept of bias for granted, but Dietmar argues that this is an assumption in the process of knowledge-making that merits exploration. If we only look at data generated by volunteers in terms of “accuracy” instead of in relation to truth and bias, then we might be missing important parts of the storyfor example, the impact of the digital divide on the data, which then gets compounded by reporting and representation. The very concept of bias implies access to objective truth. Categories of “truth” / “accuracy” do not always apply to civic media. What do we mean by bias: 1) implicit assumptionsrhetoric about the inherent value of public participation. How does design mediate the conflicts that emerge in the tension between bias and stories told through data? Implicit assumptions shape design, and design shapes the conversation. Deitmar is particularly interested in how qualitative information is key to understanding quantitative datae.g., GPS data is high quantitative, but the notion of “place” is very much qualitatively constructed. In another of his projects, unearthing tone, rhetoric, rationale in the stories surrounding quantitative data points – -e.g., the stories told in See Click Fix reportinggives new insight into the politics of participation. Urban Entropy (2015) installation at Ars Electronica in Austria: if you completely change visibility and the way data is framed in public space, you can shape the conversation and change the way people interact and think about the issues.
Justeen Hyde
The Institute for Community Health (ICH) is affiliated with three Boston area hospitals with a mission to combine real-world practice with research and evaluation to create healthier communities. Staff sees itself as applied researchers across a range of disciplines from sociology, to ethnography, to fine arts. One of the things ICH is increasingly asked to do is gather more community input. There is a growing recognition of the importance of community input in community health needs assessments. However, the same people get tapped repeatedly to represent the interests and needs of the communities these multiple hospitals serve, which creates some problems. Typical methods involve one-on-one interviews, focus groups, and sometimes surveys. In order to expand the participation, ICH partnered with the Engagement Lab at Emerson College to utilize the Community PlanIt platform to engage the neighborhoods as a mechanism to engage the community in setting the top priorities for a grant-making process. The primary goal was to better understand the areas of intervention residents thought were most important to focus upon. Community PlanIt is an online game that unfolds over 3 weeks, that includes a mix of challenge questions and social media interactivity (likes, comments) for which players earn coins in the game. These coins could then be allocated by players to small projects (Causes) that would receive funding at the end of the game.
How did it work? 889 people registered, over 400 actively played. 60% of players reported no prior involvement in community planning processes, which ICH counted as a major success. ICH used a mix of analytical methods to explore the quantitative and qualitative data. The form of the game allowed for a more continued level of engagement, beyond what is typically available in focus groups or surveys. It was fun! And, people could participate according to their own schedules. However, there were some challenges. The concept of using a game for this kind of work was itself a bit esoteric and somewhat off-putting to some people. A lot of seniors and immigrant populations did not get a chance to play. A feature that ICH and Engagement Lab saw as beneficial, that the game is open access, posed some issues within the context of private hospitals, where data like this is thought of as proprietary. Such a conservative approach to community data among hospitals leads to inhibitions about sharing data, which otherwise could allow the public and other institutions to make the most of it in other ways. Next time ICH uses such a platform as Community PlanIt, Justeen would have more conversations up front with hospitals about what to do with the data, to think about the ways the data and stories can be shared and utilized more broadly in the community, beyond the internal processes which drove the primary reason for collecting of the data.
Don Blair with Public Lab
He opens by posing the question: What is the relationship of bias and truth with respect to civic engagement projects? Public Lab began out of an aerial mapping project after an oil spill, using balloons, kites and digital cameras, and images used to stitch together to make maps. Using Google Maps as a framework, images which were more detailed, higher resolution typically available from Google satellites. The project enabled the public to collect and disperse information using relatively inexpensive tools.
Now Don is moving in the direction of understanding water quality, by setting up water-quality sensors. One of the problems, is that such data is not as decodable as visual (photographic) data. There’s a need to decode data by collaborating with experts (like hydrologists). There seems to be a tension between engagement and rigor. If you set up structures to ensure scientific rigor, you may find yourself inhibiting engagement by alienating the wider public. He discussed a public maker space aimed largely at middle schools students that allows them to use the power tools in the lab unless they have training or adult supervision (“tool restrictions”). Creating such rules scaffolds the usership and opens a pathway to eventually using them, giving them permission, rather being exclusionary.
Don Blair ends with this quote from Parts and Crafts tool restrictions: “I feel a little bit defensive going over this list of tool-restrictions it feels bad to tell people that theyre not allowed to do stuff, and deep in my heart I think that kids are, in fact, generally capable of handling most of the tools in our space. But we live in a world where we dont generally give kids tools, dont generally let them play or work with hot, sharp, or rapidly moving objects. While it might, at first glance, seem ageist and condescending to prohibit kids from using certain kinds of tools, thats not really what were doing with our tool rules. In fact, calling our rules tool-restrictions misses the point entirely. In a context in which kids are generally prohibited from using things that might seem dangerous, a clear policy on what tools can be used under what conditions functions not as a set of restrictions, but an offer of permission.“
Due to the late hour, Q+A section of the panel was deferred to informal conversation during lunch. Catherine opened up a number of tensions to consider through the lunch break: scale of projects big data vs individual stories, looking at quantitative data qualitatively and vice versa; the relationship between engagement and rigor. Catherine wants the groups to keep considering the audience of the research and its impact on methodology and reporting: who is the research for?