Qualitative Literacy: A Guide to Evaluating Ethnographic and Interview Research

“Qualitative Literacy” is an incredibly insightful book that should be on every qualitative researcher’s shelf. I feel like it is the true contender for the kinds of next-generation theory and method guidelines that are being thrown around a lot these days. For qualitative researchers that have read their fair share of textbooks on participant observation, ethnographic interviews, and even qualitative data analysis techniques, Small and Calarco’s book may be the logical next step.

The book explores the challenges of assessing the quality of data collected through qualitative research methods, specifically in-depth interviewing and participant observation. In open and pluralistic disciplines, each method should be evaluated based on its intended purpose and provide criteria for effective execution. The authors argue that there is little disagreement among experts on quality in craft, despite differences in approach and design. Such clear standards are of crucial importance for contributing to a cumulative social science. In the book Small and Calarco then present five indicators to distinguish empirically well-executed from poorly executed data collection.

The first indicator is cognitive empathy.

“Though the difference between empathy and sympathy may seem obvious, many published studies, and media accounts, use the former but mean the latter. In fact, as we discuss later, many empirical studies are ineffective because they communicate more sympathy than empathy, a feeling of solidarity with the position of those studied but not quite an understanding of how they see the world or why.”

Cognitive empathy allows researchers to understand the perspective of those they interview or observe. However, achieving cognitive empathy can be time-consuming and requires effective questioning. Limitations include the researcher’s own perception and the superficial understanding of what is being observed. To uncover meaning, interactions with spaces and people can provide valuable clues.

The second indicator is heterogeneity.

“The qualitative researcher’s most important asset is proximity to the world, and we know the world is diverse and messy. Good field-workers cannot help but uncover this diversity and to report what they have uncovered.”

The sensitivity of a narrative can be detected through high heterogeneity, which is the degree to which the perceptions, experiences, motivations, and other aspects of the population or context studied are represented as diverse. Researchers must take care to probe interviewees enough to capture this heterogeneity, as people’s experiences surrounding any issue or topic can vary dramatically over their lifetimes or across situations. Many researchers ignore heterogeneity in their data to present a bottom-line story, exclude cases that do not fit a central point, or ignore heterogeneity to meet journal word limits. Such practices undermine the empirical grounding of a qualitative study. Good field-workers report the diversity they uncover in their research.

The third indicator is palpability.

“Ultimately, only reports of distinct persons, statements, perceptions, meanings, motivations, events, actions, responses, and places have a chance of constituting palpable evidence.”

Palpability is the extent to which the reported qualitative evidence is presented concretely rather than abstractly. Palpable evidence results from concrete data that got close to the phenomenon at hand by centring on particular events, persons, utterances, interactions, or other pieces of elicited or observed data. Palpability in evidence is the foundation of an empirically convincing text. Lengthy interview quotes and detailed descriptions are not unnecessary flourishes but are necessary to collect more data and provide concrete evidence.

The third indicator is follow-up.

“following up often leads to an entirely new research question. This tendency—to follow up effectively, whether within or outside the field—is a reason that many effective ethnographies report that the question the book answered is not the one with which it began. This change is not a flaw; it is one sign of an effective researcher.”

Follow-up is crucial in qualitative research as it allows for the collection of additional data that may not have been anticipated during initial data collection. Often during qualitative data collection, opportunities arise for new questions to be asked and additional observations to be made. This is particularly important in interviews and ethnographies where data is co-created by the researcher and participants. Saturation is a common way to achieve follow-up, and effective follow-up can lead to a change in the original research question, ultimately leading to true scientific discovery.

The third indicator is self-awareness. Self-awareness is crucial in empirical fieldwork as researchers’ backgrounds inevitably shape how one perceives the world. It refers to the extent to which researchers understand the impact they have on the data collected.

“In an important sense, it is thus impossible for the interviewer or participant observer to capture thoughts or behavior unobtrusively. Yet the point is not that measuring behavior is doomed to failure. It is that the researcher aware of this fact reflects it in how they collect and interpret data.”

Interpretation of data can vary among researchers and their identities play an important role in data collection; what data they see and what they do not see. Unfortunately, lack of self-awareness is a common issue. For example, researcher rarely disclose their own ethnic background, which can hinder thoughtful discussions on the role of the researcher’s background in access.

Small and Calarco make a strong case that, with enough exposure, two researchers can arrive at the same social facts despite differences in methods. Exposure is the precondition of strong qualitative research. Only after sufficient exposure should one evaluate a qualitative research project based on the five indicators of quality: cognitive empathy, heterogeneity, palpability, follow-up, and self-awareness. Although providing some kind of evaluative measures, Qualitative Literacy is fundamentally an invitation to pluralistic qualitative research. Qualitative researchers should evaluate their methods primarily based on what they are designed to do. In other words, no method can be reasonably critiqued on the basis of something for which it was not intended.