Anti-Americanism

Unimpressed by AJS or ASR from time to time? Wonder how an article with glaring methodological errors made it to the top, or shrug helplessly when your work is misquoted in our flagship journals? Economist John Ioannidis has an explanation.

The theory of the “winner’s curse” predicts the highest bidder in an auction has likely paid too much. The “true” value of the item being auctioned is closest to the period where the greatest number of bids are made. And so it is for journal articles, according to Ioannidis. In their eagerness to publish cutting edge research, Editors routinely sacrifice standards of replicability necessary for robust scientific findings. Because cutting-edge fields are usually smaller than other parts of the discipline, Editors are often left without a choice in the matter. Cutting edge fields are also characterized by greater flexibility in research designs, definitions, outcomes, and analytical modes, of which there is little consensus within such tiny academic tribes.

Analyzing a series of popular articles in medical journals with more than 1,000 citations, Ioannidis finds that most “breakthrough” theories are refuted within a few years time. To make matters worse, once cutting-edge theories become widely accepted, Ionnidis finds that most subsequent studies are simply reflections of the prevailing bias. Because flagship journals are read more frequently than others, these errors are seldom corrected within the mainstream of academic fields.

A counter-argument can be made for the very visibility of flagship journals creating increased imperative for replication and scholarly oversight. Still, if these critical errors are not corrected in high circulation journals, the danger persists that false findings will remain seminal references in a given literature.

So does this apply to sociology? Setting aside tired questions about the balkanization of the discipline, I do not believe Ionnidis’ logic holds for our field—or at least cultural sociology. Insofar as we are interested in uncovering the relationships between actors that shape the variables of interest for other fields, we avoid the sub rosa paradox. Yet how often are “process-models” replicated? Surely well-farmed network data has produced breakthrough findings that were later refuted with more sophisticated methods.

But what of our many qualitative studies? Here, we rarely even make our data publicly available. Nor do we replicate our instruments across time and space, leaving a long parade of graduate students to question the foundation of the research question, instead of the findings themselves. And how can we blame them? In a market with greater returns for marketability than meta-analysis, will cultural sociology go the way of cultural studies?

Obama Ignorance Crosses Party Lines

Trolling through public opinion data this morning I came across this fascinating table from a recent Pew Survey. Apparently, equal numberes of Democrats and Republicans mistakenly identified Obama as a Muslim! Further details about the complexity of American ignorance (regardless of party) can be found here. This is either: a) evidence of the pitfalls of response rates below 30%; or b) a sorely needed reason to cancel my subscription to the New Yorker. We may also have George Stephanopoulos to blame.

Instant Lit. Review

ISI Web of Science has released a beta "Citation Map" feature which graphically displays all references in a given article as well as subsequent citations of the same article. The picture to the left is a screenshot of Nina Eliasoph and Paul Lichterman's AJS article "Culture in Interaction." The left side of the ellipses contains all citations made by the authors and the right side is all articles in which they are subsequently cited. The unique feature of ISI is that one can click on any reference in the map and pull up the full citation record (and create separate maps for this citation).

Identity Goes to Work: "Bringing Coal to Coalville"

At a recent conference in the idyllic countryside of Manchester England— birthplace of the Industrial Revolution and the original meeting place of Marx and Engels— I had the pleasure of meeting Nobel Laureate George Akerlof. In an act of true academic courage, the economist presented a paper on identity to an audience of sociologists and social psychologists. Having seen the title of his talk, a colleague and I predicted a reincarnation of the rational choice model of identity. What we witnessed, however, was much more intriguing. Akerlof spent the better part of four years attending graduate courses on symbolic interactionism, ethnography, and immigration at Berkeley. The result is an elegant—if entirely unsubstantiated—model of identity (Published in the Quarterly Journal of Economics) that models the “definition of the situation” as a utility function that is not constrained by the assumption of perfect information among agents in a given system. Akerlof and co-author accomplish this by defining the situation as the product of an individual’s conception of all possible definitions—which result from interaction between individuals in a system that can be stratified according to structural dimensions such as inequality, segregation, and the like.

Akerlof humbly rejected much of the uninformed criticism of the audience—many of whom had overlooked the elegance of the model in addressing both the multidimensionality of identity and its situated/nested nature. Over dinner later that night, however, I was able to convince him that his model ignores two important dimensions of boundary-work: 1) the tendency of boundaries—or identity “dimensions” in his terminology—to take on meaning independent of interaction (as with the widespread stigmatization of race as a legitimate source of exclusion since 1945); and 2) the network theory of identity that allows one to model “identity entrepreneurship” in Harrison White’s terminology. Time permitting, I am hoping to write a brief piece with he and his co-author that adds block-modeling to their model based on simulations of “identity mutations.” In the meantime, I remain absolutely impressed by Akerlof’s self-described attempt to “bring coals to coalville,” which I believe may be more useful to those who are quick to dismiss him than they realize.

Sociologists of Knowledge, Unite!

New toys from the MIT Media Lab! This nifty picture is a "history-flow" visualization of edits to the Wikipedia entry for "Capitalism." Each parallelogram represents a unique sentence or paragraph. The 'Y' axis represents time. Therefore, the "wavy" lines represent periods where a particular claim was heavily contested by users, or-- sadly-- the U.S. government. The program uses Wikipedia "meta" data, which can be easily downloaded here.

Upon further investigation I discovered a lively literature on the social production of knowledge in Wikipedia. Needless to say, these folks couldn't spot A.N.T. in a pile of T.P.S. Reports. But should we sociologists care? After all, about 5% of Wikipedia users write 95% of the entries. Moreover, much of the edit history is coordinated by a cadre of computer scientists with an uncommon obsession with punctuation and grammar. Still, a quick glimpse at the data yielded a few promising controversies (including a four week battle about the proper definition of "terrorism."). In the end, this may be further evidence of my ongoing obsession with pretty pictures of social science.