Who Goes There?

John W. Campbell’s 1938 novella Who Goes There?, subsequently adapted for film – most famously as John Carpenter’s The Thing – begins with a question whose simplicity proves deceptive. A group of men isolated in Antarctica discovers an organism capable of absorbing and imitating other living things so completely that the familiar body, voice, and behavior of a colleague no longer establish that the entity standing before you is the colleague you remember. Campbell’s title is therefore not merely a sentry’s challenge. Who goes there? presumes that behind sufficiently organized behavior there is a who to be discovered, and in Campbell’s story that presumption is terrifyingly correct.

It is an intuition with a long and generally unhappy history. Human beings confronted by patterned calamity have repeatedly searched for the creature responsible for the pattern, particularly when the actual machinery was invisible. Epidemic disease could therefore become evidence of witchcraft, Jews accused of poisoning wells, or divine punishment; the suffering was manifestly real, its distribution seemed anything but random to those living through it, and an intentional cause supplied both an explanation and something upon which fear and anger could act. Once the agent had been discovered, the calamity had an intelligible cause and something could at least be done to him, her or it. A satisfying conclusion had been reached; the unfortunate fact that the actual machinery involved fleas, bacteria, sanitation, and trade routes would not become troublesome until much later.

Lisbon presented an especially embarrassing case because God, unlike a railroad president, was supposed to be both omnipotent and benevolent. The earthquake of 1755 arrived on All Saints’ Day, destroyed churches, killed worshippers and was followed by fire and tsunami, leaving theologians with the unenviable task of explaining why this represented competent administration. Doctor Pangloss at ground zero was not merely an optimist; he was a man so committed to intentional explanation that no conceivable observation could falsify the intention he had attributed to the system. If Lisbon burned, there had to be a reason for Lisbon to burn, and the reason had somehow to vindicate the Author of All Things. Once providence had been stipulated, evidence was reduced to the comparatively modest task of agreeing with it. Heretics therefore had to be found.

The habit did not disappear when divine providence became intellectually unfashionable. We remain remarkably willing to convert organized consequences into intentional causes, although supernatural agents have generally been replaced by sociological ones. When markets, governments, corporations, media institutions, and political systems behave with sufficient regularity, ordinary grammar encourages us to supply them with desires and purposes; once an abstraction has become the subject of an active verb, the implied teleological group mind is seldom far behind. We have secularized the demon without entirely surrendering our affection for demons.

Noam Chomsky has long irritated me at precisely this boundary, which is unfortunate because much of what he observes is real and much of his humanism I share. His institutional criticism is strongest when it remains close to machinery. Media organizations have owners and depend upon advertisers, audiences, sources, and access; journalists have careers and professional norms; governments classify information and selectively grant access to it; corporations seek advantage; universities depend upon patrons, prestige and bureaucratic survival; political organizations select people capable of flourishing within their peculiar environments. Chomsky and Edward Herman’s propaganda model is interesting precisely because these mechanisms can systematically filter what gets reported, amplified or ignored without requiring somebody to telephone the editor each morning with instructions from the ruling class.

Chomsky knows this, and has repeatedly rejected conspiracy as the necessary explanation, yet his rhetoric has an unfortunate tendency to restore the intentional agent that his machinery has just rendered superfluous. “The media,” corporate interests, ruling elites, and similarly capacious abstractions gradually acquire purposes corresponding rather neatly to the aggregate behavior under discussion. The problem is therefore not the vulgar accusation that Chomsky imagines sinister gentlemen gathering over brandy to determine tomorrow’s permissible opinions. It is more interesting than that: having constructed a distributed mechanism capable of generating regular outcomes without centralized intention, he too often speaks of those outcomes in a causal grammar that makes the mechanism sound as though it does in fact possess exactly such intention. The ghost, having been carefully expelled from the palace, is quietly admitted again through the servants’ entrance.

Consider instead a newspaper publisher seeking profit, a reporter seeking a career, an editor seeking tomorrow’s front page, an official seeking favorable coverage, an advertiser seeking customers, a politician seeking reelection, an investor seeking return, and a reader who would prefer to have his prejudices confirmed before breakfast. They need share neither an ideology nor a conspiracy, and some may cordially despise one another. Couple them through money, information, reputation, regulation, employment, and attention, however, and some patterns of behavior will reproduce themselves more readily than others. People whose behavior is persistently incompatible with the surrounding structure will tend not to occupy influential positions for very long, while successful organizations alter the environment in which their competitors and successors must operate. The result may look remarkably purposeful even though no purpose exists at the level at which we are accustomed to look for one.

Cybernetics supplies a less theatrical explanation than conspiracy, but also a more troublesome one. A system of interacting actors can exhibit attractors, states toward which its feedback structure repeatedly drives it, not because anybody necessarily wants the aggregate result but because departures from it encounter forces that tend to push the system back. Changing the occupants of important positions may therefore accomplish surprisingly little if the information flows, incentives, constraints and feedback loops surrounding those positions remain substantially unchanged. The new editor encounters the economics of the old newspaper, the new legislator encounters the constituencies and institutions of the old legislature, and the reforming executive inherits the creditors, competitors, employees, customers, and accounts of his predecessor. When the replacements eventually begin behaving distressingly like the people they replaced, betrayal and corruption are possible explanations, but they are not the only ones.

Indeed, if replacing one collection of bastards repeatedly produces another collection behaving much the same way, cataloguing bastards eventually reaches diminishing returns. Such recurrence is data. The supply of villains may indeed be inexhaustible; human history offers uncomfortable support for that proposition. But an experimentally minded observer should also consider whether the intervention has been directed at the wrong variable. If replacing the occupants repeatedly fails to change the behavior, while the institutional relationships surrounding them remain substantially intact, the persistence of the behavior begins to tell us something about those relationships.

This changes what one means by power. We commonly treat power as something possessed by an actor and exercised through an institution, whereas some power may instead be an emergent property of organization itself, residing in who can communicate with whom, where information accumulates, which signals are amplified, how resources flow, what actions receive rapid feedback, which costs are delayed or exported, and which connections may be made or severed. The occupant of a powerful node certainly matters and remains morally responsible for choices made there, but the node was powerful before he arrived and may remain powerful after he leaves. Capturing a node is not the same thing as changing the topology of power relationships.

None of this makes conspiracy unreal, nor does it dissolve culpability into the warm bath of systems theory. Watergate was a conspiracy; corporations have colluded, governments have lied, executives have concealed evidence, and powerful people occasionally do gather in rooms to arrange things that would be considerably less convenient if discussed in public. When somebody has stolen the silver, asking who benefits? remains an excellent way to shorten the investigation. The mistake lies in promoting a useful forensic question into a general theory of political economy, because benefit does not establish authorship and patterned behavior does not establish intention. Campbell’s characters are rationally paranoid because there really is a Thing hiding among them. But reality is capable of something nastier: there may be no hidden creature for the blood test to reveal.

The political consequence matters more than the epistemological nicety. If objectionable social behavior is understood principally as the work of objectionable people, then the obvious remedy is to expose them, defeat them, remove them from commanding positions, and replace them with people possessing better values. Sometimes this is both necessary and just. Yet political history also provides the peculiar spectacle of reformers acquiring yesterday’s offices and then, with depressing regularity, discovering reasons to do things for which they had denounced yesterday’s incumbents. The customary explanations are corruption, cowardice, betrayal, or capture, all of which certainly occur and all of which preserve the original model: the problem remains inside the people. A topological explanation permits a less gratifying possibility. The reformer may still be a reformer, but he has entered a structure that rewards, punishes, informs and constrains him in much the same way it did his predecessor.

The American Progressives of circa 1890–1920 are particularly interesting because, for all their abundant moral fervor and their own considerable blind spots, many of their most durable interventions proceeded from a wonderfully unsentimental understanding of this problem. They did not wait for the robber barons to experience conversion. Rockefeller could remain Rockefeller, Morgan could remain Morgan, and a railroad proprietor was under no obligation to discover universal brotherhood before his public behavior could be made less obnoxious. Instead they edited the environment around the robber barons. If discriminatory freight rates were profitable because control of a transportation bottleneck made discrimination possible, change the rules governing the bottleneck; if adulterated food was profitable because the customer could not inspect the sausage, require inspection and disclosure; if combinations could acquire enough market power to extinguish competition, alter the legal environment in which combinations were permitted to operate. The moral judgment supplied the objective, but moral conversion was not the control mechanism.

There is a delicious austerity to that conception of morality: we need not improve your soul; your rate schedule will suffice.

The older common-carrier tradition makes the principle particularly clear. Its roots reach back well before the industrial transformation, but canals and turnpikes, followed by railways, telegraphs and telephones, made increasingly obvious the peculiar power acquired by whoever controlled an unavoidable connection in a network. The durable legal response was not to search for canal proprietors of superior character. Obligations were attached to the structural position itself because control of that position affected people with whom the proprietor might otherwise claim no obligation at all. Ownership could change hands, bringing with it an entirely new assortment of virtues and vices, while the duty persisted. Long before Wiener supplied the vocabulary, the law had stumbled upon a cybernetic insight: if troublesome behavior follows reliably from a position in a network, edit the conditions surrounding the position rather than awaiting improvement in its occupant.

This distinction between moral purpose and moral mechanism seems to me important well beyond the history of regulation. Political morality easily becomes evangelical, even when entirely secular, because it imagines a better society emerging from better motives: less greedy businessmen, more public-spirited politicians, less selfish voters, more socially responsible corporations, better informed citizens. All would be welcome, but an institutional order requiring unusually virtuous participants in order to produce tolerable outcomes is badly engineered. Human beings arrive equipped with ambition, vanity, generosity, appetite, loyalty, fear, tribalism, self-interest, and occasional nobility in proportions that have resisted several millennia of improvement programs. Institutions capable of producing decent public behavior from that material have an obvious advantage over institutions whose proper operation awaits universal conversion.

The Progressive achievement, where it worked, was therefore more subtle than imposing Progressive morality upon unwilling capitalists. Moral convictions determined what consequences the reformers considered unacceptable, after which institutional design altered the environment in which private choices were made. The robber baron was still free to prefer money to brotherhood; what changed was the set of strategies by which that preference could successfully be expressed. In modern language, the fitness landscape had been edited. In cybernetic language, the feedback had been changed. In ordinary language, Rockefeller did not have to become a better man in order to be made to behave differently in public.

This is also where my irritation with Chomsky becomes more than impatience with his pontificating certainty. By repeatedly locating explanatory agency in elites, corporations, or media institutions, his rhetoric directs attention toward the occupants of powerful positions when the more consequential question may concern the machinery that makes those positions powerful and selects the behavior of whoever occupies them. Exposure and replacement remain necessary when particular people commit abuses, but neither explains why substantially similar behavior so often reappears under new management. If the attractor survives the reformer, indignation at the reformer may be morally satisfying while telling us remarkably little about the failure of the reform.

There is no counsel of political passivity in this. Quite the contrary, because if power is partly topological, then the topology needs be altered. Common-carrier obligations, antitrust constraints, disclosure requirements, liability rules, interoperability requirements, access rights, and structural separations are all attempts, successful or otherwise, to change relationships rather than personalities. They lack the theatrical satisfactions of unmasking the villain and are therefore poorly suited to melodrama, but they have the considerable virtue of continuing to operate after the hero who enacted them has gone home. A good institutional rule should be able to survive the election of someone who voted against it.

Artificial intelligence belongs near the edge of this argument rather than at its center, because it promises chiefly to make an old problem more acute. Automated recommendation, allocation, trading, pricing, filtering, and decision systems can tighten feedback loops already connecting markets, institutions and human behavior, allowing consequences to return as new inputs at speeds and scales for which institutions built around human deliberation were not designed. Aggregate behavior may consequently become less dependent upon the intentions of any particular participant while appearing ever more purposeful from outside. We shall naturally ask who programmed these systems, who owns them, who profits from them, and whose values they embody; all of which are proper questions when assigning responsibility. They simply may not tell us what the larger system is doing or why it keeps doing it. The more useful question, as Lenin once put it under rather different circumstances, is What Is to Be Done? – and my answer is to edit the environment before attempting to improve the inhabitants.

Campbell gave his Antarctic scientists the comparative luxury of a world in which paranoia was an accurate ontology. Somewhere among them there really was a Thing, an intentional creature concealed behind familiar appearances, and salvation depended upon discovering where it was hiding. Medieval Europeans confronting plague were less fortunate than the scientists at Campbell’s Antarctic station when they found their Things in witches, heretics, and poisoned wells; and the theologians confronted by Lisbon had to work still harder to preserve theirs. Modern political thought has largely dispensed with demons and providence, but it has not entirely surrendered the comfort of believing that sufficiently coherent misfortune must ultimately disclose an author.

Sometimes it does, and sometimes the blood test works: the conspiracy is exposed, the guilty party is removed, and justice requires precisely that. But when the same behavior returns after the suspected Thing has been driven from the station, and then returns again after its successor has been expelled, recurrence becomes evidence against the hypothesis we are so fond of asserting. The most frightening possibility is not that the Thing keeps getting into the station. Rather, it is that we have built a station that makes one become the Thing.

We should redesign the station.