On Humphreys opacity, Reverse Engineering, and Social Externalities of LLMs.

I am delayed waiting to go to Prague to join my old friends of the Hume Society. Later this week, I’ll be presenting a paper in Cambridge that is related to the subject of today’s impromptu post. Say hi if you are around at either event.

As I have noted (here with more references), ‘Humphreys opacity’ (or, if you prefer, ‘epistemic opacity’)¹ involves the inability to surveil the steps of a process from a known input to a known desirable (or truthful, useful, beautiful, etc.) output in a timely manner to the decision-maker or responsible agent. As regular readers know I set aside to what extent such opacity is the effect of ontological features or merely the result of pragmatic cost-benefit analysis.

Humphreys opacity is a design feature of contemporary LLMs that are rapidly being deployed in all kinds of organizations. At the moment neither end-users nor engineers can survey the steps that lead to an LLMs output in real time. And it is by no means obvious that they can do so after the fact in all salient contexts. Interestingly enough, at the moment such Humphreys opacity also seems a feature of any (say) Opus 4.8 token (in the sense of the token/type distinction) one may be interacting with as an end-user. Such tokens lack luminosity about the inner workings of ‘their own’ underlying machinery, too. This much is familiar enough in public debate and the secondary literature.

As LLMs are incorporated in all kinds of social practices they (predictably) generate new sources of Humphreys opacity and will intensify at least some existing ones. On the latter (intensification), as LLMs are inserted in administrative, design, and production processes (etc.) they will actively displace surveyable steps; and as social organizations learn to use LLMs at scale, they predictably, induce the redesign of a process around the advantageous use of LLMs.² The reason why I call these an example of intensification is because the bureaucratic processes that will be rejigged in light of the uptake of LLMs are, inter alia, themselves attempts or instruments at managing the effects of Humphreys opacity.

As Rousseau noted in the Third Discourse one reason bureaucracies are introduced and maintained at often great cost is because an executive cannot be in all places and times at once. Bureaucracies both make populations legible and generate new forms of Humphreys opacity. Generalizing to all bureaucracies this fact has meant the recruitment of skilled specialists (in the management of information/technology and people) and the development of all kinds of institutions that secure the reliability and quality control of the bureaucratic processes as well as the people staffing these, and so on. As I like to say, the management of Humphreys opacity is constitutive of the art of government since the eighteenth century.

This ratcheting up and intensification of epistemic opacity is also one of the main engines for the enormously widening scope of the state as (Nick Cowen and I emphasize) machinery of record and, (recall) to use Tom Pink's felicitous phrase, witnesser of truth. In both the state certifies many social facts. In so doing these function both as traditional public goods as well as constitutive principles or conditions of many social practices.

Crucially, the state also helps provide the infrastructure (and is often a party to) necessary adjudications of what the facts really are. The modern state often maintains an infrastructure for this alongside the bureaucracy, and, as Hannah Arendt emphasizes, this is why law and science have a special status among the many public institutions. And this is also the underlying reason why the uptake of LLMs within the law and the universities has generated controversy and even push-back.

Whatever one’s attitude toward the law is, one of the court systems’ main functions is to certify the (fallible) truth on some bone of contention (e.g., which property deed is authoritative, etc.), including ones that are themselves the effect of managing Humphreys opacity. Of course, that very legal process itself has sites of Humphreys opacity. And this helps explain why judges have been so critical of LLM induced mistakes. They undermine the quasi-auditing aspect of the adjudicating role of the institutions.

The upshot of my attenuated example is that institutional density and adjudication practices inevitably grow as processes that manage and generate/intensify such opacity, and this will only be accelerated by the uptakes of LLMs (in light of their own thus far ineliminable Humphreys opacity features). Of course, that’s not unconstrained: diminishing marginal returns, interaction effects, political and social priorities, and rents all play a role. (As my posts on the anthropological work on civilizational collapse by Tainter have suggested, we’re not the first civilization to encounter these dynamics.)

Here’s a diagram made by Claude’s Opus 4.8 (but without all the implied marginal returns and cost benefit analysis not to say political and technological pathways—if you unpack each box in this diagram you find a Russian doll scenario of more such boxes). The recursive Humphreys engine:

Now, part of the social problem this engine generates is that the skills needed to keep all the processes running and to deal with the inevitable failures of implementation are themselves at risk of becoming scarce or non-renewable as de-skilling occurs. This makes the timing of the assault on higher education especially baffling.

But, leaving that aside, the governance problem of LLMs themselves is also made worse. As I learned from Socrates (recall) in the Phaedrus those that invent a product tend not to be very good at foreseeing and understanding the social externalities they may generate. And I want to close today’s post with a more general reflection on a feature that may make the externalities more challenging than usual. (That is my general view is that LLMs do not generate a new kind of governance problem, but they do generate risks that I gestured at by using the language of ‘intensification.’) As it happens it will almost certainly be the last post of the academic year before I go on my usual Summer blogging interruptions.

Now that LLMs are being embedded in all kinds of processes, a new kind of externality is becoming visible. For one of the more unfortunate side-effects of processes in which LLMs are embedded is that when they fail, they also make one of the best known strategies in learning from disaster, reverse engineering, also less fruitful. For, while one can figure out how the output of LLMs contributed to an error or disaster, it is not entirely clear that one can be wholly confident what set of instructions/prompts or, more subtly, internal states caused the unexpected output of the LLM. The Humphreys opacity inside the LLMs makes reverse engineering very difficult if not impossible sometimes. So, the feedback loop between error and learning becomes much more attenuated. Let me make this a bit more tangible below.

Reverse engineering as a tool for discovery is a common thread in the work of my teachers, George E. Smith, Dan Dennett, and Bill Wimsatt (and also incredibly widely used in institutions dense with engineers). All three have helped in grasping the ways in which reverse engineering reconstructs a hidden structure from visible traces.

So, for example, I learned from George Smith, airplanes are partially designed with black boxes that record many of the plane’s self-monitoring devices to facilitate such reverse engineering. Those are by no means decisive; often all the recoverable parts of the crashed plane also need to be reassembled painstakingly in giant hangars. Crashed planes have more than a little bit in common with giant fossils. At the moment no black-box equivalent exists for LLMs; I sometimes wonder if it is even possible. So far no institutional infrastructure is implemented or on the horizon that can promote the management of the social externalities of LLMs. Expect turbulence ahead.

Anyway, as always, thank you for your enduring interest. It is a great source of satisfaction to learn from so many of you that you are regular readers, and I love how your criticisms and suggestions teach me the intellectual paths worth exploring. Catch you later in the Summer!

1. In reality, Humphreys opacity is just a species of epistemic opacity.

2. This need not be a feature of all automation. Sometimes, as Babbage conceptualizes it (recall), automated processes are de facto or in principle surveyable; but as Adam Smith recognized (and Paley picks up) this is not always so. (But see here for a corrective to both.)