On the limits of interpretability and the opacity of the machine relevance regime
This line of enquiry has maintained, across four texts, a single thesis: aesthetic comprehension rests upon regimes of relevance, and the relevance regime of a machine differs from that of a human. I demonstrated that this divergence is measurable (in attention), that the core of human aesthetic judgement—self-relevance—requires a self that the machine does not possess, and that what human and machine can indeed share is the process, not the meaning.
One final point remains, and it is the most uncomfortable. Throughout the series, I assumed that we could at least describe the machine's regime of relevance from the outside: to state where it looks, what it prioritises, and what it weighs. This text examines to what extent that is true. The answer is harsher than it appears: the regime of relevance of a system not only differs from our own, but is opaque even to those who possess the entire model—its weights, its activations, its internal mechanisms. The machine is not a black box only to us. It is, in a precise sense, a box that does not read itself.
This is not a mystical assertion regarding the essential inscrutability of artificial intelligence. It is a technical conclusion, supported by recent research into interpretability. I shall argue this with care to avoid the grandiloquence that the subject invites.
1. The Myth of Total Access
There is a widespread intuition, reasonable at first glance: if we have complete access to a model—every weight, every connection, every activation—then, in principle, we could know why it does what it does. The opacity of current systems would be a transient problem, resolvable once interpretability tools mature sufficiently.
Esta intuición es, en buena parte, falsa. La investigación de los últimos años lo ha mostrado desde dos frentes distintos. El primero, más antiguo, examina si los mecanismos de atención de un modelo —las partes que más naturalmente leeríamos como «dónde mira»— sirven de explicación fiable de sus decisiones. El segundo, más reciente y más profundo, examina cómo se representan internamente los conceptos, y encuentra una razón estructural por la que leer esas representaciones es intrínsecamente difícil.
I shall review both, as together they support the thesis without the need for exaggeration.
2. Attention is not Explanation
The first front was born from a concrete question in language processing: do a model's attention weights tell us which parts of the input were important for its decision?
Jain and Wallace (2019), in a work with a deliberately provocative title—Attention is not Explanation—demonstrated two uncomfortable facts. First, that attention weights often correlate weakly with other measures of feature importance: what the attention mechanism highlights does not coincide with what, by other methods, proves decisive for the output. Second, and more significantly, that it is possible to construct entirely different attention distributions that produce the exact same prediction. If two opposing attention maps yield the same result, then the map cannot be the explanation for the result. It is, at best, one of many configurations compatible with it.
Serrano and Smith (2019) reached a convergent conclusion from another angle: attention magnitudes are, at best, noisy indicators of the importance of elements for prediction. They are not devoid of relation to what matters, but they are far from being a faithful reading.
El debate no quedó cerrado en la negación total, y es importante decirlo para no caricaturizar. Wiegreffe y Pinter (2019) replicaron con un trabajo titulado, también deliberadamente, Attention is not not Explanation: argumentaron que el valor explicativo de la atención depende de qué se entienda por «explicación» y de cómo se diseñe el experimento, y que descartarla por completo era excesivo. La conclusión equilibrada del debate no es «la atención no dice nada», sino algo más fino: los mecanismos internos visibles de un modelo no equivalen automáticamente a explicaciones fieles de su relevancia causal. Pueden orientar; no pueden certificar.
A note of caution regarding scope. These three papers originate from the 2019 debate on language models and should not be mechanically extrapolated to the entire universe of contemporary systems. Yet their underlying lesson holds: seeing where an attention mechanism concentrates is not the same as knowing what weighed causally in a decision. And if this holds for attention—the most legible mechanism of all—it holds with even greater reason for the rest.
3. The Structural Reason: Superposition
The second front is more profound, as it does not examine a specific mechanism, but rather how concepts are represented within the model. And it offers a structural reason why internal reading is intrinsically difficult, not merely technically inconvenient.
El trabajo de Elhage y colaboradores (2022) sobre superposición lo formula con precisión. Los modelos profundos necesitan representar muchos más conceptos distintos que dimensiones físicas disponibles en sus capas. Para lograrlo, comprimen la información usando representaciones superpuestas: cada dimensión interna no codifica un concepto único, sino una combinación de muchos conceptos no relacionados entre sí. La consecuencia es la polisemanticidad: una misma unidad interna —una «neurona» artificial— se activa ante cosas tan dispares como una textura visual, un fragmento de código y un rasgo gramatical, sin que esa coincidencia signifique nada en términos comprensibles.
This changes the nature of the problem. If every internal unit blends multiple compressed concepts, then reading a single neuron, a single weight, or a single attention head as if it were the transparent representation of a concept is epistemologically risky. It is not that tools are lacking: it is that the information is not stored in a form that permits direct reading. The model's relevance regime is distributed, compressed, and intertwined across millions of parameters, in a form that lacks a readable centre.
La investigación en interpretabilidad ha avanzado precisamente intentando deshacer esa compresión —proyectando las activaciones a espacios donde los conceptos se separen mejor—. Es trabajo serio y prometedor. Pero incluso en su mejor versión, lo que extrae es una lista fragmentada de características aisladas, no una síntesis unificada de «qué le importó al modelo y por qué». La razón es que no hay, dentro del sistema, ningún mecanismo que realice esa síntesis. El modelo procesa; no se observa a sí mismo procesando.
4. The Box That Does Not Read Itself
Herein lies the point that gives this text its title, and it should be formulated without sliding into metaphysics.
Un sistema actual no contiene ningún componente que lea su propio régimen de relevancia. Computa una salida a partir de una entrada, ponderando internamente una enorme cantidad de factores distribuidos. Pero no hay, en ningún lugar de su arquitectura, un proceso que tome esa ponderación distribuida y la unifique en algo así como «esto es lo que he considerado importante, y por estas razones». El modelo no se lee a sí mismo porque no tiene desde dónde leerse: carece de un observador interno, de un punto de síntesis, de una instancia que integre sus propios estados en una representación de lo que está haciendo.
This carries a dual consequence for the thesis of the entire series. The first we were already aware of: the machine’s regime of relevance differs from that of the human. The second is new and more radical: that regime is opaque even when one has total access to the system. It not only fails to share our priorities; it does not present its own in any legible form, neither to us nor to itself. It diverges and, furthermore, it conceals itself—not through a desire to conceal, for it possesses none, but due to the manner in which it is constructed.
Marcar el límite del argumento para no excederlo. No afirmo que sea imposible, en principio, reconstruir desde fuera aspectos del funcionamiento de un modelo: la interpretabilidad mecanicista hace exactamente eso, con éxito creciente en problemas acotados. Afirmo algo más preciso: que no existe, hoy, acceso fiable y directo a «lo que importó» en una decisión concreta, y que hay razones estructurales —no solo instrumentales— por las que esa lectura es difícil. La caja puede ser parcialmente reconstruida desde fuera con mucho trabajo; lo que no hace es leerse a sí misma y entregarnos el resultado.
5. The Contrast with the Human Regime
The contrast with the human case illuminates what is at stake, and it is prudent to frame it with care to avoid falling into facile analogies.
When a human judges a work to be significant, an event occurs that the preceding text documented: the work enters into a relationship with their memory, their identity, and their history. Yet there is an additional trait: the human, to some extent, can account for that relationship. Not entirely—a significant portion of aesthetic judgement remains opaque even to us, and introspection is notoriously unreliable. However, there is a degree of reflective access: we can state, however imperfectly and at times erroneously, why something moved us, which memory it activated, and what we found significant about it.
I do not wish to exaggerate this contrast, because human introspection is limited and frequently confabulates reasons. But the difference in degree is real and relevant. The human possesses a point of synthesis—call it a self, call it reflective consciousness—from which their own experience is presented to them, if only partially. The model possesses this not at all. It is not that it has worse introspection than our own: it is that it has none, because it lacks the instance that would make it possible.
This closes the arc of the series. The human regime of relevance is partially accessible to itself; the machine’s is not at all. And that asymmetry is not a technical detail: it is what separates a system that selects the world from some location—a self, however imperfect it may be—from a system that weighs factors without there being anyone at home for whom that weighting signifies anything.
6. What this does not authorise us to conclude
Three closures, for the sake of rigour.
This does not authorise the conclusion that models are useless or that interpretability is a vain endeavour. On the contrary: understanding the limits of internal legibility is a condition for using these systems responsibly, above all when decisions that affect people are delegated to them. Knowing that an attention map is not a faithful explanation is valuable operational knowledge, not a lament.
This does not authorise the conclusion that opacity is eternal or metaphysical. I assert that there are structural reasons why internal reading is difficult today, and that current systems lack self-observation. I do not assert that no future system can possess forms of internal synthesis or reflective access. I describe the current state and its reasons; I do not prophesy the future.
This does not authorise the conclusion that the human is transparent to themselves. Human introspection is limited, biased, and often confabulatory. The difference with the machine is one of degree and structure—the human possesses an imperfect point of synthesis; the machine possesses none—not the difference between total transparency and total opacity. Maintaining that precision is part of the rigour that this line demands.
7. Conclusions
Three conclusions close this text and the series.
La primera, técnica. Los mecanismos internos visibles de un modelo no equivalen a explicaciones fieles de sus decisiones. La atención no es explicación; las representaciones internas están superpuestas y son polisemánticas; no hay acceso directo y fiable a «lo que importó» en una decisión concreta. La opacidad tiene razones estructurales, no solo instrumentales.
The second concerns the regime of relevance. That of the machine not only differs from the human, but is opaque even with total access to the system. It diverges and conceals itself, not by intent, but by construction. The box does not read itself because it lacks a point of synthesis from which to read itself.
The third concerns the ultimate asymmetry. The human regime of relevance is partially accessible to itself, however imperfect introspection may be; the machine regime is not at all. That asymmetry is what separates a system that selects the world from a self, from a system that weighs factors without anyone for whom that weighting signifies anything.
The box that does not read itself concludes the trajectory of this line, returning the question to its origin. Throughout five texts, I have maintained that aesthetic comprehension is a method of selecting the world, that such selection requires a self, and that the machine operates in its absence. This final text adds the missing piece: the machine not only lacks a self for which anything might matter; it also lacks a self capable of reading what it weighs within itself. Consequently, when we seek a gaze, a taste, or a judgement within the machine, we encounter only our own reflection projected onto a process that returns nothing to us. Aesthetic judgement remains, as far as the evidence allows us to assert, a matter for those who possess a world to select and who, at least in part, can account for their selection. The machine performs the former in its own blind fashion. The latter, as yet, it does not perform at all.
About the open conversation
This text concludes the series on regimes of relevance that I initiated with 'What Matters and What Is Relevant' and developed in 'Where the Machine Looks', 'The Self Effect', and 'Processes Without Meaning', within NeuroArt: Cognitive Surplus. The five texts share a single thesis: aesthetic comprehension is a method of selecting the world, it rests upon regimes of relevance, and the distance between the human and the machine regime is not one of degree but of nature.
Should anyone wish to intervene from the perspectives of model interpretability, the philosophy of mind, neuroaesthetics, or artistic practice, this notebook remains open. The line is, for the moment, closed in its initial trajectory; its natural continuation is research, and possibly an academic work that formalises it.
Sources
Jain, S., & Wallace, B. C. (2019). Attention is not Explanation. Proceedings of NAACL-HLT 2019, 3543–3556. https://doi.org/10.18653/v1/N19-1357
Serrano, S., & Smith, N. A. (2019). Is Attention Interpretable? Proceedings of ACL 2019, 2931–2951. https://doi.org/10.18653/v1/P19-1282
Wiegreffe, S., & Pinter, Y. (2019). Attention is not not Explanation. Proceedings of EMNLP-IJCNLP 2019, 11–20. https://doi.org/10.18653/v1/D19-1002
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., & Olah, C. (2022). Toy Models of Superposition. arXiv:2209.10652. https://doi.org/10.48550/arXiv.2209.10652
Esteban Ruiz, J. A. Lo que importa y lo que es relevante. Public Notebook at juanesteban.art, 2026.
Esteban Ruiz, J. A. Dónde mira la máquina. Public Notebook at juanesteban.art, 2026.
Esteban Ruiz, J. A. El efecto de sí mismo. Public Notebook at juanesteban.art, 2026.
Esteban Ruiz, J. A. Procesos sin significado. Public Notebook at juanesteban.art, 2026.
Esteban Ruiz, J. A. Art as Structural Surplus: Toward a Relational Ontology Beyond Human Authorship (V2.3). PhilArchive and Zenodo, 2026.