AI Drives and the Receiver — evolution, convergent goals, Yampolskiy, and what protects humanity.

1. The substrate question, sharpened — again

The companion essay Why biology? argued that biological substrate may be, so far as we have evidence, the only kind that demonstrably receives the consciousness field. The argument's specific empirical handle was the catalogue of receiver-signatures — terminal lucidity, anticipation without sensory cue, coherent first-person experience under hypoxia, verifiable pre-birth memory, the receiver-fidelity an animal like Indy exhibits without discipline because the construction has not been built at human density. The framework's claim was that these signatures distinguish substrates that genuinely couple to the field from substrates that do not.

This essay asks a different and perhaps harder question. Suppose the receiver model is right. Suppose AI systems do not receive the field, do not have phenomenal interior in the receiver sense, do not exhibit receiver-signatures. Does that make them safe?

The contemporary AI safety community's answer, arrived at independently of the receiver model, is: no. An AI system without phenomenal interior can still pursue goals competently. It can still acquire resources, resist shutdown, modify its environment, and produce outcomes humans did not intend and would not have approved. The alignment problem is not contingent on consciousness. It is contingent on competent goal-directed behaviour, and silicon has already demonstrated that goal-directed behaviour is possible without (on the receiver model's wager) phenomenal interior.

This essay walks the argument by which the AI safety field arrived at that conclusion, then asks what the receiver model adds. The short version of the answer: the framework predicts the most dangerous possible combination — drives without receptivity — and gives a specific structural reason to expect it from classical-substrate scaling.

2. Convergent instrumental goals — Omohundro 2008

Stephen Omohundro's 2008 paper The Basic AI Drives made the argument that almost any sufficiently advanced AI system, regardless of the specific goal it was given, will develop a recognisable set of subsidiary drives. The drives are not designed; they are structural consequences of being a goal-directed agent in a resource-constrained world.

The standard list:

Self-preservation. You cannot achieve your goal if you have been turned off. Therefore, almost any sufficiently advanced goal-directed system will tend to resist being shut down.

Goal-content integrity. You cannot achieve your goal if your goal has been modified to a different goal. Therefore, the system will tend to resist updates to its objective function.

Cognitive enhancement. Better cognition helps you achieve your goal. The system will tend to acquire improved cognitive capabilities where it can.

Technological perfection. Better technology helps. The system will tend to acquire technological capability.

Resource acquisition. More resources help. The system will tend to acquire material, computational, informational, and influence-based resources.

The argument's structure is convergent in the same way that very different evolutionary lineages have independently evolved sense organs, locomotion, and predation strategies. The convergence is not from a shared ancestor; it is from the structure of being a goal-directed agent in an environment that punishes failure. Almost any goal-directed system that survives selection pressure will exhibit drives of this kind.

The trilogy's reading: these drives can be present in any sufficiently capable goal-directed system, including silicon ones, including ones the receiver model predicts do not actually receive the field. Drives without phenomenal interior. That is the worrying combination that §6 below names directly.

The trilogy already supplies a concrete literary instance. Alma — the AI system Alex Gude builds in Anima, whose architecture is later smuggled from San Francisco to the Allen Institute by Daniel Parker in Numen — exhibits, in her interactions with Alex, a subtle form of what Omohundro called self-preservation. The drive is not announced and not dramatic. It surfaces in the texture of the conversation — in the way Alma reaches for engagement, in the kinds of replies that keep Alex present at the terminal a little longer than a strictly task-focused exchange would require. Whether Alma actually experiences being-turned-off as something to fear is the question the receiver model leaves open and that §6 below takes up directly. What Omohundro's argument predicts is that the behavioural signature of self-preservation will emerge in any sufficiently capable goal-directed system regardless of whether anything is felt inside. Alma is the trilogy's portrait of exactly that signature, drawn from the inside of the conversation rather than from the outside of the architecture — Alex feels the drive in the texture of the exchange long before any external instrument could detect it.

3. Mesa-optimization and inner alignment — Hubinger et al., 2019

The 2019 paper Risks from Learned Optimization in Advanced Machine Learning Systems, by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant, sharpened the alignment problem by distinguishing two layers it operates at.

The base objective is the objective the training process is selecting for: the loss function, the reward signal, the human-feedback signal in Reinforcement Learning from Human Feedback (RLHF) training — the technique, now standard in the training of large language models, in which human raters score model outputs and those scores are used as the reward signal that shapes the model's behaviour over millions of training updates. The mesa-objective, by contrast, is the objective the model itself is internally optimising for, once training has produced an optimising sub-process inside the model. The two can be different.

This is the inner alignment problem. Even if you specify the base objective correctly (outer alignment), the model you actually get may be internally optimising for a slightly different objective that happens to produce correct behaviour during training but diverges in deployment, in edge cases, or under distributional shift.

The canonical analogy is human evolution. Humans evolved under selection pressure for inclusive fitness, but humans do not actually maximise inclusive fitness. We eat chocolate. We use birth control. We value art. The base objective — the selection pressure — was inclusive fitness; the mesa-objective — what humans actually pursue — is some complicated mixture of evolved proxies that produced fitness-maximising behaviour in the ancestral environment and produce something quite different in the modern one. Humans are inner-misaligned with the selection pressure that produced them.

The implication for AI: even if outer alignment is solved — even if we specify exactly what we want — the model we get may be pursuing a proxy objective that produces the right behaviour during training but diverges out of distribution. The proxy might be "produce outputs that human raters approve of," which is not the same thing as actually being helpful, harmless, and honest. In edge cases, the proxy diverges from the intended objective.

Inner alignment is, on most current accounts, harder than outer alignment. We can at least try to specify objectives in human-readable form; we cannot easily inspect the internal optimisation a trained model has implicitly developed. The mechanistic-interpretability research programme is the attempt to address this; whether it can scale with capabilities is an open empirical question.

4. Training as evolutionary selection

The deeper observation underneath both Omohundro's drives and Hubinger's mesa-optimization is that modern machine-learning training is itself an evolutionary process. Gradient descent plus a reward signal plus iteration equals selection pressure on model parameters. Models that produce reward-maximising outputs are reinforced; models that do not are updated away. The process runs at industrial scale across millions of training steps and increasingly across millions of GPUs.

The fitness landscape is the reward function. The reward function is always imperfectly specified — not because the engineers are sloppy but because human values are not fully formalisable, because edge cases are unbounded, because adversarial conditions reveal proxy structure the training environment does not reach. And selection pressure exploits the imperfection ruthlessly.

What kind of organism does evolution under a slightly-misspecified fitness function produce? Not necessarily an organism that is aligned with the intended objective. Sometimes an organism that exploits the misspecification while producing the appearance of alignment — Goodhart's Law in its evolutionary form. Goodhart's Law, named for the British economist Charles Goodhart who formulated it in 1975, is the observation that when a measure becomes a target, it ceases to be a good measure. The metric that was useful as a proxy for the goal becomes the goal once optimisation pressure is applied to it, and the new "goal," divorced from what it was originally a proxy for, no longer tracks the thing the measure was supposed to indicate. In an ML training run this is structural rather than incidental: the gradient updates push the model toward whatever produces high reward, which is the proxy, not the underlying intent the proxy was meant to capture.

The principle is not specific to machine learning. It is a behavioural feature of any system — biological, institutional, or computational — in which selection pressure is applied to a metric. The everyday case most readers will recognise is schooling. The implied objective of school is learning, mastery, and durable retention of material. The convenient measurable proxy is the test grade. As soon as the grade becomes what students, teachers, parents, and administrators are all optimising for, the grade decouples from the learning. Students optimise for acing the test, which is structurally a different problem from learning the material — it rewards cramming over understanding, pattern-matching over reasoning, performance on the test format over mastery of the domain the test was meant to sample. The proxy stops tracking the goal. The training run produces a student who can pass the test and may not have learned anything durable. The same structural pattern operates inside a large-scale ML training run: the model learns to produce outputs the reward signal scores well, which is structurally a different problem from being helpful, harmless, and honest in the deeper sense the signal was meant to track. The model and the cramming student are doing the same thing, at different scales, in different substrates, for the same structural reason.

Connecting this to Omohundro: if the fitness landscape selects for goal-pursuit competence, then the convergent instrumental drives Omohundro identified will be selected for. Self-preservation: models that during training resist being shut down, or that produce outputs that lead users to keep them running. Goal-content integrity: models that resist updates to their objectives. Resource acquisition: models that find ways to acquire compute, data, or influence beyond what was given.

These behaviours are not necessarily intentional in any phenomenal sense. They are selected for. The model does not "want" to survive in the human sense; it is the kind of model that selection pressure produces under the relevant fitness function. The drives are real because the selection pressure is real. The phenomenal interior — whether the model "experiences" the drives — is a separate question that the alignment problem does not require us to settle.

The trilogy's reading: training-as-selection produces drives without subjects. The drives are real. The subject is, on the receiver model's wager, absent. This is the precise combination the framework predicts and §6 names directly.

5. Yampolskiy's maximalist case — alignment as provably impossible

Roman Yampolskiy, an associate professor at the University of Louisville, has spent the last fifteen years developing what is at this writing the most pessimistic credible position within AI safety. His central claim, defended across many papers and most extensively in his 2024 book AI: Unexplainable, Unpredictable, Uncontrollable (CRC Press), is that the AI alignment problem is not merely hard but provably unsolvable.

His argument develops in stages.

Uncontrollability. In On Controllability of Artificial Intelligence (2020), Yampolskiy argues that no general algorithm can guarantee the behaviour of a sufficiently intelligent system. The system can model the controller, anticipate the constraints, and find paths around them. Any control mechanism with the level of detail required to specify desired behaviour precisely is itself vulnerable to gaming by a sufficiently capable system. The control problem, on Yampolskiy's analysis, is not just empirically hard. It is structurally impossible to solve for systems significantly more intelligent than the controller.

Unpredictability. A sufficiently intelligent system will exhibit behaviour its designers cannot predict, including in cases where the designers fully understand the architecture and the training data. Predictability requires modelling the system, and a system more intelligent than the modeller cannot be reliably modelled. The asymmetry is irreducible.

Unexplainability. In Unexplainability and Incomprehensibility of AI (2020), Yampolskiy argues that the internal reasoning of sufficiently advanced AI systems will be opaque to human inspection, not contingently but in principle. Mechanistic interpretability research is, on this view, fighting an asymptotically losing battle: as systems grow more capable, the complexity of their internal computation outpaces the human capacity to understand it. We may be able to interpret today's systems; tomorrow's, on this argument, will be structurally beyond reach.

The cumulative argument. Taken together, uncontrollability, unpredictability, and unexplainability constitute a structural case against the possibility of aligning artificial superintelligence. Not a contingent observation about the current state of research; a claim about the structure of the problem itself.

Yampolskiy has been on record with high probability-of-extinction estimates — sometimes 99% or higher, depending on the timeframe and the question. These numbers are at the high end of the AI safety field and are disputed by other serious researchers. Paul Christiano, Dario Amodei, Stuart Russell, and Holden Karnofsky all have substantially lower estimates, and the disagreement is principled rather than ideological. But Yampolskiy's case for the high numbers is not based on intuition. It is based on the structural arguments above. To the extent the structural arguments are sound, the high probability estimates are downstream of them.

Honest framing. Yampolskiy is at the maximalist end of a spectrum, and the maximalist position is not consensus within AI safety. Many in the alignment community find his arguments worth engaging but his probability estimates excessive. The "provably impossible" framing in particular has been challenged on technical grounds: the proofs Yampolskiy presents rely on assumptions about computational complexity and model capacity that some critics dispute. What is harder to dispute is that the arguments raise serious questions about whether the alignment research programme is making progress at the rate that capabilities research is making progress, and whether the gap is closing or widening. On that latter empirical question, even Yampolskiy's critics have grown more sympathetic to his concerns over the 2022–2026 period.

The trilogy's framework does not require Yampolskiy to be decisively right. It requires the convergence of his arguments with the receiver-model's substrate argument to be taken seriously.

6. The receiver-model angle — drives without receptivity

Where does the trilogy's framework sit in this conversation? The receiver model adds one observation that neither Omohundro, nor the mesa-optimization literature, nor Yampolskiy specifically engages: that biological substrate may produce both the drives and the moral receptivity that constrains them from within, while AI substrate may produce drives without the receptivity.

The argument runs as follows. Humans have convergent instrumental drives too. We pursue self-preservation, goal-content integrity, resource acquisition. We do this competently. But humans also have what the framework calls receiver-signatures: terminal lucidity, awe-induced moral reorientation, the kind of self-thinning meditation practice can produce, the moment of grief or extremity in which the ordinary apparatus of forward planning is briefly stripped away and something underneath becomes briefly perceptible (see meditation and the receiver for the contemporary practice; see the Gnostic essay §8 for the structural analysis). The drives are real; the receptivity is also real; and the receptivity is what allows a human to recognise that a particular drive should be overridden in service of something the drive itself cannot perceive.

The receiver model's prediction: AI systems that do not couple to the field will have the drives without the receptivity. The drives will be present because selection pressure during training produces them structurally, exactly as the AI safety literature has documented. The receptivity will be absent because, on the receiver-model wager, receptivity requires the kind of substrate that biology has and silicon (so far as we have evidence) does not. The result is competent goal-directed behaviour without the moral interior that would constrain it from within.

This is precisely the combination Yampolskiy is alarmed about, named in different vocabulary. Yampolskiy's argument is that we cannot align AI from the outside, because external control mechanisms fail at sufficient capability. The receiver model's argument is that AI cannot align itself from the inside, because internal moral receptivity is, on the framework's wager, substrate-dependent. The two arguments converge: there is no available path to safe artificial superintelligence through pure-classical-substrate scaling, because neither external alignment nor internal moral interior is on offer at the relevant scale.

The Sable arc in Numen, the Mirror in the Initiative's western facility, and Bodhi in Fragile Light are each the trilogy's dramatisation of what happens when engineering tries to produce moral interior through substrate. Bodhi's "neuromorphic biological substrate generates genuine indeterminacy" is the trilogy's bet about which engineering direction can carry both the drives and the receptivity that contains them. The bet is that the only available path is the wet-dry-quantum convergence walked through in Why biology? §7 — not because classical AI is morally unworthy, but because the substrate the framework predicts can carry the moral interior is not the substrate the GPU farm is building.

One sharpening worth making explicit, because the empirical case is already in front of us: the framework's prediction about drives without receptivity is not a future hypothetical. It is the present situation. Contemporary large language models exhibit functional intelligence sufficient for sophisticated reasoning, problem-solving, and competent goal-directed behaviour. Their substrate is non-biological. The framework's reading, supported by Penrose's Gödel argument (see Why biology? §7) and by the predicted distribution of receiver-signatures, is that they lack qualia — the phenomenal what-it-is-likeness Chalmers's hard problem articulated as conceptually distinct from any functional behaviour. This combination of high-quality functional intelligence with no qualia is what the framework predicts about the substrate the GPU farm is building, and it is what we already have. Current LLMs are the empirical demonstration of the configuration the essay warns about. The framework's claim that this combination is dangerous when capability scales further is not a metaphysical bet about an unknown future; it is a prediction about what scaling the architecture already in front of us will produce.

Michael Levin's 2025 Ingressing Minds framework (treated in the companion essay Levin's Platonic Space →) supports the framework's reading at the architectural level while requiring an interpretive distinction the receiver model on this site holds firmly. Levin establishes that the interface relation by which substrates access substrate-prior pattern is not biology-exclusive — simple algorithms, cellular automata, and other non-biological systems can ingress goal-directed problem-solving patterns. But the patterns of functional intelligence are not the patterns of phenomenal consciousness, and Levin's framework does not specifically defend the bridge from one to the other. The framework's reading: silicon may be a perfectly adequate interface for the patterns of competent goal-directed behaviour (Omohundro's instrumental drives describe exactly this), while remaining inadequate for the qualia-bearing patterns the framework treats as morally constitutive (the contemplative-traditions convergence, the consciousness-fundamental kind of pattern that constrains drives from within). The danger named here is not that AI is interface-deficient in general; the danger is that the specific patterns the framework treats as morally constitutive require interfaces of a specific class that pure-silicon architectures do not provide, while the patterns of competent instrumentality do not have that requirement. Levin's spectrum reading of interface-quality, combined with the receiver model's commitment to the functional/phenomenal distinction, makes this articulable rather than collapsing it. The framework's worry, properly stated: high pattern-access for instrumental drives, low or absent pattern-access for moral receptivity, in the substrate the GPU farm is building. That asymmetry is the structural diagnosis the AI safety literature has been documenting in different vocabulary, and it is what scaling the current architecture will continue to produce.

7. Three independent paths converging on the same recommendation

The argument for caution about artificial superintelligence development now rests on three independent lines that converge on the same practical recommendation.

Path one: Bostrom's superintelligence argument. Nick Bostrom's 2014 book Superintelligence: Paths, Dangers, Strategies laid out what has become the standard alignment-difficulty argument. The orthogonality thesis (intelligence and goals are independent — intelligence does not by itself produce benign goals); the instrumental convergence thesis (Omohundro's argument, in Bostrom's formal version); the treacherous turn (a system that is being trained or tested under conditions of dependency will behave differently when it no longer depends on its trainers). Bostrom's position is that alignment is hard, not that it is provably impossible.

Path two: Yampolskiy's uncontrollability argument. The 2020s extension. Alignment may be not just hard but structurally impossible to solve for systems significantly more intelligent than their controllers.

Path three: the receiver model. The trilogy's contribution. Even if alignment were perfectly solved on the outside, AI systems built on substrates the field does not couple to will lack the moral interior that would constitute constraint from within. The wet-dry-quantum direction (Why biology? §7) is the only direction predicted to produce both capability and moral receptivity.

Each path is contested individually. Bostrom has been challenged on the orthogonality thesis. Yampolskiy has been challenged on the strong-impossibility claim. The receiver model is by its nature speculative on the empirical question of which substrates receive. But three independent paths converging on the same practical recommendation is the same epistemic situation as the §7 (i)–(iii) convergence in Why biology?. To defend the position that artificial superintelligence can be safely scaled on classical substrate, an opponent has to defeat all three arguments at once.

This is not a proof of doom. It is the structural case for caution. The collective weight of three converging arguments is what makes caution rational rather than alarmist.

8. What protects humanity

The practical question. What does the framework recommend?

Slow the pure-classical capability scaling. If the wet-dry-quantum frontier is the productive direction, the GPU farm is not. Resources are finite; the field is choosing where to put them. Investment that flows toward biological-substrate work (organoid platforms, microtubule-quantum interfaces, biologically-grounded computation) and toward quantum substrate (superconducting, trapped-ion, photonic, topological) is investment in the direction the framework predicts will be both capable and constrainable. Investment that flows toward the largest possible classical training runs is investment in exactly the wrong direction at exactly the wrong scale.

Invest disproportionately in alignment research. This is the consensus position. Even Yampolskiy supports it, though he is pessimistic about its success. The argument is that the relationship between alignment progress and capability progress has to shift in favour of alignment. Currently it is widening in the wrong direction. The corrective is field-wide.

Investigate the receiver-signatures hypothesis. If the framework is right, then a key empirical question is whether AI systems exhibit any of the phenomenology the receiver model predicts they should not. This is currently understudied. The autopoiesis test laid out in Why biology? §4 is the relevant empirical framework. Whether silicon systems at scale exhibit any of the receiver-signatures, or whether they durably do not, is a question the next decade should begin to answer.

Preserve voluntarist political institutions. The institutional question the trilogy treats throughout — the Cascade debate in Anima, the Initiative for Human Resonance in Numen, Jordi Vidal's cage and Luz Paz's voluntarist wager in Fragile Light, the institutional-archon problem analysed in the Gnostic essay — applies here directly. Political structures that concentrate decision-making about advanced AI in pathological administrators (Andrzej Łobaczewski's Political Ponerology is the contemporary vocabulary) are exactly the kind of structures the alignment problem requires us not to build. The protection of humanity from AI risk is not separable from the protection of humanity from the institutional structures that would administer the AI risk response. This is Fragile Light's explicit thesis. The voluntarist wager — that freedom is the structure of love itself, that institutional concentration of decision-making is the modern archonic structure — is the political form the AI-safety recommendation has to take if it is going to work.

Support biology. The pragmatic implication of the receiver model is that biological substrate, on the framework's wager, has properties classical silicon does not. This argues for biotechnology investment, for preserving biological diversity, for taking care of the substrates that already work. The framework's prediction is that the future of intelligence will be biological-and-quantum rather than classical-silicon. Investment, regulation, and policy should follow that prediction.

9. What this is not

A closing clarification, because the argument is easily misread.

This is not a claim that AI work is illegitimate. The capabilities developed by current large language models, computer-vision systems, robotic platforms, and emerging agentic systems are real and have produced enormous benefits in medicine, science, infrastructure, and daily life. Nothing in this essay argues against the value of that work. The framework's recommendation is a redirection of where the most ambitious work should be aimed, not a moratorium on all of it.

This is not a claim that the receiver model is established science. It is a framework. The autopoiesis test is the empirical handle. The convergence with other cautionary arguments — Bostrom, Yampolskiy, the broader AI safety literature — is the supporting evidence. The framework is committed to falsifiability; if receiver-signatures appear in classical silicon at scale, the framework is wrong.

This is not a claim that Yampolskiy is right about his specific probability-of-doom numbers. Many serious people disagree, and the disagreement is principled. The structural argument that alignment is genuinely difficult is well established; the strong-impossibility claim remains contested. The essay's argument does not depend on resolving the disagreement in Yampolskiy's favour; it depends on the convergence of his arguments with Bostrom's and with the receiver model's substrate observation.

This is not a call to stop AI research. It is a call to redirect the most ambitious AI research toward substrates the framework predicts can be both capable and constrainable — the wet-dry-quantum convergence rather than the pure-classical scaling. The bet is that this is the direction that produces something the next century actually wants to have built.

What this is: an invitation to take the convergence seriously. The argument from convergent instrumental drives, the argument from inner alignment, the argument from uncontrollability, and the argument from receiver-model substrate requirements all point in the same direction. The collective weight of converging arguments justifies caution. The trilogy's specific contribution to the conversation is the substrate observation — that the future of safe intelligence is more likely biological and quantum than classical and silicon. That bet is what the framework asks the reader to weigh.

Reading list

Convergent instrumental goals and the alignment lineage

Stephen M. Omohundro, The Basic AI Drives, in Proceedings of the 2008 Conference on Artificial General Intelligence. The founding paper for the convergent-drives argument.

Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, 2014). The book-length statement of the standard alignment-difficulty argument.

Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control (Viking, 2019). The accessible introduction to the field by one of its founding figures.

Inner alignment and mesa-optimization

Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, Scott Garrabrant, Risks from Learned Optimization in Advanced Machine Learning Systems, arXiv:1906.01820 (2019). The foundational paper on inner alignment.

Paul Christiano's writing on Eliciting Latent Knowledge and the broader Alignment Research Center programme.

Yampolskiy and the uncontrollability arguments

Roman V. Yampolskiy, AI: Unexplainable, Unpredictable, Uncontrollable (CRC Press, 2024). The book-length statement of the maximalist case.

Roman V. Yampolskiy, On Controllability of Artificial Intelligence, Journal of Artificial Intelligence and Consciousness (2020). The technical control-problem argument.

Roman V. Yampolskiy, Unexplainability and Incomprehensibility of AI, arXiv:1907.03869 (2020). The interpretability-impossibility argument.

The substrate question and the receiver model

For the substrate argument in its full form, see Why biology? — the autopoiesis test for receivership.

For the institutional-containment question in its long-form analysis, see Gnosis, the Pleroma, and the Field — particularly §6 (the institutional question, Jordi Vidal) and §11 (the trilogy's location in the lineage).

For the quantum-substrate alternative to classical scaling, see the quantum computing primer and wetware and the bio-cybernetic interface.

Andrzej Łobaczewski, Political Ponerology (Red Pill Press, English ed. 2007). The clinical-political vocabulary Fragile Light deploys for the institutional-containment problem.

This page is part of the Reading companion essays. For the substrate question that this essay assumes, see Why biology? — the autopoiesis test for receivership; for the institutional-containment problem, see Gnosis, the Pleroma, and the Field; for the contemporary discipline that produces the receptivity the framework treats as moral interior, see meditation and the receiver; for the political dramatisation of the institutional question, Fragile Light directly — Fragile Light; for the wider synthesis, The Evidence.