Superintelligence: Paths, Dangers, Strategies

... the whole ecosystem of artificial intelligence is optimized for a lack of accountability.

Naturalism -- the doctrine that there is nothing apart from the physical order, and certainly nothing supernatural -- is an incorrigibly incoherent concept, and one that is ultimately indistinguishable from pure magical thinking. The very notion of nature as a closed system entirely sufficient to itself is plainly one that cannot be verified, deductively or empirically, from within the system of nature. It is a metaphysical (which is to say 'extranatural') conclusion regarding the whole of reality, which neither reason nor experience legitimately warrants. . . .

If moreover, naturalism is correct (however implausible that is), and if consciousness is then an essentially material phenomenon, then there is no reason to believe that our minds, having evolved purely through natural selection, could possibly be capable of knowing what is or is not true about reality as a whole. Our brains may necessarily have equipped us to recognize certain sorts of physical objects around them, but there is no reason to suppose that such structures have access to any abstract 'truth' about the totality of things. If naturalism is true as a picture of reality, it is necessarily false as a philosophical precept.

The one thing of which it can give no account, and which its most fundamental principles make it entirely impossible to explain at all, is nature's very existence. For existence is definitely not a natural phenomenon; it is logically prior to any physical cause whatsoever; and anyone who imagines that it is susceptible of a natural explanation simply has no grasp of what the question of existence really is. . . . There is something of a popular impression out there the naturalist position rests upon a particularly sound rational foundation. But, in fact, materialism is among the most problematic of philosophical standpoints, the most impoverished in its explanatory range, and among the most willful and (for want of a better word) magical in its logic. . . . The heuristic metaphor of a purely mechanical cosmos has become a kind of ontology, a picture of reality as such. The mechanistic view of consciousness remains a philosophical and scientific premise only because it is now an established cultural bias, a story we have been telling ourselves for centuries, without any real warrant from either reason or science. . . .

Naturalism commits the genetic fallacy, the mistake of thinking that to have described a thing's material history or physical origins is to have explained that thing exhaustively. We tend to presume that if one can discover the temporally prior physical causes of some object -- the world, an organism, a behavior, a religion, a mental event, an experience, or anything else -- one has thereby eliminated all other possible causal explanations of that object. . . . To bracket form and finality out of one's investigations as far as reason allows is a matter of method, but to deny their reality altogether is a matter of metaphysics. if common sense tells us that real causality is limited solely to material motion and the transfer of energy, that is because a great deal of common sense is a cultural artifact produced by an ideological heritage. . . .

Consciousness is a reality that cannot be explained in any purely physiological terms at all. The widely cherished expectation that neuroscience will one day discover an explanation of consciousness solely within the brain's electrochemical processes is no less enormous a category error than the expectation that physics will one day discover the reason for the existence of the material universe. It is a fundamental conceptual confusion, unable to explain how any combination of diverse material forces, even when fortuitously gathered into complex neurological systems, could just somehow add up to the simplicity and immediacy of consciousness, to its extraordinary openness to the physical world, to its reflective awareness of itself. . . .

Naturalists fall victim to the pleonastic fallacy, another hopeless attempt to overcome qualitative difference by way of an indeterminately large number of gradual quantitative steps. This is the great non sequitir that pervades practically all attempts, evolutionary or mechanical, to reduce consciousness wholly to its basic physiological constituents. At what point precisely was the qualitative difference between brute physical causality and unified intentional subjectivity vanquished? And how can that transition fail to have been an essentially magical one? There is no bit of the nervous system that can become the first step toward intentionality.

One can conduct an exhaustive surveillance of all those electrical events in the neurons of the brain that are undoubtedly the physical concomitants of mental states, but one does not thereby gain access to that singular, continuous, and wholly interior experience of being this person that is the actual substance of conscious thought. . . . There is an absolute qualitative abyss between the objective facts of neurophysiology and the subjective experience of being a conscious self. . . . Consciousness as we commonly conceive of it is also almost certainly irreconcilable with a materialist view of reality, and there is no ‘question’ of whether subjective consciousness really exists -- subjective consciousness is an indubitable primordial datum, the denial of which is simply meaningless.

Many of the points made in this book are probably wrong. It is also likely that there are considerations of critical importance that I fail to take into account, thereby invalidating some or all of my conclusions. I have gone to some length to indicate nuances and degrees of uncertainty throughout the text — encumbering it with an unsightly smudge of “possibly,” “might,” “may,” “could well,” “it seems,” “probably,” “very likely,” “almost certainly.” Each qualifier has been placed where it is carefully and deliberately. Yet these topical applications of epistemic modesty are not enough; they must be supplemented here by a systemic admission of uncertainty and fallibility. This is not false modesty: for while I believe that my book is likely to be seriously wrong and misleading, I think that the alternative views that have been presented in the literature are substantially worse - including the default view, according to which we can for the time being reasonably ignore the prospect of superintelligence.

1. Just being intelligent doesn't imply being benign; intelligence and goals can be independent. (the orthogonality thesis.)
2. Any agent which seeks resources and lacks explicit moral programming would default to dangerous behaviour. You are made of things it can use; hate is superfluous. (Instrumental convergence.)
3. It is conceivable that AIs might gain capability very rapidly through recursive self-improvement. (Non-negligible possibility of a hard takeoff.)
4. Since AIs will not be automatically nice, would by default do harmful things, and could obtain a lot of power very quickly*, AI safety is morally significant, deserving public funding, serious research, and international scrutiny.

For a child with an undetonated bomb in its hands, a sensible thing to do would be to put it down gently, quickly back out of the room, and contact the nearest adult. Yet what we have here is not one child but many, each with access to an independent trigger mechanism. The chances that we will all find the sense to put down the dangerous stuff seem almost negligible. Some little idiot is bound to press the ignite button just to see what happens. Nor can we attain safety by running away, for the blast of an intelligence explosion would bring down the firmament. Nor is there a grown-up in sight...

This is not a prescription of fanaticism. The intelligence explosion might still be many decades off in the future. Moreover, the challenge we face is, in part, to hold on to our humanity: to maintain our groundedness, common sense, and goodhumored decency even in the teeth of this most unnatural and inhuman problem. We need to bring all human resourcefulness to bear on its solution.

* People sometimes choke on this point, but note that the first intelligence to obtain half a billion dollars virtually, anonymously, purely via mastery of maths occurred... just now. Robin Hanson chokes eloquently here and for god's sake let's hope he's right.