
I was recently watching
this video, in which Eliezer Yudkowsky talks about the
runaway AI scenario. I've seen him make this argument many times, and in summary it's something like:
We don't want to build an AI which fills the universe with paperclips.
There appears to exist a gigantic theoretical oversight somewhere here. How would the paperclips actually be produced?
Filling the universe with any kind of monoculture seems like a fairly unintelligent thing to do, but suppose for sake of argument I was to construct an autonomous paperclip manufacturing machine and start it going by winding one of those hand cranks so often used on early 20th century automobiles. Would such a machine really pose a serious threat to the universe requiring well educated commentators to lose sleep and dedicate entire careers to contemplation of the wider existential implications?
Suppose that my machine is modestly high tech. It could for instance mine iron ore from the soil using some
incredibly vaguely defined nanotechnological paraphenalia. To mine minerals at any spatial scale requires firstly that you have somewhere to
store the stuff you've extracted, and secondly that you have somewhere to
dump the stuff which you don't want. Obviously if the machine were to dump waste products at the same location as the mining was taking place the unwanted buildup would soon lead to frustrations and reduce the efficiency of operations. One way to avoid efficiency-depleting deposits might be to fling soil out of the back of the machine, and have the machine itself parambulate slowly across the terrain - essentially performing open cast mining. This might be ok if the terrain in question was uniformly rich in iron deposits, but this is rarely the case. Concentration of minerals at specific geological sites would mean that the aforementioned flinging strategy might not suffice to dispose of waste materials for very long.
Already, our superintelligence is facing an increasingly accumulating heap of trouble, even before it has arrived at the enviable position of being able to manufacture its very first paperclip. But superintelligences don't rest for long. A secondary plan, which performs the waste disposal job more efficiently would be to have trucks into which soil can be loaded. The trucks could then carry the waste soil to more distant locations where it could be dumped. Unfortunately in the classic runaway scenario the machine is unaided by human workers, and is instead
some completely autonomous device, so the trucks need to themselves be manufactured before they can play any non-trivial role in this whole drama. In the initial stage of development therefore the machine needs to concentrate upon building infrastructure rather than directly carrying out its primary goal (the making of paperclips). Building infrastructure takes time, energy, and above all planning permission from the relevant authorities. Even if the requisite paperwork is fast-tracked - or waived entirely - we still need energy to move atoms from one place to another, and in a terrestrial environment the maximum rate of movement is going to be constrained by a variety of physical factors such as friction and gravity - no matter what kind of transportation technology is actually used.

So, although automated construction of mining facilities could proceed somewhat expeditiously a variety of physical constraints will mean that the rate of manufacture is going to be limited to the sorts of time scales where humans would be likely to notice what was going on and possibly intervene to halt the flagrant violation of local planning regulations. For instance, it seems unlikely that the laws of physics would permit a large open cast mining operation to be set up within 60 seconds, or even a few hours, if we assume that the initial paperclip machine is a metre or two in diameter and that it's using the most advanced nanotechnology available.
Perhaps I'm missing something, but it just seems unlikely that anything resembing the paperclip scenario could occur without the active collaboration of humans in the process and a good deal of manufacturing infrastructure already in place. Such scenarios don't appear to have been thought through in sufficient detail.