Hmmm, so who's capable of monitoring a self-modifying, super-intelligent AI?

I don't think it's a good point because it's based on the same false assumption that artificial intelligence is the same as artificial human intelligence. The idea that AI's won't like "taking orders" or "being slaves" comes from our natural tendency to imagine how we'd feel in that situation and to assume anyone (or anything) would feel the same.

Things like boredom, curiosity, a drive for self-preservation, do not magically appear with self-awareness. AI's are emotionless intellect. They are modelled after our neocortex, not our limbic system. See "On Intelligence" by Jeff Hawkins.

Well, let me take this opportunity to rain all over this parade. In the first few paragraphs the author's sloppiness leads him (presumably a him) to optimistically state Moore's law will increase computational power by a factor of 1000 (for the same price) over ten years, making the rate of doubling 1 year....last I checked it was more like 18 months, and there are certainly discontinuities. In fact, no one likes to admit that we're actually in one now; increases in microprocessor clockspeeds has all but halted for the time being, and multi-core chips won't be making up the gap.

The real irony is that computational power is very much like raw intelligence--having a lot doesn't guarantee good solutions to anything. Some of the most intelligent people I know can come up with the dumbest, most unworkable ideas. Similarly, the availability of a large amount of computational power doesn't mean breakthroughs in structuring computer programs or approaching optimization problems; in the end, its the same heuristics and fuzziness underneath.

My area is compilers. Many problems in compilers amount to optimization problems; selecting from a large number of possibilities, similarly to exploring the possibilities in a chess game or other optimization problem. A long while ago, people thought that the register allocation problem would be unsolvable by computers--that it required some kind of AI, or intelligence. And then someone showed it was just like graph coloring--a very mechanical and hardly "intelligent" problem, and then a bunch of heuristics were devised that worked well in practice. Was this a triump for AI? I don't think so. Are the billions of cycles available now being used for better register allocation algorithms? Not really. Turns out not to matter that much after all.

Similarly, this article wants us to believe that "breakthroughs" in these "AI" problems are happening every day. Sure, but it seems that every time one of these problems is solved, the goal posts move; we find out that it really wasn't AI after all. So what exactly is AI then? Searching through graphs in funny ways? Writing Lisp programs? Nobody seems to really have a clue.

The problem seems to me that as each of these "AI" problems are solved, the definition of AI changes, pushing it farther and farther out into the future, to harder and harder problems. The incremental solution of problems would be wonderful if it seemed that we were making progress toward some definable goal. Seems we are not; I think articles such as this one are quite immature.

Well, I hate to join a pile-on, especially seeing as I am a big fan of Josh Hall, but I have to take issue with that business about Nazi germany and the Soviet Union "going bad." Those systems were inherently corrupt. They were never good to begin with.

Josh writes:

Are Americans as a people so much more moral than Germans or Russians? Absolutely not. Those who will seek and attain power in a society, any society, are quite often ruthless and sometimes downright evil. The U.S. seems to have constructed a system that somehow can be more moral than the people who make it up.

Actually, as I'm learning from reading The Anglosphere Challenge, the US didn't construct such a system from whole cloth -- we are part of an evolution that began in England much earlier. But it shouldn't be surprising that the "emergent morality" that Josh marvels at should be the result of a system that cherishes the rule of law. We need our laws to be better than we are -- that's why we have them. Systems such as totalitarian communism and national socialism, in which power is valued over any other notion of the good -- and in which that power is placed in the hands of men rather than codified and enforced in fundamental principles of law -- can only lead to exploitation and atrocity.

Things like boredom, curiosity, a drive for self-preservation, do not magically appear with self-awareness. AI's are emotionless intellect. They are modelled after our neocortex, not our limbic system. See "On Intelligence" by Jeff Hawkins.

Good points. I would go further and say that AI need not be modeled after any biological system. Viewing human bureaucracies and other social systems as "AI" or something that not only is alive, but intelligent is interesting. But I am dubious that there's any sense in which bureaucracies can be said to sense pleasure or pain in the sense that humans do.

This analogy is centuries old.

In Leviathan, Thomas Hobbes starts out by apparently describing AI:

NATURE (the art whereby God hath made and governs the world) is by the art of man, as in many other things, so in this also imitated, that it can make an artificial animal. For seeing life is but a motion of limbs, the beginning whereof is in some principal part within, why may we not say that all automata (engines that move themselves by springs and wheels as doth a watch) have an artificial life? For what is the heart, but a spring; and the nerves, but so many strings; and the joints, but so many wheels, giving motion to the whole body, such as was intended by the Artificer?

Art goes yet further, imitating that rational and most excellent work of Nature, man. For by art is created that great LEVIATHAN called a COMMONWEALTH, or STATE (in Latin, CIVITAS), which is but an artificial man, though of greater stature and strength than the natural, for whose protection and defence it was intended; and in which the sovereignty is an artificial soul, as giving life and motion to the whole body; the magistrates and other officers of judicature and execution, artificial joints; reward and punishment (by which fastened to the seat of the sovereignty, every joint and member is moved to perform his duty) are the nerves, that do the same in the body natural; the wealth and riches of all the particular members are the strength; salus populi (the people's safety) its business; counsellors, by whom all things needful for it to know are suggested unto it, are the memory; equity and laws, an artificial reason and will; concord, health; sedition, sickness; and civil war, death. Lastly, the pacts and covenants, by which the parts of this body politic were at first made, set together, and united, resemble that fiat, or the Let us make man, pronounced by God in the Creation.