Continuing to read Accelerating Future, I note this snippet of a quote from Nick Bostrom:

I would argue that at least all humans, and probably many other sentient creatures on earth should get a significant share in the superintelligence's beneficence. If the benefits that the superintelligence could bestow are enormously vast, then it may be less important to haggle over the detailed distribution pattern and more important to seek to ensure that everybody gets at least some significant share, since on this supposition, even a tiny share would be enough to guarantee a very long and very good life. One risk that must be guarded against is that those who develop the superintelligence would not make it generically philanthropic but would instead give it the more limited goal of serving only some small group, such as its own creators or those who commissioned it.

Michael proceeds to comment on that last bit about the dangers of a superintelligence that serves the interests of only a limited numer of people. Such a scenario is undoubtedly a significant risk which needs to be addressed. However, even if everything works just right and the superintelligence is keen on helping all of us, I think that promises to be a lot more disruptive than we might expect.

For example, there are certain cultures in the world that continue to regard women as little more than chattels -- sub-humans who can be ordered around, beaten, or even killed as the man (owner) see fits. Children face the same treatment, but at least they get to grow out of being children. Unless it is really just interested in serving "man," I don't think a truly empathetic superintelligence is going to be satisfied with giving women in such circumstances a longer life and greater material abundance without addressing the underlying injustice. To do so would be to spruce up the cell while extending the prisoner's sentence indefinitely.

And speaking of prisoners, wouldn't the superintelligence be inclined to free all prisoners of concscience everywhere? And it might go further than that if it concluded that most or all incarcerations are a violation of human rights.

But it can't just declare all of these individuals to be free. The suerintelligence would have to be prepared to intervene to protect these individuals' new-found freedoms. Assuming it has the capacity to do so, the superintelligence will then -- for good or ill -- become the de facto government of the world.

Would a friendly, caring superintelligence really take over? If it really cares, it seems to me that it has no choice.

So yeah, we're going to need to be sure that the superintelligence truly is friendly and nice and wise. Because I don't think it will be handing the reins back over to us until (unless) we are able to demonstrate that we'll do roughly as good a job taking care of each other as it can do.

(Oh, please excuse the title of this post. How often do you have the opportunity to provide an established ironic title with a third, equally ironic meaning?)