Another reason we're told not to worry is that these machines can't help but share our values because they will be literally extensions of ourselves.
另外一个试图安慰我们的理由是，那些机器必须拥有和我们一样的价值观，因为它们将会是我们自身的延伸。
They'll be grafted onto our brains, and we'll essentially become their limbic systems.
它们将会被嫁接到我们的大脑上，我们将会成它们的边缘系统。
Now take a moment to consider that the safest and only prudent path forward, recommended, is to implant this technology directly into our brains.
现在我们再思考一下最安全的，也是唯一经慎重考虑后推荐的发展方向，是将这项技术直接植入我们大脑。
Now, this may in fact be the safest and only prudent path forward,
这也许确实是最安全的，也是唯一慎重的发展方向，
but usually one's safety concerns about a technology have to be pretty much worked out before you stick it inside your head.
但通常在我们把它塞进脑袋之前，会充分考虑这项技术的安全性。
The deeper problem is that building superintelligent AI on its own seems likely to be easier than
更深一层的问题是：仅仅制造出超级人工智能机器可能要比
building superintelligent AI and having the completed neuroscience that allows us to seamlessly integrate our minds with it.
既制造超级人工智能，又让其拥有能让我们的思想和超级人工智能无缝对接的完整的神经科学系统要简单很多。
And given that the companies and governments doing this work are likely to perceive themselves as being in a race against all others,
而做这些研究的公司或政府，很可能将彼此视作竞争对手，
given that to win this race is to win the world, provided you don't destroy it in the next moment,
因为赢得了比赛就意味着称霸了世界，前提是不在刚成功后就将其销毁，
then it seems likely that whatever is easier to do will get done first.
所以结论是：简单的选项一定会被先实现。
Now, unfortunately, I don't have a solution to this problem, apart from recommending that more of us think about it.
但很遗憾，除了建议更多人去思考这个问题，我对此并无解决方案。
I think we need something like a Manhattan Project on the topic of artificial intelligence.
我觉得在人工智能问题上，我们需要一个“曼哈顿计划”。
Not to build it, because I think we'll inevitably do that,
不是用于讨论如何制造人工智能，
but to understand how to avoid an arms race and to build it in a way that is aligned with our interests.
因为我们一定会这么做，而是去避免军备竞赛，最终以一种有利于我们的方式去打造它。
When you're talking about superintelligent AI that can make changes to itself,
当你在谈论一个可以自我改造的超级人工智能时，
it seems that we only have one chance to get the initial conditions right,
我们似乎只有一次正确搭建初始系统的机会，
and even then we will need to absorb the economic and political consequences of getting them right.
而这个正确的初始系统需要我们在经济以及政治上做出很大的努力。
But the moment we admit that information processing is the source of intelligence,
但是当我们承认信息处理是智慧的源头，
that some appropriate computational system is what the basis of intelligence is,
承认一些电脑系统是智能的基础，
and we admit that we will improve these systems continuously,
承认我们会不断改善这些系统，
and we admit that the horizon of cognition very likely far exceeds what we currently know,
承认我们现存的认知远没有达到极限，将很可能被超越，
then we have to admit that we are in the process of building some sort of god.
我们又必须同时承认我们在某种意义上正在创造一个新的“上帝”。
Now would be a good time to make sure it's a god we can live with. Thank you very much.
现在正是思考人类是否能与这个“上帝”和睦相处的最佳时机。非常感谢！
