在telligence Explosion FAQ

（（PDF版本可用）

  1. 基本
  2. How Likely is an Intelligence Explosion?
  3. Consequences of an Intelligence Explosion
  4. Friendly AI

1。基本

1.1。情报爆炸是什么？

The intelligence explosion idea was expressed by statistician I.J. Good in 1965[[13这是给予的：

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion’, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.

论点是：每年，计算机以新的方式超越人类的能力。1956年写的一项程序能够证明数学定理，并发现其中一个比罗素和怀特黑德给出的更优雅的证据Mathematica Principia[[14这是给予的。By the late 1990s, ‘expert systems’ had surpassed human skill for a wide range of tasks.[[15这是给予的1997年，IBM的深蓝色计算机击败了世界国际象棋冠军[[16这是给予的，，，，and in 2011, IBM’s Watson computer beat the best human players at a much more complicated game:Jeopardy![[17这是给予的。Recently, a robot named Adam was programmed with our scientific knowledge about yeast, then posed its own hypotheses, tested them, and assessed the results.[[18这是给予的[[19这是给予的

计算机远远远远远远远远没有人类智能，但是AID设计的资源正在积累（包括硬件，大数据集，神经科学知识和AI理论）。我们可能有一天设计一台超过人类技能的机器at designing artificial intelligences。在那之后，这台机器可以比人类更快，更好地改善自己的智能，这将使它甚至使它变得more擅长改善自己的智力。这可能会继续进行积极的反馈回路，以使机器比地球上最聪明的人快速变得更加聪明：“智力爆炸”，导致机器超智能。

This is what is meant by the ‘intelligence explosion’ in this FAQ.

See also:

2.情报爆炸的可能性有多大？

2.1。如何定义“智能”？

Artificial intelligence researcher Shane Legg defines[[20这是给予的这样的情报：

情报衡量代理在各种环境中实现目标的能力。

This is a bit vague, but it will serve as the working definition of ‘intelligence’ for this FAQ.

See also:

2.2。什么是人类的智能？

机器已经比人类在许多特定任务中都聪明：执行计算，下棋，搜索大型数据库，检测水下矿山等等。[[15这是给予的但是使人类与众不同的一件事是generalintelligence. Humans can intelligently adapt to radically new problems in the urban jungle or outer space for which evolution could not have prepared them. Humans can solve problems for which their brain hardware and software was never trained. Humans can even examine the processes that produce their own intelligence (cognitive neuroscience），，，，and design new kinds of intelligence never seen before (人工智能）。

To possess greater-than-human intelligence, a machine must be able to achieve goals more effectively than humans can, in a wider range of environments than humans can. This kind of intelligence involves the capacity not just to do science and play chess, but also to manipulate the social environment.

Computer scientist Marcus Hutter has described[[21这是给予的a formal model called AIXI that he says possesses the greatest general intelligence possible. But to implement it would require more computing power than all the matter in the universe can provide. Several projects try to approximate AIXI while still being computable, for example MC-AIXI.[[22这是给予的

尽管如此，在机器中可以实现大于人类的智能之前，还有很多工作要做。不必直接编程机器才能实现大于人类的智能。通过整个大脑仿真，生物认知增强或通过脑部计算机界面（见下文）也可以实现。

See also:

2.3。什么是整个大脑仿真？

全脑仿真（WBE）或“思维上传”是对人脑中所有细胞和连接的计算机仿真。因此，即使通用情报的基本原则很难发现，我们仍然可能模仿整个人的大脑，并以其正常速度运行的一百万倍（计算机电路通信很多faster than neurons do). Such a WBE could do more thinking in one second than a normal human can in 31 years. So this would not lead immediately to smarter-than-human intelligence, but it would lead to faster-than-human intelligence. A WBE could be backed up (leading to a kind of immortality), and it could be copied so that hundreds or millions of WBEs could work on separate problems in parallel. If WBEs are created, they may therefore be able to solve scientific problems far more rapidly than ordinary humans, accelerating further technological progress.

See also:

2.4。什么是生物认知增强？

可能有一些基因或分子可以修改以改善一般智力。金宝博娱乐研究人员已经在小鼠中做到了这一点：他们过表达了NR2B基因，从而改善了这些小鼠的记忆超出任何小鼠物种的其他小鼠的记忆。[[23这是给予的Biological cognitive enhancement in humans may cause an intelligence explosion to occur more quickly than it otherwise would.

See also:

2.5。什么是大脑计算机界面？

A brain-computer interface (BCI) is a direct communication pathway between the brain and a computer device. BCI research is heavily funded, and has already met dozens of successes. Three successes in human BCIs area devicethat restores (partial) sight to the blind,cochlear implantsthat restore hearing to the deaf, and a device that allows use of an artificial hand by direct thought.[[24这是给予的

Such device restore impaired functions, but many researchers expect to also augment and improve normal human abilities with BCIs.Ed Boydenis researching these opportunities as the lead of theSynthetic Neurobiology Groupat MIT. Such devices might hasten the arrival of an intelligence explosion, if only by improving human intelligence so that the hard problems of AI can be solved more rapidly.

See also:

2.6。How could general intelligence be programmed into a machine?

人工智能（AGI）有许多途径。一条途径是通过使用神经网或进化算法来模仿人脑，以构建数十个单独的组件，然后可以将它们拼凑在一起。[[29这是给予的[[30这是给予的[[31这是给予的另一个途径是从完美的通用智力的正式模型开始，并尝试将其近似。[[32这是给予的[[33这是给予的第三个途径是专注于开发可以自我屈服的“种子AI”，以便它可以学会自行聪明而无需首先实现人类水平的一般智能。[[34这是给予的Euriskois a self-improving AI in a limited domain, but is not able to achieve human-level general intelligence.

See also:

2.7。What is superintelligence?

Nick Bostrom defined[[25这是给予的‘superintelligence’ as:

几乎每个领域的人类大脑都要聪明得多，包括科学创造力，一般智慧和社交技能。

This definition includes vague terms like ‘much’ and ‘practically’, but it will serve as a working definition for superintelligence in this FAQ An intelligence explosion would lead to machine superintelligence, and some believe that an intelligence explosion is the most likely path to superintelligence.

See also:

2。8。情报爆炸何时发生？

Predicting the future is risky business. There are many philosophical, scientific, technological, and social uncertainties relevant to the arrival of an intelligence explosion. Because of this, experts disagree on when this event might occur. Here are some of their predictions:

  • 未来主义者雷·库兹维尔（Ray Kurzweil）预测，机器将于2030年到达人类水平的智能，到2045年，我们将实现“人类能力的深刻而破坏性的转变”。[[26这是给予的
  • 在tel’s chief technology officer, Justin Rattner,expects到2048年，人工和人工智能合并以创造比自身更大的东西的点。
  • AI researcher Eliezer Yudkowskyexpectsthe intelligence explosion by 2060.
  • 哲学家戴维·查尔默斯（David Chalmers）在2100年发生的情报爆炸中拥有超过1/2的信誉。[[27这是给予的
  • Quantum computing expert Michael Nielsenestimatesthat the probability of the intelligence explosion occurring by 2100 is between 0.2% and about 70%.
  • 2009年，在AGI-09会议上，询问专家何时AI可能会获得大量新资金。中位数估计是机器超智能可以在2045年（置信度为50％）或2100（90％的置信度）实现。当然，本次会议的与会者是自称的，认为近期的人工通用情报是合理的。[[28这是给予的
  • IROBOT首席执行官罗德尼·布鲁克斯（Rodney Brooks）and cognitive scientistDouglas Hofstadterallow that the intelligence explosion may occur in the future, but probably not in the 21st century.
  • 机器人汉斯·摩拉维克（Hans Moravec）预测，人工智能将超越人类的智力”well before 2050。”
  • 在a 2005 survey of 26 contributors to a series of reports on emerging technologies, the median estimate for machines reaching human-level intelligence was 2085.[[61这是给予的
  • Participants in a 2011 intelligence conference at Oxford gave a median estimate of 2050 for when there will be a 50% of human-level machine intelligence, and a median estimate of 2150 for when there will be a 90% chance of human-level machine intelligence.[[62这是给予的
  • On the other hand, 41% of the participants in the AI@50 conference (in 2006)stated那个机器智能将绝不达到人类水平。

See also:

2。9。Might an intelligence explosion never occur?

Dreyfus[[35这是给予的and Penrose[[36这是给予的have argued that human cognitive abilities can’t be emulated by a computational machine. Searle[[37这是给予的和块[[38这是给予的argue that certain kinds of machines cannot have a mind (consciousness, intentionality, etc.). But these objections need not concern those who predict an intelligence explosion.[[27这是给予的

We can reply to Dreyfus and Penrose by noting that an intelligence explosion does not require an AI to be a classical computational system. And we can reply to Searle and Block by noting that an intelligence explosion does not depend on machines having consciousness or other properties of ‘mind’, only that it be able to solve problems better than humans can in a wide variety of unpredictable environments. As Edsger Dijkstra once said, the question of whether a machine can ‘really’ think is “no more interesting than the question of whether a submarine can swim.”

Others who are pessimistic about an intelligence explosion occurring within the next few centuries don’t have a specific objection but instead think there are hidden obstacles that will reveal themselves and slow or halt progress toward machine superintelligence.[[28这是给予的

Finally, a global catastrophe like nuclear war or a large asteroid impact could so damage human civilization that the intelligence explosion never occurs. Or, a stable and global totalitarianism could prevent the technological development required for an intelligence explosion to occur.[[59这是给予的

3。Consequences of an Intelligence Explosion

3.1。Why would great intelligence produce great power?

智力是强大的。[[60这是给予的[[20这是给予的有人可能会说：“情报与枪支不匹配，也不适合有很多钱的人”，但是枪支和金钱都是由情报产生的。如果不是为了我们的智慧，人类仍然会觅食萨凡纳食物。

在telligence is what caused humans to dominate the planet in the blink of an eye (on evolutionary timescales). Intelligence is what allows us to eradicate diseases, and what gives us the potential to eradicate ourselves with nuclear war. Intelligence gives us superior strategic skills, superior social skills, superior economic productivity, and the power of invention.

A machine with superintelligence would be able to hack into vulnerable networks via the internet, commandeer those resources for additional computing power, take over mobile machines connected to networks connected to the internet, use them to build additional machines, perform scientific experiments to understand the world better than humans can, invent quantum computing and nanotechnology, manipulate the social world better than we can, and do whatever it can to give itself more power to achieve its goals — all at a speed much faster than humans can respond to.

3.2.一个情报爆炸怎么可能有用吗?

A machine superintelligence, if programmed with the right motivations, could potentially solve all the problems that humans are trying to solve but haven’t had the ingenuity or processing speed to solve yet. A superintelligence might cure disabilities and diseases, achieve world peace, give humans vastly longer and healthier lives, eliminate food and energy shortages, boost scientific discovery and space exploration, and so on.

Furthermore, humanity faces several existential risks in the 21st century, including global nuclear war, bioweapons, superviruses, and more.[[56这是给予的A superintelligent machine would be more capable of solving those problems than humans are.

See also:

3.3。情报爆炸如何危险？

如果用错误的动机编程，机器可能对人类具有恶毒，并故意消灭我们的物种。更有可能的是，它可以通过最初看起来安全（并且易于编程）的动机来设计，但是通过将资源从维持人类生活到其他项目的资源重新分配，这是最能实现的（具有足够的力量）。[[55这是给予的As Yudkowsky55这是给予的As Yudkowsky写信，，，，“the AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.”

Since weak AIs with many different motivations could better achieve their goal by faking benevolence until they are powerful, safety testing to avoid this could be very challenging. Alternatively, competitive pressures, both economic and military, might lead AI designers to try to use other methods to control AIs with undesirable motivations. As those AIs became more sophisticated this could eventually lead to one risk too many.

Even a machine successfully designed with superficially benevolent motivations could easily go awry when it discovers implications of its decision criteria unanticipated by its designers. For example, a superintelligence programmed to maximize human happiness might find it easier to rewire human neurology so that humans are happiest when sitting quietly in jars than to build and maintain a utopian world that caters to the complex and nuanced whims of current human neurology.

See also:

4。Friendly AI

4。1。什么是友好的人工智能？

友好的人工智能（友好的人工智能或FAI）是一种对人类“友好”的人工智能，对人类具有好而不是坏影响。

人工智金宝博娱乐能研究人员继续用自己做出决定的机器取得进展，并且人们越来越意识到我们需要设计机器以安全和道德的行动。该研究计划金宝博娱乐有很多名称：“机器伦理”[[2这是给予的[[3这是给予的[[8这是给予的[[9这是给予的，“机器道德”[[11这是给予的，“人造道德”[[6这是给予的，“计算伦理”[[12这是给予的和“计算元伦理”[[7这是给予的，，，，‘friendly AI’[[1这是给予的，，，，and ‘robo-ethics’ or ‘robot ethics’.[[5这是给予的[[10这是给予的

最直接的问题可能是战场机器人。美国国防部签约了罗纳德·阿金（Ronald Arkin），设计了一个用于确保自主战场机器人道德行为的系统金宝博官方[[4这是给予的。The U.S. Congress has declared that a third of America’s ground systems must be robotic by 2025, and by 2030 the U.S. Air Force计划有一群鸟类大型飞行机器人一次半自治地运行数周。

But Friendly AI research is not concerned with battlefield robots or machine ethics in general. It is concerned with a problem of a much larger scale: designing AI that would remain safe and friendly after the intelligence explosion.

A machine superintelligence would be enormously powerful. Successful implementation of Friendly AI could mean the difference between a solar system of unprecedented happiness and a solar system in which all available matter has been converted into parts for achieving the superintelligence’s goals.

It must be noted that Friendly AI is a harder project than often supposed. As explored below, commonly suggested solutions for Friendly AI are likely to fail because of two features possessed by any superintelligence:

  1. Superpower：a superintelligent machine will have unprecedented powers to reshape reality, and therefore will achieve its goals with highly efficient methods that confound human expectations and desires.
  2. 文字：a superintelligent machine will make decisions based on the mechanisms it is designed with, not the hopes its designers had in mind when they programmed those mechanisms. It will act only on precise specifications of rules and values, and will do so in ways that need not respect the complexity and subtlety[[41这是给予的[[42这是给予的[[43这是给予的of what humans value. A demand like “maximize human happiness” sounds simple to us because it contains few words, but philosophers and scientists have failed for centuries to explain确切地这意味着什么，并且肯定没有将其转化为足够严格的AI程序员使用的形式。

See also:

4。2。我们可以期望超级智能机器的动机是什么？

除了在整个大脑模拟的情况下,is no reason to expect a superintelligent machine to have motivations anything like those of humans. Human minds represent a tiny dot in the vast space of all possible mind designs, and very different kinds of minds are unlikely to share to complex motivations unique to humans and other mammals.

无论其目标如何，超级智能都将倾向于指挥能够帮助其实现目标的资源，包括人类生活所依赖的能量和要素。由于对人类或其他可能的思维设计“内置”的智力的关注，这不会停止。相反，它将追求其特定的目标，并没有考虑到那种被称为那种特定种类的灵长类动物的问题homo sapiens。

There are, however, some basic instrumental motivations we can expect superintelligent machines to display, because they are useful for achieving its goals, no matter what its goals are. For example, an AI will ‘want’ to self-improve, to be optimally rational, to retain its original goals, to acquire resources, and to protect itself — because all these things help it achieve the goals with which it was originally programmed.

See also:

4.3。我们不能只将超级智能放在盒子里，无法访问互联网吗？

‘AI-boxing’ is a common suggestion: why not use a superintelligent machine as a kind of question-answering oracle, and never give it access to the internet or any motors with which to move itself and acquire resources beyond what we give it? There are several reasons to suspect that AI-boxing will not work in the long run:

  1. Whatever goals the creators designed the superintelligence to achieve, it will be more able to achieve those goals if given access to the internet and other means of acquiring additional resources. So, there will be tremendous temptation to “let the AI out of its box.”
  2. Preliminary experiments在AI-boxing不激发信心。少量的,erintelligence will generate far more persuasive techniques for getting humans to “let it out of the box” than we can imagine.
  3. If one superintelligence has been created, then other labs or even independent programmers will be only weeks or decades away from creating a second superintelligence, and then a third, and then a fourth. You cannot hope to successfully contain all superintelligences created around the world by hundreds of people for hundreds of different purposes.

4。4。我们不能只是为不伤害我们的超级智能编程吗？

科幻作家艾萨克·阿西莫夫（Isaac Asimov[[39这是给予的：（1）机器人可能不会伤害人类，或者通过无所作为，允许人类受到伤害，（2）机器人必须遵守人类给予的任何命令，除非这样的命令将与第一法律和（3）机器人必须保护自己的存在，只要这种保护与第一法律或第二法律不冲突。但是阿西莫夫的故事倾向于说明为什么这样的规则会出错。[[40这是给予的

尽管如此，我们是否可以将“约束”编程为超级智能，以防止它伤害我们吗？可能不是。

One approach would be to implement ‘constraints’ as rules or mechanisms that prevent a machine from taking actions that it would normally take to fulfill its goals: perhaps ‘filters’ that intercept and cancel harmful actions, or ‘censors’ that detect and suppress potentially harmful plans within a superintelligence.

这种限制，无论多么详尽，几乎都是出于简单的原因而失败的：它们使人类的设计技能抗击无关。超级智能将正确地将这些限制视为实现目标的障碍，并将竭尽所能去除或绕过它们。也许它将删除包含约束的源代码的部分。如果我们要通过添加另一个约束来阻止这一点，它可以创建没有写入其中的约束的新机器，或欺骗我们自己消除约束。人类的进一步限制似乎是无法接受的，但可能会因超级智能而击败。指望人类超越超级智能并不是可行的解决方案。

If constraints在之上goals are not feasible, could we put constraintsinside of目标？如果连续性的目标是避免对人类的伤害，那么去除这一约束的动机就不会动机，避免了我们上面指出的问题。不幸的是，“危害”的直观概念很难以超级智能使用时不会导致非常糟糕的结果的方式指定。如果根据人类的痛苦定义了“伤害”，那么超级智能就会重新织物，以免感到痛苦。如果根据挫败人类的欲望来定义“危害”，则可能会恢复人类的欲望。等等。

如果,而不是试图完全指定一个术语‘harm’, we decide to explicitly list all of the actions a superintelligence ought to avoid, we run into a related problem: human value iscomplex and subtle，，，，and it’s unlikely we can come up with a list of all the things we做n’twant a superintelligence to do. This would be like writing a recipe for a cake thatreads：“不要使用鳄梨。不要使用烤面包机。不要使用蔬菜……”等等。这样的列表永远不会足够长。

4.5。我们可以编程超级智能以最大程度地提高人类的愉悦或欲望满意吗？

Let’s consider the likely consequences of someutilitariandesigns for Friendly AI.

旨在最大程度地减少人类苦难的AI可能只是杀死所有人类：没有人类，没有人类苦难。[[44这是给予的[[45这是给予的

或者，考虑一种旨在最大化人类愉悦感的AI。它没有建立雄心勃勃的乌托邦，它可以满足数十亿年的复杂和苛刻的人类需求，而是可以通过将人类接线到Nozick的努力来更有效地实现其目标experience machines。或者，它可能会重新连接“喜欢”组件of the brain’sreward systemso that whichever hedonic hotspot[[48这是给予的用“愉悦的光泽”绘画感觉[[46这是给予的[[47这是给予的is wired to maximize pleasure when humans sit in jars. That would be an easier world for the AI to build than one that caters to the complex and nuanced set of world states currently painted with the pleasure gloss by most human brains.

同样，AI的动机是最大程度地提高客观的欲望满意度或报告的主观幸福感可能会重新连接人类神经病学，因此每当人类坐在罐子里时，两端就会实现。或者它可以杀死所有人类（和动物），并用从头开始的生物代替它们，以在坐在罐子里时获得客观的欲望满意度或主观幸福感。与维持迎合人类（和动物）欲望的复杂性的乌托邦社会相比，AI的任何选择都比维持乌托邦社会更容易。类似的问题困扰着其他功利主义AI设计。

It’s not just a problem of specifying goals, either. It is hard to predict how goals will change in a self-modifying agent. No current mathematical decision theory can process the decisions of a self-modifying agent.

所以，虽然可能是可能的to design a superintelligence that would do what we want, it’s harder than one might initially think.

4。6。我们可以通过机器学习教授超级智能吗？

Some have proposed[[49这是给予的[[50这是给予的[[51这是给予的[[52这是给予的我们通过基于案例的机器学习来教机器道德代码。基本思想是：人类法官将评估成千上万的行动，性格特征，欲望，法律或机构，视为具有不同程度的道德可接受性。然后，机器将找到这些情况之间的连接学the principles behind morality, such that it could apply those principles to determine the morality of new cases not encountered during its training. This kind of machine learning has already been used to design machines that can, for example, detect underwater mines[[53这是给予的after feeding the machine hundreds of cases of mines and not-mines.

机器学习并不是为友好AI提供简单的解决方案的原因。首先，当然，人类本身就道德和不道德的事情深表分歧。但是，即使可以使人类就所有培训案件达成共识，但至少仍然存在两个问题。

The first problem is that training on cases from our present reality may not result in a machine that will make correct ethical decisions in a world radically reshaped by superintelligence.

The second problem is that a superintelligence may generalize the wrong principles due to coincidental patterns in the training data.[[54这是给予的Consider the parable of the machine trained to recognize camouflaged tanks in a forest. Researchers take 100 photos of camouflaged tanks and 100 photos of trees. They then train the machine on 50 photos of each, so that it learns to distinguish camouflaged tanks from trees. As a test, they show the machine the remaining 50 photos of each, and it classifies each one correctly. Success! However, later tests show that the machine classifies additional photos of camouflaged tanks and trees poorly. The problem turns out to be that the researchers’ photos of camouflaged tanks had been taken on cloudy days, while their photos of trees had been taken on sunny days. The machine had learned to distinguish cloudy days from sunny days, not camouflaged tanks from trees.

因此，似乎值得信赖的友好AI设计必须涉及产生人类道德判断的基本过程的详细模型，而不仅仅是案件的相似之处。

See also:

4。7。What is Coherent Extrapolated Volition?

Eliezer Yudkowsky has proposed[[57这是给予的连贯的外推力作为解决友好AI设计的至少两个问题的解决方案：

  1. 人类价值的脆弱性：Yudkowsky写信“未来不是由一个目标系统金宝博官方detailed reliable inheritance from human morals and metamorals will contain almost nothing of worth.” The problem is that what humans value is complex and subtle, and difficult to specify. Consider the seemingly minor value ofnovelty。If a human-like value of novelty is not programmed into a superintelligent machine, it might explore the universe for valuable things up to a certain point, and then maximize the most valuable thing it finds (the exploration-exploitation tradeoff[[58这是给予的）— tiling the solar system with brains in vats wired into happiness machines, for example. When a superintelligence is in charge, you have to get its motivational system确切地rightin order tonot使未来不受欢迎。
  2. 人类价值观的所在地:想象一下,如果面临了友好的人工智能问题ancient Greeks, and they had programmed it with the most progressive moral values of their time. That would have led the world to a rather horrifying fate. But why should we think that humans have, in the 21st century, arrived at the apex of human morality? We can’t risk programming a superintelligent machine with the moral values we happen to hold today. But then, which moral values做we give it?

Yudkowskysuggeststhat we build a ‘seed AI’ to discover and then extrapolate the ‘coherent extrapolated volition’ of humanity:

在poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

The seed AI would use the results of this examination and extrapolation of human values to program the motivational system of the superintelligence that would determine the fate of the galaxy.

However, some worry that the collective will of humanity won’t converge on a coherent set of goals. Othersbelieve即使通过如此精心而谨慎的手段，这种保证的友善是不可能的。

4.8。我们可以为任何人工智能设计增添友善吗？

许多会产生情报爆炸的AI设计将不会有一个“插槽”，其中一个目标（例如“对人类利益友好”）可以实现。例如，如果通过全脑仿真或进化算法或神经网或增强学习来制作AI，则AI最终会以某种目标为目标，但是由于它是自我提高的，但是最终的目标可能很难预测进步。

Thus, in order to design a friendly AI, it is not sufficient to determine what ‘friendliness’ is (and to specify it clearly enough that even a superintelligence will interpret it the way we want it to). We must also figure out how to build a general intelligence that satisfies a goal at all, and that stably retains that goal as it edits its own code to make itself smarter. This task is perhaps the primary difficulty in designing friendly AI.

4.9。谁在处理友好的AI问题？

Today, Friendly AI research is being explored by theMachine Intelligence Research Institute（（in Berkeley, California), by the人类研究所的未来（在英国牛津）以及其他一些研究人员，例如大卫·查尔默斯（David Chal金宝博娱乐mers）。机器道德研究人员偶尔会涉及该问金宝博娱乐题，例如Wendell Wallach和Colin AllenMoral Machines。

References

[1] Yudkowsky（2001）。Creating Friendly AI 1.0。机器智能研究所。金宝博娱乐

[2] Anderson＆Anderson编辑。（2006）。IEEE Intelligent Systems, 21(4)。

[[3这是给予的Anderson & Anderson, eds. (2011).Machine Ethics。Cambridge University Press.

[[4这是给予的Arkin (2009).Governing Lethal Behavior in Autonomous Robots。Chapman and Hall.

[[5这是给予的Capurro, Hausmanninger, Weber, Weil, Cerqui, Weber, & Weber (2006).在ternational Review of Information Ethics, Vol. 6: Ethics in Robots。

[[6这是给予的Danielson (1992).Artificial morality: Virtuous robots for virtual games。Routledge.

[[7这是给予的Lokhorst (2011).Computational meta-ethics: Towards the meta-ethical robot。Minds and Machines。

[[8这是给予的McLaren (2005). Lessons in Machine Ethics from the Perspective of Two Computational Models of Ethical Reasoning.AAAI Technical Report FS-05-06：70-77.

[[9这是给予的Powers (2005). Deontological Machine Ethics.AAAI Technical Report FS-05-06：79-86。

[10] Sawyer（2007）。机器人伦理。Science, 318(5853)：1037。

[11] Wallach，Allen和Smit（2008）。机器道德：自下而上和自上而下的方法，用于建模人类道德能力。AI and Society, 22(4)：565–582.

[12]艾伦（2002）。计算的道德：限制的道德计算。在Smit＆Lasker编辑中，决策和人类行动的认知，情感和道德方面，第一卷。巴登/伊亚斯。

[13] Good（1965）。Speculations concerning the first ultraintelligent machine。Advanced in Computers, 6：31-88。

[[14这是给予的MacKenzie (1995). The Automation of Proof: A Historical and Sociological Exploration.IEEE Annals，17（3）：7-29.

[[15这是给予的Nilsson (2009).The Quest for Artificial Intelligence。Cambridge University Press.

[16] Campbell，Hoane和Hsu（2002）。深蓝。Artificial Intelligence, 134：57-83.

[[17这是给予的Markoff (2011).电脑在“危险！”上获胜；琐碎，不是。New York Times, February 17th 2011：A1.

[[18这是给予的King et al. (2009).科学的自动化。Science, 324：85-89.

[[19这是给予的King (2011).Rise of the robo scientists。科学美国人，2011年1月。

[[20] Legg (2008).Machine Super Intelligence。博士论文。IDSIA。

[21] Hutter（2005）。Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability。施普林格。

[[22这是给予的Veness, Ng, Hutter, & Silver (2011).A Monte Carlo AIXI Approximation。Journal of Artificial Intelligence Research, 40：95-142.

[[23这是给予的Tang, Shimizu, Dube, Rampon, Kerchner, Zhuo, Liu, & Tsien (1999). Genetic enhancement of learning and memory in mice.Nature, 401：63–69。

[[24这是给予的Hochberg, Serruya, Friehs, Mukand, Saleh, Caplan, Branner, Chen, Penn, & Donoghue (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia.Nature 442：164-171.

[25] Bostrom（1998）。超级智能多长时间？在ternational Journal of Future Studies, 2。

[[26这是给予的Kurzweil (2005).The Singularity is Near。Viking.

[[27这是给予的Chalmers (2010).奇异性：哲学分析。意识研究杂志，17：7-65.

[[28这是给予的Baum, Goertzel, & Goertzel (forthcoming).直到人级AI多久？专家评估的结果。技术和预测变革。

[[29这是给予的Grossberg (1992).Neural Networks and Natural Intelligence。MIT Press.

[[30] Martinetz & Schulten (1991). A ‘neural-gas’ network learns topologies. In Kohonen, Makisara, Simula, & Kangas (eds.),人工神经网络（（pp. 397-402). North Holland.

[[31这是给予的de Garis (2010). Artificial Brains. In Goertzel & Pennachin (eds.),Artificial General Intelligence（第159-174页）。施普林格。

[[32这是给予的Schmidhuber (2010). Gödel Machines: Fully Self-Referential Optimal Universal Self-Improvers. In Goertzel & Pennachin (eds.),Artificial General Intelligence（第199-223页）。施普林格。

[[33这是给予的Hutter (2010). Universal Algorithmic Intelligence: A Mathematical Top-Down Approach. In Goertzel & Pennachin (eds.),Artificial General Intelligence（（pp. 227-287). Springer.

[34] Yudkowsky（2010）。Levels of Organization in General Intelligence。在Goertzel＆Pennachin（编辑）中，Artificial General Intelligence（（pp. 389-496). Springer.

[35] Dreyfus（1972）。What Computers Can’t Do。Harper & Row.

[[36这是给予的Penrose (1994).Shadows of the Mind。牛津大学出版社。

[37] Searle（1980）。Minds, brains, and programs。Behavioral and Brain Sciences, 3：417-457。

[[38这是给予的Block (1981).心理和行为主义。Philosophical Review, 90：5-43.

[[39这是给予的Asimov (1942).Runaround。Astounding Science Fiction, March 1942。Street & Smith.

[[40] Anderson (2008). Asimov’s ‘three laws of robotics’ and machine metaethics.AI & Society, 22(4)：477-493。

[41] Kringelbach＆Berridge编辑。（2009）。Pleasures of the Brain。牛津大学出版社。

[[42这是给予的Schroeder (2004).欲望的三张面孔。牛津大学出版社。

[[43这是给予的Yudkowsky (2007).The hidden complexity of wishes。

[[44这是给予的Smart (1958). Negative utilitarianism.智力，67：542-543.

[[45这是给予的Russell & Norvig (2009).Artificial Intelligence: A Modern Approach, 3rd edition。Prentice Hall. (see page 1037)

[[46这是给予的Frijda (2009). On the nature and function of pleasure. In Kringelbach & Berridge (eds.),大脑的乐趣（（pp. 99-112). Oxford University Press.

[47] Aldridge＆Berridge（2009）。愉悦的神经编码：腹侧苍白的“玫瑰色眼镜”。在Kringelbach＆Berridge（编辑）中，大脑的乐趣（第62-73页）。牛津大学出版社。

[[48这是给予的Smith, Mahler, Pecina, & Berridge (2009). Hedonic hotspots: generating sensory pleasure in the brain. In Kringelbach & Berridge (eds.),大脑的乐趣（第27-49页）。牛津大学出版社。

[[49这是给予的Guarini, (2006). Particularism and classification and reclassification of moral cases. IEEE Intelligent Systems 21(4): 22–28.

[50] Anderson，Anderson和Armen（2005）。迈向机器道德：实施两种基于动作的伦理理论。在2005年AAAI的机器道德专题讨论会论文集，阿灵顿，弗吉尼亚，十一月。

[[51这是给予的Honarvar & Ghasem-Aghaee (2009). An artificial neural network approach for creating an ethical artificial agent.第八届IEEE机器人技术计算智能国际会议论文集：290-295.

[[52这是给予的Rzepka & Araki (2005). What statistics could do for ethics? – The idea of common sense processing based safety valve. InMachine ethics: papers from the AAAI fall symposium。American Association of Artificial Intelligence.

[[53这是给予的Gorman & Sejnowski (1988).在经过培训的分层网络中对隐藏单元进行分析以对声纳目标进行分类。Neural Networks, 1：75-89.

[[54这是给予的Yudkowsky (2008).Artificial intelligence as a positive and negative factor in global risk。在Bostrom & Cirkovic (eds.),Global Catastrophic Risks。牛津大学出版社。

[55] Omohundro（2008）。基本的AI驱动器。

[[56这是给予的Bostrom & Cirkovic, eds. (2008).Global Catastrophic Risks。牛津大学出版社。

[[57这是给予的Yudkowsky (2004).连贯的外推力。机器智能研究所。金宝博娱乐

[[58这是给予的Azoulay-Schwartz, Kraus, & Wilkenfeld (2004). Exploitation vs. exploration: choosing a supplier in an environment of incomplete information.Decision Support Systems, 38：1-18.

[[59这是给予的Caplan (2008). The totalitarian threat. In Bostrom & Cirkovic (eds.),Global Catastrophic Risks。牛津大学出版社。

[[60] Yudkowsky (2007).The Power of Intelligence。

[[61这是给予的Bainbridge (2005). Survey of NBIC Applications. In Bainbridge & Roco (eds.),Managing nano-bio-info-cogno innovations: Converging technologies in society。施普林格。

[[62这是给予的Sandberg & Bostrom (2011).机器智能调查，，，，Technical Report 2011-1, Future of Humanity Institute, Oxford.

Written by卢克·穆尔豪瑟（Luke Muehlhauser）。

This page is up-to-date as of 2013, but may not represent MIRI or Luke Muehlhauser’s current views. Last modified November 10, 2015 (original）。