Data Ethics

数据伦理

Sidebar: Acknowledgement: Dr. Rachel Thomas

###侧边栏：致谢：Rachel Thomas博士

This chapter was co-authored by Dr. Rachel Thomas, the cofounder of fast.ai, and founding director of the Center for Applied Data Ethics at the University of San Francisco. It largely follows a subset of the syllabus she developed for the Introduction to Data Ethics course.

本章由fast.ai联合创始人、旧金山大学应用数据伦理中心创始主任Rachel Thomas博士共同撰写。本章很大程度上遵循了她为数据伦理导论课程开发的教学大纲的一个子集。

End sidebar

###结束侧边栏

As we discussed in Chapters 1 and 2, sometimes machine learning models can go wrong. They can have bugs. They can be presented with data that they haven't seen before, and behave in ways we don't expect. Or they could work exactly as designed, but be used for something that we would much prefer they were never, ever used for.

Because deep learning is such a powerful tool and can be used for so many things, it becomes particularly important that we consider the consequences of our choices. The philosophical study of ethics is the study of right and wrong, including how we can define those terms, recognize right and wrong actions, and understand the connection between actions and consequences. The field of data ethics has been around for a long time, and there are many academics focused on this field. It is being used to help define policy in many jurisdictions; it is being used in companies big and small to consider how best to ensure good societal outcomes from product development; and it is being used by researchers who want to make sure that the work they are doing is used for good, and not for bad.

As a deep learning practitioner, therefore, it is likely that at some point you are going to be put in a situation where you need to consider data ethics. So what is data ethics? It's a subfield of ethics, so let's start there.

正如我们在第1章和第2章中所讨论的，机器学习模型有时也会出错。它们可能有bugs。它们可以用以前没见过的数据来呈现，并以我们不期望的方式运行。或者它们可以完全按照设计的那样工作，但被用于一些我们不希望它们被用于的事情。

因为深度学习是一个非常强大的工具，可以用于很多事情，所以考虑我们选择的后果就变得尤为重要。伦理学的哲学研究是对与错的研究，包括我们如何定义这些术语，识别对与错的行为，以及理解行为与后果之间的联系。数据伦理领域已经存在了很长一段时间，有很多学者关注这个领域。它正被用于帮助确定许多司法管辖区的政策;它被大大小小的公司用来考虑如何最好地确保产品开发产生良好的社会结果;研究人员正在使用它，他们希望确保他们所做的工作是用于好的方面，而不是坏的方面。

因此，作为一名深度学习从业者，很可能在某些时候会遇到需要考虑数据伦理的情况。那么什么是数据伦理呢?这是道德的一个子领域，我们就从这里开始吧。

J: At university, philosophy of ethics was my main thing (it would have been the topic of my thesis, if I'd finished it, instead of dropping out to join the real world). Based on the years I spent studying ethics, I can tell you this: no one really agrees on what right and wrong are, whether they exist, how to spot them, which people are good, and which bad, or pretty much anything else. So don't expect too much from the theory! We're going to focus on examples and thought starters here, not theory.

J：在大学里，伦理哲学是我的主要工作(如果我完成了它，而不是辍学进入现实世界，它可能会成为我论文的主题)。基于我研究伦理学的那些年，我可以告诉你:对于什么是对什么是错，它们是否存在，如何识别它们，哪些人是好的，哪些人是坏的，或者几乎所有其他的事情，没有人真正达成一致。所以不要对理论抱有太大的期望!我们将在这里关注示例和思想开端，而不是理论。

In answering the question "What Is Ethics", The Markkula Center for Applied Ethics says that the term refers to:

• Well-founded standards of right and wrong that prescribe what humans ought to do
• The study and development of one's ethical standards.

There is no list of right answers. There is no list of do and don't. Ethics is complicated, and context-dependent. It involves the perspectives of many stakeholders. Ethics is a muscle that you have to develop and practice. In this chapter, our goal is to provide some signposts to help you on that journey.

Spotting ethical issues is best to do as part of a collaborative team. This is the only way you can really incorporate different perspectives. Different people's backgrounds will help them to see things which may not be obvious to you. Working with a team is helpful for many "muscle-building" activities, including this one.

This chapter is certainly not the only part of the book where we talk about data ethics, but it's good to have a place where we focus on it for a while. To get oriented, it's perhaps easiest to look at a few examples. So, we picked out three that we think illustrate effectively some of the key topics.

在回答“什么是伦理”的问题时，马库拉应用伦理中心表示，该术语指的是：

• 有充分根据的是非标准，规定了人类应该做什么
• 一个人道德标准的研究和发展。

没有正确答案的清单。没有做和不做的清单。伦理是复杂的，而且与环境有关。它涉及到许多利益相关者的观点。道德是一块肌肉，你必须发展和练习。在本章中，我们的目标是提供一些路标来帮助你完成这一旅程。

发现道德问题最好是作为一个协作团队的一部分来做。这是你能真正融合不同观点的唯一方法。不同的人的背景会帮助他们看到对你来说可能不明显的事情。与团队合作对许多“肌肉锻炼”活动都是有帮助的，包括这项活动。

这一章当然不是本书中我们讨论数据伦理的唯一部分，但有一个地方让我们暂时关注它是很好的。为了获得方向性，看几个例子可能是最容易的。因此，我们挑选了三个我们认为能有效说明一些关键主题的例子。

Key Examples for Data Ethics

数据伦理的关键例子

We are going to start with three specific examples that illustrate three common ethical issues in tech:

1. Recourse processes—Arkansas's buggy healthcare algorithms left patients stranded.
2. Feedback loops—YouTube's recommendation system helped unleash a conspiracy theory boom.
3. Bias—When a traditionally African-American name is searched for on Google, it displays ads for criminal background checks.

In fact, for every concept that we introduce in this chapter, we are going to provide at least one specific example. For each one, think about what you could have done in this situation, and what kinds of obstructions there might have been to you getting that done. How would you deal with them? What would you look out for?

我们将从三个具体的例子开始，说明技术中三个常见的道德问题：

1. 追索过程——阿肯色州漏洞百出的医疗保健算法让病人束手无策。
2. 反馈循环——YouTube的推荐系统帮助引发了一场阴谋论热潮。
3. 偏见——当在谷歌上搜索一个传统的非裔美国人名字时，它会显示用于犯罪背景调查广告。

事实上，对于我们在本章中介绍的每个概念，我们将提供至少一个具体的例子。对于每一个，想想在这种情况下你可以做什么，以及在你完成这一任务时可能会遇到哪些障碍。你会如何处理它们？你会注意什么？

Bugs and Recourse: Buggy Algorithm Used for Healthcare Benefits

错误和追索权：用于医疗保健福利的错误算法

The Verge investigated software used in over half of the US states to determine how much healthcare people receive, and documented their findings in the article "What Happens When an Algorithm Cuts Your Healthcare". After implementation of the algorithm in Arkansas, hundreds of people (many with severe disabilities) had their healthcare drastically cut. For instance, Tammy Dobbs, a woman with cerebral palsy who needs an aid to help her to get out of bed, to go to the bathroom, to get food, and more, had her hours of help suddenly reduced by 20 hours a week. She couldn’t get any explanation for why her healthcare was cut. Eventually, a court case revealed that there were mistakes in the software implementation of the algorithm, negatively impacting people with diabetes or cerebral palsy. However, Dobbs and many other people reliant on these healthcare benefits live in fear that their benefits could again be cut suddenly and inexplicably.

The Verge调查了美国超过一半的州使用的软件，以确定人们接受了多少医疗保健，并将他们的发现记录在"当算法削减你的医疗保健时会发生什么"一文中。在阿肯色州实施该算法后，数百人(许多患有严重残疾)的医疗费用大幅削减。例如，一名患有脑瘫的妇女Tammy Dobbs，需要护理帮助她起床、去洗手间、拿食物等等，她每周的帮助时间突然减少了20个小时。 她无法解释为什么她的医疗保健被削减了。最终，法院的案件显示，该算法的软件实现存在错误，对糖尿病或脑瘫患者产生了负面影响。然而，Dobbs和许多依赖这些医疗福利的人在担心他们的福利可能会再次突然被莫名其妙地削减。

Feedback Loops: YouTube's Recommendation System

反馈循环：YouTube的推荐系统

Feedback loops can occur when your model is controlling the next round of data you get. The data that is returned quickly becomes flawed by the software itself.

For instance, YouTube has 1.9 billion users, who watch over 1 billion hours of YouTube videos a day. Its recommendation algorithm (built by Google), which was designed to optimize watch time, is responsible for around 70% of the content that is watched. But there was a problem: it led to out-of-control feedback loops, leading the New York Times to run the headline "YouTube Unleashed a Conspiracy Theory Boom. Can It Be Contained?". Ostensibly recommendation systems are predicting what content people will like, but they also have a lot of power in determining what content people even see.

当你的模型控制你得到的下一轮数据时，可能会出现反馈循环。快速返回的数据会因软件本身而存在缺陷。

例如，YouTube拥有19亿用户，他们每天观看超过10亿小时的YouTube视频。它的推荐算法(由谷歌构建)是为了优化观看时间而设计的，负责大约 70% 的观看内容。但有一个问题:它导致了失控的反馈循环，导致纽约时报刊登了标题“YouTube 引发了一场阴谋论热潮。它可以被遏制吗？”。表面上看，推荐系统是在预测人们会喜欢什么内容，但在决定人们会看到什么内容方面，它们也有很大的影响力。

Bias: Professor Latanya Sweeney "Arrested"

偏见：Latanya Sweeney教授“被捕”

Dr. Latanya Sweeney is a professor at Harvard and director of the university's data privacy lab. In the paper "Discrimination in Online Ad Delivery" (see <<latanya_arrested>>) she describes her discovery that Googling her name resulted in advertisements saying "Latanya Sweeney, arrested?" even though she is the only known Latanya Sweeney and has never been arrested. However when she Googled other names, such as "Kirsten Lindquist," she got more neutral ads, even though Kirsten Lindquist has been arrested three times.

Latanya Sweeney博士是哈佛大学的教授，也是该校数据隐私实验室的主任。在论文“在线广告投递中的歧视”(见<<latanya_arrested>>)中，她描述了她的发现，在谷歌上搜索她的名字会导致广告上写着“Latanya Sweeney，被捕?”即使她是唯一已知的Latanya Sweeney并且从未被捕过。然而，当她在谷歌上搜索其他名字时，比如“Kirsten Lindquist”，她得到的广告更加中性，尽管Kirsten Lindquist已经被逮捕了三次。

Being a computer scientist, she studied this systematically, and looked at over 2000 names. She found a clear pattern where historically Black names received advertisements suggesting that the person had a criminal record, whereas, white names had more neutral advertisements.

This is an example of bias. It can make a big difference to people's lives—for instance, if a job applicant is Googled it may appear that they have a criminal record when they do not.

作为一名计算机科学家，她系统地研究了这个问题，研究了2000多个名字。她发现了一个明显的规律，从历史上看，黑人名字收到的广告暗示这个人有犯罪记录，而白人名字收到的广告更中性。

这是偏见的一个例子。它可以对人们的生活产生很大的影响——例如，如果在谷歌上搜索一个求职者，可能会显示他有犯罪记录，而实际上并没有。

Why Does This Matter?

为什么这很重要？

One very natural reaction to considering these issues is: "So what? What's that got to do with me? I'm a data scientist, not a politician. I'm not one of the senior executives at my company who make the decisions about what we do. I'm just trying to build the most predictive model I can."

These are very reasonable questions. But we're going to try to convince you that the answer is that everybody who is training models absolutely needs to consider how their models will be used, and consider how to best ensure that they are used as positively as possible. There are things you can do. And if you don't do them, then things can go pretty badly.

One particularly hideous example of what happens when technologists focus on technology at all costs is the story of IBM and Nazi Germany. In 2001, a Swiss judge ruled that it was not unreasonable "to deduce that IBM's technical assistance facilitated the tasks of the Nazis in the commission of their crimes against humanity, acts also involving accountancy and classification by IBM machines and utilized in the concentration camps themselves."

IBM, you see, supplied the Nazis with data tabulation products necessary to track the extermination of Jews and other groups on a massive scale. This was driven from the top of the company, with marketing to Hitler and his leadership team. Company President Thomas Watson personally approved the 1939 release of special IBM alphabetizing machines to help organize the deportation of Polish Jews. Pictured in <> is Adolf Hitler (far left) meeting with IBM CEO Tom Watson Sr. (second from left), shortly before Hitler awarded Watson a special “Service to the Reich” medal in 1937.

考虑这些问题的一个很自然的反应是:“那又怎样?这跟我有什么关系?我是数据科学家，不是政客。我不是公司里负责决定我们做什么的高管之一。我只是想建立我所能建立的最具预测性的模型。”

这些问题都很合理。但我们将试图说服你，答案是每个训练模型的人绝对需要考虑他们的模型将如何被使用，并考虑如何最好地确保它们被尽可能积极地使用。有些事情你可以做。如果你不这样做，事情会变得很糟糕。

当技术专家不惜一切代价关注技术时会发生什么，一个特别可怕的例子是IBM和纳粹德国的故事。2001年，一名瑞士法官裁定，“推断IBM的技术援助为纳粹犯下危害人类罪的任务提供了便利，这种行为也涉及IBM机器的会计和分类，并被集中营本身使用。”

你看，IBM为纳粹提供了追踪大规模屠杀犹太人和其他群体所必需的数据表格产品。这是由公司高层推动的，向希特勒和他的领导团队进行营销。公司总裁Thomas Watson亲自批准了IBM在1939年发布特殊的字母排序机，以帮助组织驱逐波兰犹太人。图为Adolf Hitler(左一)与IBM首席执行官老Tom Watson(左二)会面，不久之后，Hitler于1937年授予Watson特别的“为帝国服务”奖章。

But this was not an isolated incident—the organization's involvement was extensive. IBM and its subsidiaries provided regular training and maintenance onsite at the concentration camps: printing off cards, configuring machines, and repairing them as they broke frequently. IBM set up categorizations on its punch card system for the way that each person was killed, which group they were assigned to, and the logistical information necessary to track them through the vast Holocaust system. IBM's code for Jews in the concentration camps was 8: some 6,000,000 were killed. Its code for Romanis was 12 (they were labeled by the Nazis as "asocials," with over 300,000 killed in the Zigeunerlager, or “Gypsy camp”). General executions were coded as 4, death in the gas chambers as 6.

但这不是一个孤立的事件——该组织的参与范围很广。IBM及其子公司在集中营提供定期培训和现场维护：打印卡片、配置机器并在机器经常损坏时进行维修。IBM在其穿孔卡片系统上对每个人的被杀方式、他们被分配到哪个群体以及在庞大的大屠杀系统中追踪他们所需的后勤信息进行分类。IBM集中营中犹太人的编码是8：约600万人被杀。它对吉普赛人的编码是12（他们被纳粹称为“异教徒”，超过30万人在Zigeunerlager或“吉普赛集中营”被杀）。一般处决被编码为4，毒气室死亡被编码为6。

Of course, the project managers and engineers and technicians involved were just living their ordinary lives. Caring for their families, going to the church on Sunday, doing their jobs the best they could. Following orders. The marketers were just doing what they could to meet their business development goals. As Edwin Black, author of IBM and the Holocaust (Dialog Press) observed: "To the blind technocrat, the means were more important than the ends. The destruction of the Jewish people became even less important because the invigorating nature of IBM's technical achievement was only heightened by the fantastical profits to be made at a time when bread lines stretched across the world."

Step back for a moment and consider: How would you feel if you discovered that you had been part of a system that ended up hurting society? Would you be open to finding out? How can you help make sure this doesn't happen? We have described the most extreme situation here, but there are many negative societal consequences linked to AI and machine learning being observed today, some of which we'll describe in this chapter.

It's not just a moral burden, either. Sometimes technologists pay very directly for their actions. For instance, the first person who was jailed as a result of the Volkswagen scandal, where the car company was revealed to have cheated on its diesel emissions tests, was not the manager that oversaw the project, or an executive at the helm of the company. It was one of the engineers, James Liang, who just did what he was told.

Of course, it's not all bad—if a project you are involved in turns out to make a huge positive impact on even one person, this is going to make you feel pretty great!

Okay, so hopefully we have convinced you that you ought to care. But what should you do? As data scientists, we're naturally inclined to focus on making our models better by optimizing some metric or other. But optimizing that metric may not actually lead to better outcomes. And even if it does help create better outcomes, it almost certainly won't be the only thing that matters. Consider the pipeline of steps that occurs between the development of a model or an algorithm by a researcher or practitioner, and the point at which this work is actually used to make some decision. This entire pipeline needs to be considered as a whole if we're to have a hope of getting the kinds of outcomes we want.

Normally there is a very long chain from one end to the other. This is especially true if you are a researcher, where you might not even know if your research will ever get used for anything, or if you're involved in data collection, which is even earlier in the pipeline. But no one is better placed to inform everyone involved in this chain about the capabilities, constraints, and details of your work than you are. Although there's no "silver bullet" that can ensure your work is used the right way, by getting involved in the process, and asking the right questions, you can at the very least ensure that the right issues are being considered.

Sometimes, the right response to being asked to do a piece of work is to just say "no." Often, however, the response we hear is, "If I don’t do it, someone else will." But consider this: if you’ve been picked for the job, you’re the best person they’ve found to do it—so if you don’t do it, the best person isn’t working on that project. If the first five people they ask all say no too, even better!

当然，项目经理、工程师和技术人员只是过着普通的生活。照顾家人，周日去教堂，尽他们所能做好他们的工作。服从命令。营销人员只是尽其所能来实现他们的业务发展目标。正如IBM与大屠杀(对话出版社)的作者Edwin Black 所说:“对盲目的技术官僚来说，手段比目的更重要。犹太人的毁灭变得不那么重要，因为IBM 技术成就的令人振奋的本质是只有在面包生产线遍布全球的时候，才能获得惊人的利润。”

退一步想想:如果你发现自己是一个最终伤害社会体系的一部分，你会有什么感觉?你愿意去看看吗?你怎样才能确保这种情况不会发生呢?我们在这里描述了最极端的情况，但有许多负面的社会后果与人工智能和机器学习有关，我们将在本章中描述其中一些。

这不仅仅是一种道德负担。有时技术人员为他们的行为付出非常直接的代价。例如，大众汽车被曝在柴油排放测试中作弊的丑闻，导致第一个入狱的人不是负责监督该项目的经理，也不是公司掌舵人。正是其中一位工程师James Liang，他只是做了他被告知的要求。

当然，这也不全是坏事——如果你参与的项目最终对一个人产生了巨大的积极影响，这将会让你感觉非常棒!

好的，希望我们已经说服了你，你应该关心。但是你该怎么做呢?作为数据科学家，我们自然倾向于通过优化某些指标使我们的模型更好。但优化这个指标实际上可能不会带来更好的结果。即使它确实有助于创造更好的结果，它也几乎肯定不是唯一重要的事情。考虑一下研究者或从业者开发模型或算法之间的步骤管道，以及这项工作实际用于做出某些决定的时间点。如果我们希望得到我们想要的结果，整个管道需要作为一个整体来考虑。

通常从一端到另一端有一条很长的链。如果你是一名研究人员，这一点尤其正确，因为你可能甚至不知道你的研究是否会被用于任何事情，或者你是否参与了数据收集，这甚至是在管道的早期阶段。但是没有人比您更适合将您工作的能力、限制和细节告知这条链中的每个人。虽然没有什么“灵丹妙药”可以确保你的工作被正确地使用，但通过参与到这个过程中，并提出正确的问题，你至少可以确保你考虑到了正确的问题。

有时候，对被要求做一项工作的正确回应就是说“不”。然而，我们经常听到的回答是:“如果我不做，别人会做的。”但请考虑一下:如果你被选中了，你就是他们找到的最适合这份工作的人——所以如果你不做，那么最佳人选就不会从事该项目。如果他们问的前五个人也都说不，那就更好了!

Integrating Machine Learning with Product Design

将机器学习与产品设计集成

Presumably the reason you're doing this work is because you hope it will be used for something. Otherwise, you're just wasting your time. So, let's start with the assumption that your work will end up somewhere. Now, as you are collecting your data and developing your model, you are making lots of decisions. What level of aggregation will you store your data at? What loss function should you use? What validation and training sets should you use? Should you focus on simplicity of implementation, speed of inference, or accuracy of the model? How will your model handle out-of-domain data items? Can it be fine-tuned, or must it be retrained from scratch over time?

These are not just algorithm questions. They are data product design questions. But the product managers, executives, judges, journalists, doctors… whoever ends up developing and using the system of which your model is a part will not be well-placed to understand the decisions that you made, let alone change them.

For instance, two studies found that Amazon’s facial recognition software produced inaccurate and racially biased results. Amazon claimed that the researchers should have changed the default parameters, without explaining how this would have changed the biased results. Furthermore, it turned out that Amazon was not instructing police departments that used its software to do this either. There was, presumably, a big distance between the researchers that developed these algorithms and the Amazon documentation staff that wrote the guidelines provided to the police. A lack of tight integration led to serious problems for society at large, the police, and Amazon themselves. It turned out that their system erroneously matched 28 members of congress to criminal mugshots! (And the Congresspeople wrongly matched to criminal mugshots were disproportionately people of color, as seen in <>.)

大概你做这项工作的原因是你希望它会被用于某些事情。否则，你只是在浪费时间。所以，让我们先假设你的工作最终会在某个地方结束。现在，当你收集你的数据和开发你的模型时，你要做很多决定。你将把数据存储在什么级别？你应该使用什么损失函数？你应该使用什么验证和训练集？你应该关注实现的简单性、推理的速度还是模型的准确性？你的模型将如何处理域外数据项？它可以微调，还是必须随着时间的推移从头开始重新训练？

这不仅仅是算法问题。它们是数据产品设计问题。但是产品经理、高管、法官、记者、医生……无论谁最终开发和使用你的模型所在的系统，都无法很好地理解你所做的决定，更不用说改变它们了。

例如，两项研究发现，亚马逊的面部识别软件产生的结果不准确，而且带有种族偏见。亚马逊声称，研究人员应该改变默认参数，但没有解释这将如何改变有偏差的结果。此外，事实证明，亚马逊并没有指示警察部门使用它的软件来做到这一点。据推测，开发这些算法的研究人员与撰写提供给警方的指南的亚马逊文档工作人员之间存在很大的差距。缺乏紧密的整合给整个社会、警察和亚马逊本身带来了严重的问题。原来，他们的系统错误地将28名国会议员的照片与犯罪嫌疑人的照片相匹配!(与罪犯面部照片匹配错误的国会议员不成比例地是有色人种，如<>所示。)

Data scientists need to be part of a cross-disciplinary team. And researchers need to work closely with the kinds of people who will end up using their research. Better still is if the domain experts themselves have learned enough to be able to train and debug some models themselves—hopefully there are a few of you reading this book right now!

The modern workplace is a very specialized place. Everybody tends to have well-defined jobs to perform. Especially in large companies, it can be hard to know what all the pieces of the puzzle are. Sometimes companies even intentionally obscure the overall project goals that are being worked on, if they know that their employees are not going to like the answers. This is sometimes done by compartmentalising pieces as much as possible.

In other words, we're not saying that any of this is easy. It's hard. It's really hard. We all have to do our best. And we have often seen that the people who do get involved in the higher-level context of these projects, and attempt to develop cross-disciplinary capabilities and teams, become some of the most important and well rewarded members of their organizations. It's the kind of work that tends to be highly appreciated by senior executives, even if it is sometimes considered rather uncomfortable by middle management.

数据科学家需要成为跨学科团队的一员。研究人员需要与最终会使用他们的研究的人密切合作。如果领域专家自己学到了足够的知识，能够自己训练和调试一些模型，那就更好了——希望你们中的一些人现在正在阅读这本书！

现代工作场所是一个非常专业化的地方。每个人往往都有明确的工作要做。尤其是在大公司，很难知道拼图的所有部分是什么。有时，如果公司知道员工不会喜欢最终结果，他们甚至会故意模糊正在进行的整体项目目标。有时，这是通过尽可能划分碎片来实现的。

换句话说，我们并不是说这一切都很容易。这很难。真的很难。我们都必须尽力而为。我们经常看到，那些参与这些项目的高层背景，并试图发展跨学科能力和团队的人，成为了他们组织中最重要、回报最高的成员。这种工作往往会得到高级管理人员的高度赞赏，即使有时中层管理人员认为这相当不舒服。

Topics in Data Ethics

数据伦理主题

Data ethics is a big field, and we can't cover everything. Instead, we're going to pick a few topics that we think are particularly relevant:

• The need for recourse and accountability
• Feedback loops
• Bias
• Disinformation

数据伦理是一个大领域，我们不能涵盖所有内容。相反，我们将选择一些我们认为特别相关的主题：

• 追索权和问责制的必要性
• 反馈循环
• 偏差
• 造谣

Let's look at each in turn.

让我们依次看看。

Recourse and Accountability

追索权和问责制

In a complex system, it is easy for no one person to feel responsible for outcomes. While this is understandable, it does not lead to good results. In the earlier example of the Arkansas healthcare system in which a bug led to people with cerebral palsy losing access to needed care, the creator of the algorithm blamed government officials, and government officials blamed those who implemented the software. NYU professor Danah Boyd described this phenomenon: "Bureaucracy has often been used to shift or evade responsibility... Today's algorithmic systems are extending bureaucracy."

An additional reason why recourse is so necessary is because data often contains errors. Mechanisms for audits and error correction are crucial. A database of suspected gang members maintained by California law enforcement officials was found to be full of errors, including 42 babies who had been added to the database when they were less than 1 year old (28 of whom were marked as “admitting to being gang members”). In this case, there was no process in place for correcting mistakes or removing people once they’d been added. Another example is the US credit report system: in a large-scale study of credit reports by the Federal Trade Commission (FTC) in 2012, it was found that 26% of consumers had at least one mistake in their files, and 5% had errors that could be devastating. Yet, the process of getting such errors corrected is incredibly slow and opaque. When public radio reporter Bobby Allyn discovered that he was erroneously listed as having a firearms conviction, it took him "more than a dozen phone calls, the handiwork of a county court clerk and six weeks to solve the problem. And that was only after I contacted the company’s communications department as a journalist."

As machine learning practitioners, we do not always think of it as our responsibility to understand how our algorithms end up being implemented in practice. But we need to.

在一个复杂的系统中，没有一个人会觉得对结果负责是很容易的。虽然这是可以理解的，但它不会带来好的结果。在阿肯色州医疗保健系统的早期例子中，一个漏洞导致脑瘫患者无法获得所需的护理，算法的创造者指责公职人员，公职人员指责那些实施软件的人。纽约大学教授Danah Boyd这样描述这种现象:“官僚主义经常被用来转移或逃避责任……如今的算法系统正在扩大官僚主义。”

追索如此必要的另一个原因是，数据经常包含错误。审计和错误纠正机制至关重要。由加州执法官员维护的犯罪团伙嫌疑人数据库被发现错误百出，其中包括42名不到1岁的婴儿被添加到数据库(其中28人被标记为“承认是犯罪团伙成员”)。在这种情况下，没有适当的流程来纠正错误或删除添加的人员。另一个例子是美国的信用报告系统：在2012年美国联邦贸易委员会(FTC)对信用报告进行的一项大规模研究中，发现26%的消费者的文件中至少有一个错误，5%的人的错误可能是毁灭性的。然而，纠正这些错误的过程极其缓慢和不透明。当公共电台记者Bobby Allyn发现他被错误地列为持有枪支罪时，他“打了十几通电话，县法院书记员花了六个星期才解决了这个问题”。那是在我以记者的身份联系了该公司的公关部门之后。”

作为机器学习的从业者，我们并不认为了解我们的算法最终如何在实践中实现是我们的责任。但我们需要。

Feedback Loops

反馈循环

We explained in <<chapter_intro>> how an algorithm can interact with its environment to create a feedback loop, making predictions that reinforce actions taken in the real world, which lead to predictions even more pronounced in the same direction. As an example, let's again consider YouTube's recommendation system. A couple of years ago the Google team talked about how they had introduced reinforcement learning (closely related to deep learning, but where your loss function represents a result potentially a long time after an action occurs) to improve YouTube's recommendation system. They described how they used an algorithm that made recommendations such that watch time would be optimized.

However, human beings tend to be drawn to controversial content. This meant that videos about things like conspiracy theories started to get recommended more and more by the recommendation system. Furthermore, it turns out that the kinds of people that are interested in conspiracy theories are also people that watch a lot of online videos! So, they started to get drawn more and more toward YouTube. The increasing number of conspiracy theorists watching videos on YouTube resulted in the algorithm recommending more and more conspiracy theory and other extremist content, which resulted in more extremists watching videos on YouTube, and more people watching YouTube developing extremist views, which led to the algorithm recommending more extremist content... The system was spiraling out of control.

And this phenomenon was not contained to this particular type of content. In June 2019 the New York Times published an article on YouTube's recommendation system, titled "On YouTube’s Digital Playground, an Open Gate for Pedophiles". The article started with this chilling story:

我们在<<chapter_intro>>中解释了算法如何与它的环境交互来创建一个反馈循环，做出预测以加强在现实世界中采取的行动，这导致在同一方向上的预测更加明显。让我们再次以YouTube的推荐系统为例。几年前谷歌团队谈到他们如何引入强化学习(与深度学习密切相关，但你的损失函数代表一个动作发生后很长一段时间的结果)来改进YouTube的推荐系统。他们描述了他们如何使用一种算法来提出建议，从而优化观看时间。

然而，人们往往会被有争议的内容所吸引。这意味着，有关阴谋论等内容的视频开始被推荐系统越来越多地推荐。此外，事实证明，那些对阴谋论感兴趣的人也是那些看很多在线视频的人!所以，他们开始越来越多地被YouTube吸引。越来越多的阴谋论者在YouTube上观看视频，导致算法推荐越来越多的阴谋论和其他极端主义内容，这导致越来越多的极端分子在YouTube上观看视频，越来越多的人在YouTube上发展极端主义观点，这导致算法推荐更多的极端主义内容……这个系统正在逐渐失去控制。

而这一现象并不包含在这一特定类型的内容中。2019年6月，《纽约时报》在YouTube的推荐系统上发表了一篇题为“在YouTube的数字游乐场，为恋童癖者敞开的大门”的文章。这篇文章以一个令人毛骨悚然的故事开篇:

: Christiane C. didn’t think anything of it when her 10-year-old daughter and a friend uploaded a video of themselves playing in a backyard pool… A few days later… the video had thousands of views. Before long, it had ticked up to 400,000... “I saw the video again and I got scared by the number of views,” Christiane said. She had reason to be. YouTube’s automated recommendation system… had begun showing the video to users who watched other videos of prepubescent, partially clothed children, a team of researchers has found.

: On its own, each video might be perfectly innocent, a home movie, say, made by a child. Any revealing frames are fleeting and appear accidental. But, grouped together, their shared features become unmistakable.

：当Christiane C.10岁的女儿和一个朋友上传了一段他们在后院游泳池玩耍的视频时，她没想到...几天后...这段视频有成千上万的浏览量。不久，它就上升到了40万...Christiane说：“我又看了一遍视频，我被浏览量吓到了，”。她有理由这样做。YouTube的自动推荐系统...一组研究人员发现，YouTube的自动推荐系统已经开始向观看其他青春期前、半裸儿童视频的用户展示视频。

：就其本身而言，每个视频都可能是完全无辜的，比如一部由孩子制作的家庭电影。任何暴露的画面都是转瞬即逝的，看起来是偶然的。但是，组合在一起，它们的共同特征变得明确无误。

YouTube's recommendation algorithm had begun curating playlists for pedophiles, picking out innocent home videos that happened to contain prepubescent, partially clothed children.

No one at Google planned to create a system that turned family videos into porn for pedophiles. So what happened?

Part of the problem here is the centrality of metrics in driving a financially important system. When an algorithm has a metric to optimize, as you have seen, it will do everything it can to optimize that number. This tends to lead to all kinds of edge cases, and humans interacting with a system will search for, find, and exploit these edge cases and feedback loops for their advantage.

There are signs that this is exactly what has happened with YouTube's recommendation system. The Guardian ran an article called "How an ex-YouTube Insider Investigated its Secret Algorithm" about Guillaume Chaslot, an ex-YouTube engineer who created AlgoTransparency, which tracks these issues. Chaslot published the chart in <<ethics_yt_rt>>, following the release of Robert Mueller's "Report on the Investigation Into Russian Interference in the 2016 Presidential Election."

YouTube的推荐算法已经开始为恋童癖者整理播放列表，挑选出碰巧包含未到青春期、衣衫不整的儿童的无辜家庭视频。

谷歌没有人打算创建一个把家庭视频变成恋童癖色情片的系统。那么发生了什么?

这里的部分问题在于驱动一个财务重要系统的指标的中心地位。如您所见，当一个算法有一个指标需要优化时，它会尽其所能来优化这个数字。这往往会导致各种边缘情况，而与系统交互的人将搜索、发现并利用这些边缘情况和反馈循环。

有迹象表明，这正是YouTube推荐系统所发生的事情。《卫报》发表了一篇名为"前 YouTube 内部人员如何调查其秘密算法"的文章，介绍了创建 AlgoTransparency 的前 YouTube 工程师 Guillaume Chaslot，该工程师跟踪了这些问题。继Robert Mueller的《关于俄罗斯干涉2016年总统大选的调查报告》发布之后，Chaslot在<<ethics_yt_rt>>中发表了图表。

Russia Today's coverage of the Mueller report was an extreme outlier in terms of how many channels were recommending it. This suggests the possibility that Russia Today, a state-owned Russia media outlet, has been successful in gaming YouTube's recommendation algorithm. Unfortunately, the lack of transparency of systems like this makes it hard to uncover the kinds of problems that we're discussing.

One of our reviewers for this book, Aurélien Géron, led YouTube's video classification team from 2013 to 2016 (well before the events discussed here). He pointed out that it's not just feedback loops involving humans that are a problem. There can also be feedback loops without humans! He told us about an example from YouTube:

: One important signal to classify the main topic of a video is the channel it comes from. For example, a video uploaded to a cooking channel is very likely to be a cooking video. But how do we know what topic a channel is about? Well… in part by looking at the topics of the videos it contains! Do you see the loop? For example, many videos have a description which indicates what camera was used to shoot the video. As a result, some of these videos might get classified as videos about “photography.” If a channel has such a misclassified video, it might be classified as a “photography” channel, making it even more likely for future videos on this channel to be wrongly classified as “photography.” This could even lead to runaway virus-like classifications! One way to break this feedback loop is to classify videos with and without the channel signal. Then when classifying the channels, you can only use the classes obtained without the channel signal. This way, the feedback loop is broken.

There are positive examples of people and organizations attempting to combat these problems. Evan Estola, lead machine learning engineer at Meetup, discussed the example of men expressing more interest than women in tech meetups. taking gender into account could therefore cause Meetup’s algorithm to recommend fewer tech meetups to women, and as a result, fewer women would find out about and attend tech meetups, which could cause the algorithm to suggest even fewer tech meetups to women, and so on in a self-reinforcing feedback loop. So, Evan and his team made the ethical decision for their recommendation algorithm to not create such a feedback loop, by explicitly not using gender for that part of their model. It is encouraging to see a company not just unthinkingly optimize a metric, but consider its impact. According to Evan, "You need to decide which feature not to use in your algorithm... the most optimal algorithm is perhaps not the best one to launch into production."

While Meetup chose to avoid such an outcome, Facebook provides an example of allowing a runaway feedback loop to run wild. Like YouTube, it tends to radicalize users interested in one conspiracy theory by introducing them to more. As Renee DiResta, a researcher on proliferation of disinformation, writes:

《今日俄罗斯》对穆勒报告的报道在有多少渠道推荐它方面是一个极端的异常值。这表明，俄罗斯国有媒体《今日俄罗斯》很可能成功地利用了YouTube的推荐算法。不幸的是，这样的系统缺乏透明度使得我们很难发现我们正在讨论的各种问题。

本书的一位审稿人Aurélien Géron在2013年至2016年期间领导了YouTube的视频分类团队(早在这里讨论的事件之前)。他指出，问题不仅仅在于人类参与的反馈循环。也可以有没有人的反馈循环!他给我们讲了一个来自YouTube的例子:

:区分视频主题的一个重要信号是它来自的频道。例如，上传到烹饪频道的视频很有可能是烹饪视频。但是我们怎么知道一个频道的主题是什么呢?嗯，部分是通过看它所包含的视频主题!你看到这个循环了吗?例如，许多视频都有一个说明，说明是用什么相机拍摄的视频。因此，其中一些视频可能会被归类为关于“摄影”的视频。如果一个频道有这样一个错误分类的视频，它可能会被归类为“摄影”频道，这就更有可能使该频道未来的视频被错误分类为“摄影”。这甚至可能导致失控的病毒式分类！打破这种反馈循环的一种方法是对带有和不带有通信信号的视频进行分类。然后在对通道进行分类时，只能使用在没有通道信号的情况下获得的类。这样，反馈循环就被打破了。

有一些积极的例子表明，个人和组织试图解决这些问题。Meetup的首席机器学习工程师Evan Estola讨论了这个例子男性比女性对科技聚会更感兴趣。因此，将性别因素考虑在内可能会导致Meetup的算法向女性推荐更少的科技聚会，结果，更少的女性会发现并参加科技聚会，这可能会导致算法向女性推荐更少的科技聚会，从而形成一个自我强化的反馈循环。因此，Evan和他的团队为他们的推荐算法做出了道德上的决定，不创建这样的反馈循环，明确地在他们的模型中不使用性别。看到一家公司不只是不思进取地优化指标，而是考虑其影响，这是令人鼓舞的。根据Evan的说法，“你需要决定在你的算法中不使用哪个特性……最优算法可能并不是投入生产的最佳算法。”

虽然Meetup选择避免这样的结果，但Facebook提供了一个让失控的反馈循环失控的例子。和YouTube一样，它倾向于通过向对一种阴谋论感兴趣的用户介绍更多内容来激化他们。正如研究虚假信息扩散的研究人员Renee DiResta所写:

: Once people join a single conspiracy-minded [Facebook] group, they are algorithmically routed to a plethora of others. Join an anti-vaccine group, and your suggestions will include anti-GMO, chemtrail watch, flat Earther (yes, really), and "curing cancer naturally groups. Rather than pulling a user out of the rabbit hole, the recommendation engine pushes them further in."

:一旦人们加入了一个有阴谋思想的[Facebook]群组，他们就会被算法路由到大量其他群组。加入一个反疫苗小组，你的建议将包括反转基因、化学追踪观察、地平论(是的，真的)和“自然治疗癌症小组”。推荐引擎不是把用户从兔子洞中拉出来，而是推动他们再往里。”

It is extremely important to keep in mind that this kind of behavior can happen, and to either anticipate a feedback loop or take positive action to break it when you see the first signs of it in your own projects. Another thing to keep in mind is bias, which, as we discussed briefly in the previous chapter, can interact with feedback loops in very troublesome ways.

记住这种行为可能会发生是非常重要的，并且在您自己的项目中看到反馈循环的第一个迹象时，要么预测反馈循环，要么采取积极行动打破它。另一件要记住的事情是偏见，正如我们在上一章中简要讨论的那样，它可能会以非常麻烦的方式与反馈循环相互作用。

Bias

偏差

Discussions of bias online tend to get pretty confusing pretty fast. The word "bias" means so many different things. Statisticians often think when data ethicists are talking about bias that they're talking about the statistical definition of the term bias. But they're not. And they're certainly not talking about the biases that appear in the weights and biases which are the parameters of your model!

What they're talking about is the social science concept of bias. In "A Framework for Understanding Unintended Consequences of Machine Learning" MIT's Harini Suresh and John Guttag describe six types of bias in machine learning, summarized in <> from their paper.

网上关于偏见的讨论往往会很快变得非常混乱。“偏见”这个词意味着很多不同的东西。统计学家经常认为，当数据伦理学家谈论偏见时，他们在谈论术语偏见的统计定义。但他们不是。他们当然不是在谈论作为模型参数的权重和偏差中出现的偏差！

他们说的是社会科学中的偏见。在"理解机器学习意外后果的框架"中，麻省理工学院的Harini Suresh和John Guttag描述了机器学习中的六种偏见，总结在他们论文中（见<>）。

We'll discuss four of these types of bias, those that we've found most helpful in our own work (see the paper for details on the others).

我们将讨论其中四种类型的偏见，这些偏见对我们自己的工作最有帮助（有关其他偏见的详细信息，请参阅论文）。

Historical bias

历史偏见

Historical bias comes from the fact that people are biased, processes are biased, and society is biased. Suresh and Guttag say: "Historical bias is a fundamental, structural issue with the first step of the data generation process and can exist even given perfect sampling and feature selection."

For instance, here are a few examples of historical race bias in the US, from the New York Times article "Racial Bias, Even When We Have Good Intentions" by the University of Chicago's Sendhil Mullainathan:

• When doctors were shown identical files, they were much less likely to recommend cardiac catheterization (a helpful procedure) to Black patients.
• When bargaining for a used car, Black people were offered initial prices $700 higher and received far smaller concessions.
• Responding to apartment rental ads on Craigslist with a Black name elicited fewer responses than with a white name.
• An all-white jury was 16 percentage points more likely to convict a Black defendant than a white one, but when a jury had one Black member it convicted both at the same rate.

The COMPAS algorithm, widely used for sentencing and bail decisions in the US, is an example of an important algorithm that, when tested by ProPublica, showed clear racial bias in practice (<<bail_algorithm>>).

历史偏见来自人是有偏见的，过程是有偏见的，社会是有偏见的。Suresh和Guttag说：“历史偏见是数据生成过程第一步的一个基本的结构性问题，即使给出完美的采样和特征选择，它也可能存在。”

例如，这里有几个美国历史上种族偏见的例子，来自芝加哥大学的森希尔·穆莱纳森在《纽约时报》上的文章“种族偏见，即使我们有良好的意图”：

例如，以下是几个美国历史上种族偏见的例子，摘自芝加哥大学的Sendhil Mullainathan在《纽约时报》上的文章"种族偏见，即使我们有良好的意图":

• 当医生看到相同的病历时，他们不太可能向黑人患者推荐心导管插入术(一种有益的手术)。
• 在为一辆二手车讨价还价时，黑人得到的初始价格要高出700美元，而且得到的让步要小得多。
• 在Craigslist网站上，使用黑人名字的公寓出租广告得到的回应比使用白人名字的要少。
• 一个全是白人的陪审团判定黑人被告有罪的可能性比白人被告高出16%，但当一个陪审团有一名黑人成员时，两者的定罪率相同。

COMPAS算法在美国广泛用于量刑和保释决定，是一个重要算法的示例，在ProPublica测试时，该算法在实践中显示出明显的种族偏见(<<bail_algorithm>>)。

Any dataset involving humans can have this kind of bias: medical data, sales data, housing data, political data, and so on. Because underlying bias is so pervasive, bias in datasets is very pervasive. Racial bias even turns up in computer vision, as shown in the example of autocategorized photos shared on Twitter by a Google Photos user shown in <<google_photos>>.

任何涉及人类的数据集都可能存在这种偏见：医疗数据、销售数据、住房数据、政治数据等等。由于潜在的偏见如此普遍，数据集中的偏见非常普遍。种族偏见甚至出现在计算机视觉中，如<>中显示的Google Photos用户在Twitter上分享的自动分类照片示例所示。

Yes, that is showing what you think it is: Google Photos classified a Black user's photo with their friend as "gorillas"! This algorithmic misstep got a lot of attention in the media. “We’re appalled and genuinely sorry that this happened,” a company spokeswoman said. “There is still clearly a lot of work to do with automatic image labeling, and we’re looking at how we can prevent these types of mistakes from happening in the future.”

Unfortunately, fixing problems in machine learning systems when the input data has problems is hard. Google's first attempt didn't inspire confidence, as coverage by The Guardian suggested (<>).

是的，这就是你所想的：谷歌照片将一名黑人用户和他们朋友的照片归类为“大猩猩”！这一算法失误引起了媒体的大量关注。“我们对发生这种情况感到震惊和真诚的遗憾，”公司发言人表示。“在自动图像标记方面显然还有很多工作要做，我们正在研究如何防止此类错误在未来发生。”

不幸的是，当输入数据有问题时，解决机器学习系统中的问题是很困难的。正如《卫报》报道的那样，谷歌的第一次尝试并没有激发信心(<>)。

These kinds of problems are certainly not limited to just Google. MIT researchers studied the most popular online computer vision APIs to see how accurate they were. But they didn't just calculate a single accuracy number—instead, they looked at the accuracy across four different groups, as illustrated in <<face_recognition>>.

这类问题当然不仅限于谷歌。麻省理工学院的研究人员研究了最流行的在线计算机视觉API，以了解它们的准确性。但他们不仅仅计算一个准确率数字——相反，他们查看了四个不同组的准确率，如<>所示。

IBM's system, for instance, had a 34.7% error rate for darker females, versus 0.3% for lighter males—over 100 times more errors! Some people incorrectly reacted to these experiments by claiming that the difference was simply because darker skin is harder for computers to recognize. However, what actually happened was that, after the negative publicity that this result created, all of the companies in question dramatically improved their models for darker skin, such that one year later they were nearly as good as for lighter skin. So what this actually showed is that the developers failed to utilize datasets containing enough darker faces, or test their product with darker faces.

One of the MIT researchers, Joy Buolamwini, warned: "We have entered the age of automation overconfident yet underprepared. If we fail to make ethical and inclusive artificial intelligence, we risk losing gains made in civil rights and gender equity under the guise of machine neutrality."

Part of the issue appears to be a systematic imbalance in the makeup of popular datasets used for training models. The abstract to the paper "No Classification Without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World" by Shreya Shankar et al. states, "We analyze two large, publicly available image data sets to assess geo-diversity and find that these data sets appear to exhibit an observable amerocentric and eurocentric representation bias. Further, we analyze classifiers trained on these data sets to assess the impact of these training distributions and find strong differences in the relative performance on images from different locales." <<image_provenance>> shows one of the charts from the paper, showing the geographic makeup of what was, at the time (and still are, as this book is being written) the two most important image datasets for training models.

例如，IBM的系统对深色女性的错误率为34.7%，而对浅色男性的错误率为0.3%——超过100倍！一些人对这些实验的回应是错误的，他们声称这种差异仅仅是因为深色皮肤对计算机来说更难识别。然而，实际情况是，在这一结果造成了负面宣传之后，所有相关公司都大幅改进了他们的深色皮肤模型，以至于一年后它们几乎和浅色皮肤一样好。因此，这实际上表明开发人员未能利用包含足够多深色面孔的数据集，或者用深色面孔测试他们的产品。

麻省理工学院的一名研究人员Joy Buolamwini警告说:“我们已经进入了一个过度自信但准备不足的自动化时代。如果我们不能创造出具有道德和包容性的人工智能，我们就有可能在机器中立的幌子下失去在民权和性别平等方面取得的成果。”

问题的一部分似乎是用于训练模型的流行数据集构成的系统性失衡。由Shreya Shankar等人撰写的论文"没有代表的分类：评估发展中国家开放数据集中的地理多样性问题"的摘要指出，“我们分析了两个大型的、公开的图像数据集来评估地理多样性，发现这些数据集似乎表现出可观察到的以美国为中心和以欧洲为中心的表示偏差。此外，我们分析了在这些数据集上训练的分类器，以评估这些训练分布的影响，并发现在不同地区的图像上的相对表现有很大的差异。”<<image_provenance>>显示了论文中的一个图表，显示了当时（现在仍然是，正如本书正在编写的那样）用于训练模型的两个最重要的图像数据集的地理构成。

The vast majority of the images are from the United States and other Western countries, leading to models trained on ImageNet performing worse on scenes from other countries and cultures. For instance, research found that such models are worse at identifying household items (such as soap, spices, sofas, or beds) from lower-income countries. <<object_detect>> shows an image from the paper, "Does Object Recognition Work for Everyone?" by Terrance DeVries et al. of Facebook AI Research that illustrates this point.

绝大多数图像来自美国和其他西方国家，导致在ImageNet上训练的模型在其他国家和文化的场景中表现更差。例如，研究发现，这些模型在识别来自低收入国家的家庭用品(如肥皂、香料、沙发或床)方面表现较差。<<object_detect>>显示了一张来自论文"物体识别对每个人都有效吗？"的图片，该论文由Facebook AI Research的Terrance DeVries等人撰写，阐述了这一点。

In this example, we can see that the lower-income soap example is a very long way away from being accurate, with every commercial image recognition service predicting "food" as the most likely answer!

As we will discuss shortly, in addition, the vast majority of AI researchers and developers are young white men. Most projects that we have seen do most user testing using friends and families of the immediate product development group. Given this, the kinds of problems we just discussed should not be surprising.

Similar historical bias is found in the texts used as data for natural language processing models. This crops up in downstream machine learning tasks in many ways. For instance, it was widely reported that until last year Google Translate showed systematic bias in how it translated the Turkish gender-neutral pronoun "o" into English: when applied to jobs which are often associated with males it used "he," and when applied to jobs which are often associated with females it used "she" (<<turkish_gender>>).

在这个例子中，我们可以看到，低收入的肥皂例子离准确的答案还有很长的路要走，每一个商业图像识别服务都预测“食物”是最有可能的答案!

此外，正如我们稍后将讨论的那样，绝大多数AI研究人员和开发人员是年轻的白人男性。我们看到的大多数项目都是使用直接产品开发团队的朋友和家人进行用户测试。考虑到这一点，我们刚才讨论的问题就不足为奇了。

在用作自然语言处理模型数据的文本中也发现了类似的历史偏差。这以多种方式出现在下游机器学习任务中。例如，据广泛报道,直到去年谷歌翻译在将土耳其中性代词“o”翻译成英语方面表现出了系统性的偏见:当应用于经常与男性有关的工作时，它使用“他”，当应用于经常与女性有关的工作时，它使用“她”(<<turkish_gender>>)。

We also see this kind of bias in online advertisements. For instance, a study in 2019 by Muhammad Ali et al. found that even when the person placing the ad does not intentionally discriminate, Facebook will show ads to very different audiences based on race and gender. Housing ads with the same text, but picture either a white or a Black family, were shown to racially different audiences.

我们在网络广告中也看到了这种偏见。例如，Muhammad Ali 等人在2019年的一项研究发现，即使投放广告的人没有故意歧视，Facebook也会根据种族和性别向截然不同的受众展示广告。带有相同文本但图片是白人或黑人家庭的房屋广告被展示给不同种族的受众。

Measurement bias

测量偏差

In the paper "Does Machine Learning Automate Moral Hazard and Error" in American Economic Review, Sendhil Mullainathan and Ziad Obermeyer look at a model that tries to answer the question: using historical electronic health record (EHR) data, what factors are most predictive of stroke? These are the top predictors from the model:

• Prior stroke
• Cardiovascular disease
• Accidental injury
• Benign breast lump
• Colonoscopy
• Sinusitis

However, only the top two have anything to do with a stroke! Based on what we've studied so far, you can probably guess why. We haven’t really measured stroke, which occurs when a region of the brain is denied oxygen due to an interruption in the blood supply. What we’ve measured is who had symptoms, went to a doctor, got the appropriate tests, and received a diagnosis of stroke. Actually having a stroke is not the only thing correlated with this complete list—it's also correlated with being the kind of person who actually goes to the doctor (which is influenced by who has access to healthcare, can afford their co-pay, doesn't experience racial or gender-based medical discrimination, and more)! If you are likely to go to the doctor for an accidental injury, then you are likely to also go the doctor when you are having a stroke.

This is an example of measurement bias. It occurs when our models make mistakes because we are measuring the wrong thing, or measuring it in the wrong way, or incorporating that measurement into the model inappropriately.

Sendhil Mullainathan和Ziad Obermeyer在美国经济评论上发表了一篇文章"机器学习是否会自动化道德风险和错误"，他们研究了一个试图回答以下问题的模型:使用历史电子健康记录(EHR)数据，哪些因素最能预测中风?以下是该模型的主要预测因素:

• 先前的中风
• 心血管病
• 意外伤害
• 良性乳房肿块
• 结肠镜检查
• 鼻窦炎

然而，只有前两名与中风有关!根据我们目前所学的知识，你大概可以猜到原因。我们还没有真正测量中风，中风发生在大脑某个区域由于血液供应中断而缺氧的时候。我们所测量的是那些有症状的人，去看医生，做了适当的测试，并得到了中风的诊断。实际上，中风并不是与这个完整列表相关的唯一因素——它还与真正去看医生的人有关(这取决于谁能获得医疗保健，谁能支付他们的自付费用，谁不会经历种族或性别医疗歧视，等等)!如果你很可能因为意外受伤而去看医生，那么当你中风时，你也很可能去看医生。

这是测量偏差的一个例子。它发生在我们的模型出错的时候，因为我们测量了错误的东西，或者用了错误的方式测量，或者不恰当地将测量纳入模型中。

Aggregation bias

聚合偏差

Aggregation bias occurs when models do not aggregate data in a way that incorporates all of the appropriate factors, or when a model does not include the necessary interaction terms, nonlinearities, or so forth. This can particularly occur in medical settings. For instance, the way diabetes is treated is often based on simple univariate statistics and studies involving small groups of heterogeneous people. Analysis of results is often done in a way that does not take account of different ethnicities or genders. However, it turns out that diabetes patients have different complications across ethnicities, and HbA1c levels (widely used to diagnose and monitor diabetes) differ in complex ways across ethnicities and genders. This can result in people being misdiagnosed or incorrectly treated because medical decisions are based on a model that does not include these important variables and interactions.

当模型没有以包含所有适当因素的方式聚合数据时，或者当模型没有包含必要的交互项、非线性等时，就会出现聚合偏差。这尤其可能发生在医疗环境中。例如，糖尿病的治疗方式通常是基于简单的单变量统计数据和涉及异质人群的小群体研究。对结果的分析通常不考虑不同的种族或性别。然而，事实证明，糖尿病患者在不同种族有不同的并发症，而且HbA1c水平(广泛用于诊断和监测糖尿病)在不同种族和性别之间存在复杂的差异。这可能导致人们被误诊或错误治疗，因为医疗决策是基于一个不包括这些重要变量和相互作用的模型。

Representation bias

表征偏差

The abstract of the paper "Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting" by Maria De-Arteaga et al. notes that there is gender imbalance in occupations (e.g., females are more likely to be nurses, and males are more likely to be pastors), and says that: "differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances."

In other words, the researchers noticed that models predicting occupation did not only reflect the actual gender imbalance in the underlying population, but actually amplified it! This type of representation bias is quite common, particularly for simple models. When there is some clear, easy-to-see underlying relationship, a simple model will often simply assume that this relationship holds all the time. As <<representation_bias>> from the paper shows, for occupations that had a higher percentage of females, the model tended to overestimate the prevalence of that occupation.

Maria De-Arteaga等人的论文"Bios中的偏见:高风险背景下语义表征偏见的案例研究"指出，职业中存在性别失衡(例如，女性更有可能成为护士，男性更有可能成为牧师)，并指出:“性别之间真阳性率的差异与职业中现有的性别失衡相关，这可能会加剧这些失衡。”

换句话说，研究人员注意到，预测职业的模型不仅反映了潜在人口中实际的性别失衡，而且实际上放大了这种失衡!这种类型的表示偏差是很常见的，特别是对于简单的模型。当存在一些清晰、容易看到的潜在关系时，一个简单的模型通常会简单地假设这种关系一直存在。正如论文中<<representation_bias>>所示，对于女性比例较高的职业，该模型倾向于高估该职业的流行程度。

For example, in the training dataset 14.6% of surgeons were women, yet in the model predictions only 11.6% of the true positives were women. The model is thus amplifying the bias existing in the training set.

Now that we've seen that those biases exist, what can we do to mitigate them?

例如，在训练数据集中，14.6%的外科医生是女性，但在模型预测中，只有11.6 %的真正的女性是女性。因此，该模型放大了训练集中存在的偏见。

既然我们已经看到这些偏见的存在，我们能做些什么来减轻它们？

Addressing different types of bias

解决不同类型的偏见

Different types of bias require different approaches for mitigation. While gathering a more diverse dataset can address representation bias, this would not help with historical bias or measurement bias. All datasets contain bias. There is no such thing as a completely debiased dataset. Many researchers in the field have been converging on a set of proposals to enable better documentation of the decisions, context, and specifics about how and why a particular dataset was created, what scenarios it is appropriate to use in, and what the limitations are. This way, those using a particular dataset will not be caught off guard by its biases and limitations.

不同类型的偏见需要不同的缓解方法。虽然收集更多样的数据集可以解决表示偏见，但这对历史偏见或测量偏见没有帮助。所有数据均存在偏差。不存在完全去偏的数据集这样的东西。该领域的许多研究人员已经集中在一组建议上，以便更好地记录关于如何和为什么创建特定数据集的决策、上下文和细节，以及它适合使用哪些场景，以及有哪些限制。这样，那些使用特定数据集的人就不会被其偏见和限制所困扰。

We often hear the question—"Humans are biased, so does algorithmic bias even matter?" This comes up so often, there must be some reasoning that makes sense to the people that ask it, but it doesn't seem very logically sound to us! Independently of whether this is logically sound, it's important to realize that algorithms (particularly machine learning algorithms!) and people are different. Consider these points about machine learning algorithms:

• Machine learning can create feedback loops:: Small amounts of bias can rapidly increase exponentially due to feedback loops.
• Machine learning can amplify bias:: Human bias can lead to larger amounts of machine learning bias.
• Algorithms & humans are used differently:: Human decision makers and algorithmic decision makers are not used in a plug-and-play interchangeable way in practice.
• Technology is power:: And with that comes responsibility.

As the Arkansas healthcare example showed, machine learning is often implemented in practice not because it leads to better outcomes, but because it is cheaper and more efficient. Cathy O'Neill, in her book Weapons of Math Destruction (Crown), described the pattern of how the privileged are processed by people, whereas the poor are processed by algorithms. This is just one of a number of ways that algorithms are used differently than human decision makers. Others include:

• People are more likely to assume algorithms are objective or error-free (even if they’re given the option of a human override).
• Algorithms are more likely to be implemented with no appeals process in place.
• Algorithms are often used at scale.
• Algorithmic systems are cheap.

Even in the absence of bias, algorithms (and deep learning especially, since it is such an effective and scalable algorithm) can lead to negative societal problems, such as when used for disinformation.

我们经常听到这样一个问题——“人类是有偏见的，那么算法偏见有关系吗？”这个问题经常出现，一定有一些推理对问它的人来说是有意义的，但对我们来说，这似乎不太符合逻辑！不管这在逻辑上是否合理，重要的是要意识到算法（尤其是机器学习算法！）和人是不同的。考虑一下关于机器学习算法的以下几点：

• 机器学习可以创建反馈回路:: 由于反馈回路，少量的偏见会迅速成倍增加。
• 机器学习会放大偏见:: 人类偏见会导致更多的机器学习偏见。
• 算法和人的使用方式是不同的::在实践中，人类决策者和算法决策者并不是按即插即用的方式来使用的。
• 技术就是力量:: 随之而来的是责任。

正如阿肯色州医疗保健的例子所表明的那样，机器学习在实践中经常被应用，不是因为它会带来更好的结果，而是因为它更便宜、更高效。Cathy O'Neill在她的数学毁灭武器(Crown)一书中描述了特权阶层被人处理，而穷人被算法处理的模式。这只是算法与人类决策者不同使用的众多方式之一。其他的包括:

• 人们更有可能认为算法是客观的或没有错误的(即使他们可以选择人工替代)。
• 在没有上诉程序的情况下，算法更有可能被实现。
• 算法经常被大规模地使用。
• 算法系统很便宜。

即使在没有偏见的情况下，算法(因为它是一种有效且可扩展的算法)也会导致负面的社会问题，例如当被用于虚假信息时。

Disinformation

造谣

Disinformation has a history stretching back hundreds or even thousands of years. It is not necessarily about getting someone to believe something false, but rather often used to sow disharmony and uncertainty, and to get people to give up on seeking the truth. Receiving conflicting accounts can lead people to assume that they can never know whom or what to trust.

Some people think disinformation is primarily about false information or fake news, but in reality, disinformation can often contain seeds of truth, or half-truths taken out of context. Ladislav Bittman was an intelligence officer in the USSR who later defected to the US and wrote some books in the 1970s and 1980s on the role of disinformation in Soviet propaganda operations. In The KGB and Soviet Disinformation (Pergamon) he wrote, "Most campaigns are a carefully designed mixture of facts, half-truths, exaggerations, and deliberate lies."

In the US this has hit close to home in recent years, with the FBI detailing a massive disinformation campaign linked to Russia in the 2016 election. Understanding the disinformation that was used in this campaign is very educational. For instance, the FBI found that the Russian disinformation campaign often organized two separate fake "grass roots" protests, one for each side of an issue, and got them to protest at the same time! The Houston Chronicle reported on one of these odd events (<>).

: A group that called itself the "Heart of Texas" had organized it on social media—a protest, they said, against the "Islamization" of Texas. On one side of Travis Street, I found about 10 protesters. On the other side, I found around 50 counterprotesters. But I couldn't find the rally organizers. No "Heart of Texas." I thought that was odd, and mentioned it in the article: What kind of group is a no-show at its own event? Now I know why. Apparently, the rally's organizers were in Saint Petersburg, Russia, at the time. "Heart of Texas" is one of the internet troll groups cited in Special Prosecutor Robert Mueller's recent indictment of Russians attempting to tamper with the U.S. presidential election.

虚假信息的历史可以追溯到几百年甚至几千年前。它并不一定是让某人相信虚假的东西，而是经常用来散播不和谐和不确定性，让人们放弃寻求真理。收到相互矛盾的信息会让人们认为他们永远不知道该相信谁或信任什么。

有些人认为虚假信息主要是虚假信息或假新闻，但实际上，虚假信息往往包含真相的种子，或断章取义的半真半假。Ladislav Bittman是苏联的一名情报官员，他后来叛逃到美国，并在上世纪70年代和80年代写了一些关于假情报在苏联宣传行动中的作用的书。在《克格勃与苏联虚假情报》(Pergamon)一书中，他写道:“大多数竞选活动都是经过精心设计的，混合了事实、半真半假、夸张和故意谎言的混合体。”

在美国，这近年来已经影响到美国本土，美国联邦调查局(FBI)详细记录了2016年大选中与俄罗斯有关的大规模虚假信息活动。了解这场运动中使用的虚假信息是很有教育意义的。例如，美国联邦调查局发现，俄罗斯的造谣运动经常组织两场独立的假“草根”抗议活动，分别针对某一问题的每一方，并让他们同时抗议！ 休斯顿纪事报报道了其中一个奇怪的事件(<>)。

：一个自称“德克萨斯之心”的组织在社交媒体上组织了这次活动——他们说，这是对德克萨斯“伊斯兰化”的抗议。在特拉维斯街的一边，我发现了大约10名抗议者。在另一边，我发现了大约50名反抗议者。但我找不到集会组织者。没有“德克萨斯之心”。我觉得这很奇怪，并在文章中提到:什么样的团体会不出席自己的活动?现在我知道为什么了。显然，集会的组织者当时在俄罗斯的圣彼得堡。在特别检察官Robert Mueller最近对俄罗斯人试图篡改美国总统选举的指控中，“德克萨斯之心”是其中一个网络攻击组织。

Disinformation often involves coordinated campaigns of inauthentic behavior. For instance, fraudulent accounts may try to make it seem like many people hold a particular viewpoint. While most of us like to think of ourselves as independent-minded, in reality we evolved to be influenced by others in our in-group, and in opposition to those in our out-group. Online discussions can influence our viewpoints, or alter the range of what we consider acceptable viewpoints. Humans are social animals, and as social animals we are extremely influenced by the people around us. Increasingly, radicalization occurs in online environments; influence is coming from people in the virtual space of online forums and social networks.

Disinformation through autogenerated text is a particularly significant issue, due to the greatly increased capability provided by deep learning. We discuss this issue in depth when we delve into creating language models, in <<chapter_nlp>>.

One proposed approach is to develop some form of digital signature, to implement it in a seamless way, and to create norms that we should only trust content that has been verified. The head of the Allen Institute on AI, Oren Etzioni, wrote such a proposal in an article titled "How Will We Prevent AI-Based Forgery?": "AI is poised to make high-fidelity forgery inexpensive and automated, leading to potentially disastrous consequences for democracy, security, and society. The specter of AI forgery means that we need to act to make digital signatures de rigueur as a means of authentication of digital content."

Whilst we can't hope to discuss all the ethical issues that deep learning, and algorithms more generally, brings up, hopefully this brief introduction has been a useful starting point you can build on. We'll now move on to the questions of how to identify ethical issues, and what to do about them.

虚假信息通常涉及不真实行为的协调行动。例如，欺诈性账户可能试图让它看起来像是许多人持有一个特定的观点。虽然我们大多数人都喜欢认为自己是独立思考的，但事实上，我们进化为受到我们内部群体中其他人的影响，而与外部群体的人相反。在线讨论可以影响我们的观点，或者改变我们认为可以接受的观点的范围。人类是社会性动物，作为社会性动物，我们会受到周围人的极大影响。激进主义越来越多地发生在网络环境中;影响力来自在线论坛和社交网络的虚拟空间中的人们。

由于深度学习提供的能力大大提高,通过自动生成文本产生虚假信息是一个特别重要的问题。当我们深入研究创建语言模型时，我们会在<<chapter_nlp>>中深入讨论这个问题。

一种提议的方法是开发某种形式的数字签名，以无缝的方式实现它，并创建我们应该只信任经过验证的内容的规范。艾伦人工智能研究所的负责人Oren Etzioni在一篇题为"我们将如何防止基于人工智能的伪造?"的文章中写下了这样的提议：“人工智能正准备使高保真伪造变得廉价和自动化，这可能会给民主、安全和社会带来潜在的灾难性后果。人工智能伪造的幽灵意味着我们需要采取行动，使数字签名成为必要的数字内容认证手段。”

虽然我们不希望讨论深度学习和算法带来的所有伦理问题，但希望这篇简短的介绍是一个有用的起点，您可以以此为基础。现在我们将继续讨论如何识别道德问题，以及如何处理这些问题。

Identifying and Addressing Ethical Issues

识别和解决道德问题

Mistakes happen. Finding out about them, and dealing with them, needs to be part of the design of any system that includes machine learning (and many other systems too). The issues raised within data ethics are often complex and interdisciplinary, but it is crucial that we work to address them.

So what can we do? This is a big topic, but a few steps towards addressing ethical issues are:

• Analyze a project you are working on.
• Implement processes at your company to find and address ethical risks.
• Support good policy.
• Increase diversity.

Let's walk through each of these steps, starting with analyzing a project you are working on.

错误时有发生。发现并处理它们需要成为包括机器学习（以及许多其他系统）的系统设计的一部分。数据伦理中提出的问题通常是复杂和跨学科的，但我们努力解决它们至关重要。

那么我们能做些什么呢？这是一个大话题，但是解决伦理问题的几个步骤是：

• 分析你正在做的一个项目。
• 在你的公司实施发现和解决道德风险的程序。
• 良好的政策支持。
• 增加多样性。

让我们从分析您正在进行的项目开始，逐一介绍这些步骤。

Analyze a Project You Are Working On

分析您正在进行的项目

It's easy to miss important issues when considering ethical implications of your work. One thing that helps enormously is simply asking the right questions. Rachel Thomas recommends considering the following questions throughout the development of a data project:

• Should we even be doing this?
• What bias is in the data?
• Can the code and data be audited?
• What are the error rates for different sub-groups?
• What is the accuracy of a simple rule-based alternative?
• What processes are in place to handle appeals or mistakes?
• How diverse is the team that built it?

These questions may be able to help you identify outstanding issues, and possible alternatives that are easier to understand and control. In addition to asking the right questions, it's also important to consider practices and processes to implement.

One thing to consider at this stage is what data you are collecting and storing. Data often ends up being used for different purposes than what it was originally collected for. For instance, IBM began selling to Nazi Germany well before the Holocaust, including helping with Germany’s 1933 census conducted by Adolf Hitler, which was effective at identifying far more Jewish people than had previously been recognized in Germany. Similarly, US census data was used to round up Japanese-Americans (who were US citizens) for internment during World War II. It is important to recognize how data and images collected can be weaponized later. Columbia professor Tim Wu wrote that “You must assume that any personal data that Facebook or Android keeps are data that governments around the world will try to get or that thieves will try to steal.”

在考虑你的工作的道德影响时，很容易忽略重要的问题。有一件事非常有用，那就是问正确的问题。Rachel Thomas建议在数据项目的开发过程中考虑以下问题:

• 我们应该这样做吗?
• 数据中有什么偏差?
• 是否可以审核代码和数据?
• 不同子组的错误率是多少?
• 一个简单的基于规则的替代方法的准确性是多少?
• 采取什么程序处理上诉或错误?
• 构建它的团队是多么的多样化?

这些问题可以帮助您识别突出的问题，以及更容易理解和控制可能的替代方案。除了提出正确的问题之外，考虑要实施的实践和过程也很重要。

在这个阶段要考虑的一件事是收集和存储什么数据。数据通常最终被用于与最初收集的目的不同的目的。例如，早在大屠杀发生之前，IBM就开始向纳粹德国出售产品，包括帮助阿道夫·希特勒(Adolf Hitler)在德国1933年进行的人口普查，该普查有效地识别出了比之前在德国识别出的多得多的犹太人。类似地，美国人口普查数据被用于在二战期间拘留日裔美国人(他们是美国公民)。重要的是要认识到收集到的数据和图像如何在以后被武器化。哥伦比亚大学教授Tim Wu写道:“你必须假设，Facebook或Android保存的任何个人数据都是世界各国政府试图获取或小偷试图窃取的数据。”

Processes to Implement

实施过程

The Markkula Center has released An Ethical Toolkit for Engineering/Design Practice that includes some concrete practices to implement at your company, including regularly scheduled sweeps to proactively search for ethical risks (in a manner similar to cybersecurity penetration testing), expanding the ethical circle to include the perspectives of a variety of stakeholders, and considering the terrible people (how could bad actors abuse, steal, misinterpret, hack, destroy, or weaponize what you are building?).

Even if you don't have a diverse team, you can still try to pro-actively include the perspectives of a wider group, considering questions such as these (provided by the Markkula Center):

• Whose interests, desires, skills, experiences, and values have we simply assumed, rather than actually consulted?
• Who are all the stakeholders who will be directly affected by our product? How have their interests been protected? How do we know what their interests really are—have we asked?
• Who/which groups and individuals will be indirectly affected in significant ways?
• Who might use this product that we didn’t expect to use it, or for purposes we didn’t initially intend?

Markkula中心发布了一份工程/设计实践道德工具包，其中包括在贵公司实施的一些具体实践，包括定期清理以主动寻找道德风险(以类似网络安全渗透测试的方式)，将道德圈扩展到包括各种利益相关者的观点，并考虑可怕的人(不良行为者如何滥用、窃取、曲解、破解、破坏或武器化你正在构建的东西？)。

即使你没有一个多元化的团队，你仍然可以尝试主动地纳入更广泛群体的观点，考虑以下问题(由Markkula中心提供):

• 我们只是假设，而不是实际咨询谁的兴趣、愿望、技能、经验和价值观?
• 哪些利益相关者会直接受到我们产品的影响?他们的利益是如何得到保护的?我们怎么知道他们真正的兴趣是什么——我们问过了吗?
• 谁/哪些群体和个人将受到重大的间接影响?
• 谁可能会使用我们不希望使用的产品，或者用于我们最初不打算使用的目的?

Ethical lenses

道德镜头

Another useful resource from the Markkula Center is its Conceptual Frameworks in Technology and Engineering Practice. This considers how different foundational ethical lenses can help identify concrete issues, and lays out the following approaches and key questions:

• The rights approach:: Which option best respects the rights of all who have a stake?
• The justice approach:: Which option treats people equally or proportionately?
• The utilitarian approach:: Which option will produce the most good and do the least harm?
• The common good approach:: Which option best serves the community as a whole, not just some members?
• The virtue approach:: Which option leads me to act as the sort of person I want to be?

Markkula's recommendations include a deeper dive into each of these perspectives, including looking at a project through the lenses of its consequences:

• Who will be directly affected by this project? Who will be indirectly affected?
• Will the effects in aggregate likely create more good than harm, and what types of good and harm?
• Are we thinking about all relevant types of harm/benefit (psychological, political, environmental, moral, cognitive, emotional, institutional, cultural)?
• How might future generations be affected by this project?
• Do the risks of harm from this project fall disproportionately on the least powerful in society? Will the benefits go disproportionately to the well-off?
• Have we adequately considered "dual-use"?

The alternative lens to this is the deontological perspective, which focuses on basic concepts of right and wrong:

• What rights of others and duties to others must we respect?
• How might the dignity and autonomy of each stakeholder be impacted by this project?
• What considerations of trust and of justice are relevant to this design/project?
• Does this project involve any conflicting moral duties to others, or conflicting stakeholder rights? How can we prioritize these?

One of the best ways to help come up with complete and thoughtful answers to questions like these is to ensure that the people asking the questions are diverse.

Markkula中心的另一个有用资源是它的技术和工程实践中的概念框架。它考虑了不同的基本道德视角如何有助于识别具体问题，并列出了以下方法和关键问题:

• 权利方法:哪一种选择最尊重所有利害相关者的权利？
• 公正的方法:哪个选项平等或相称地对待人们？
• 功利主义的方法:哪种选择能产生最多的好处，造成最少的伤害?
• 共同的好方法:哪种选择最好地服务于整个社区，而不是部分成员?
• 美德方法:哪一种选择让我成为我想成为的那种人?

Markkula的建议包括对这些观点的深入探讨，包括从项目后果的角度来看待项目:

• 谁会直接受到这个项目的影响?谁会受到间接影响?
• 总的来说，这些影响带来的好处会大于坏处吗?哪些类型的好处和坏处?
• 我们是否考虑到了所有相关类型的伤害/利益(心理、政治、环境、道德、认知、情感、制度、文化)?
• 这个项目会对子孙后代产生怎样的影响?
• 这个项目的伤害风险会不成比例地落在社会中最弱势的群体身上吗?福利会不成比例地流向富人吗?
• 我们是否充分考虑过“双重用途”?

另一种视角是义务论视角，它关注的是对与错的基本概念:

• 我们必须尊重他人的哪些权利和义务?
• 该项目如何影响每个利益相关者的尊严和自主权?
• 在这个设计/项目中，有哪些关于信任和公正的考虑?
• 这个项目是否涉及到与他人相冲突的道德义务，或与利益相关者的权利相冲突?我们如何区分优先级?

要想对这些问题做出完整而周到的回答，最好的方法之一就是确保提问的人是多样化的。

The Power of Diversity

多样性的力量

Currently, less than 12% of AI researchers are women, according to a study from Element AI. The statistics are similarly dire when it comes to race and age. When everybody on a team has similar backgrounds, they are likely to have similar blindspots around ethical risks. The Harvard Business Review (HBR) has published a number of studies showing many benefits of diverse teams, including:

• "How Diversity Can Drive Innovation"
• "Teams Solve Problems Faster When They’re More Cognitively Diverse"
• "Why Diverse Teams Are Smarter", and
• "Defend Your Research: What Makes a Team Smarter? More Women"

Diversity can lead to problems being identified earlier, and a wider range of solutions being considered. For instance, Tracy Chou was an early engineer at Quora. She wrote of her experiences, describing how she advocated internally for adding a feature that would allow trolls and other bad actors to be blocked. Chou recounts, “I was eager to work on the feature because I personally felt antagonized and abused on the site (gender isn’t an unlikely reason as to why)... But if I hadn’t had that personal perspective, it’s possible that the Quora team wouldn’t have prioritized building a block button so early in its existence.” Harassment often drives people from marginalized groups off online platforms, so this functionality has been important for maintaining the health of Quora's community.

A crucial aspect to understand is that women leave the tech industry at over twice the rate that men do, according to the Harvard Business Review (41% of women working in tech leave, compared to 17% of men). An analysis of over 200 books, white papers, and articles found that the reason they leave is that “they’re treated unfairly; underpaid, less likely to be fast-tracked than their male colleagues, and unable to advance.”

Studies have confirmed a number of the factors that make it harder for women to advance in the workplace. Women receive more vague feedback and personality criticism in performance evaluations, whereas men receive actionable advice tied to business outcomes (which is more useful). Women frequently experience being excluded from more creative and innovative roles, and not receiving high-visibility “stretch” assignments that are helpful in getting promoted. One study found that men’s voices are perceived as more persuasive, fact-based, and logical than women’s voices, even when reading identical scripts.

Receiving mentorship has been statistically shown to help men advance, but not women. The reason behind this is that when women receive mentorship, it’s advice on how they should change and gain more self-knowledge. When men receive mentorship, it’s public endorsement of their authority. Guess which is more useful in getting promoted?

As long as qualified women keep dropping out of tech, teaching more girls to code will not solve the diversity issues plaguing the field. Diversity initiatives often end up focusing primarily on white women, even though women of color face many additional barriers. In interviews with 60 women of color who work in STEM research, 100% had experienced discrimination.

Element AI的一项研究显示，目前，只有不到12%的人工智能研究人员是女性。当涉及到种族和年龄时，统计数据同样可怕。当团队中的每个人都有相似的背景时，他们很可能在道德风险方面有相似的盲点。哈佛商业评论(HBR)发表了许多研究，显示了多元化团队的许多好处，包括:

• "多样性如何驱动创新"
• "当团队的认知更多样化时，他们解决问题的速度更快"
• "为什么多样化的团队更聪明"，以及
• "捍卫你的研究:什么让一个团队更聪明?更多的女性"

多样性可以导致更早地发现问题，并考虑更广泛的解决方案。例如，Tracy Chou是Quora的早期工程师。她写下了自己的经历，描述了她如何在内部主张添加一个允许屏蔽喷子和其他不良行为功能。Chou 回忆道:“我非常渴望做这个功能，因为我个人觉得在这个网站上受到了敌视和虐待(性别并不是一个不可能的原因)……但如果我没有这样的个人观点，Quora团队可能不会在这么早的时候就优先创建一个屏蔽按钮。”骚扰经常会让边缘化群体的人离开在线平台，所以这个功能对于维护Quora社区的健康非常重要。

哈佛商业评论指出，需要了解的一个关键方面是，女性离开科技行业的比例是男性的两倍多(科技行业41%的女性离职，而男性的比例为17%)。一项对200多本书、白皮书和文章的分析发现，他们离开的原因是“他们受到了不公平的对待;她们的薪水过低，不太可能比男同事获得快速晋升，而且无法升职。”

研究已经证实了一些使女性在职场中更难获得晋升的因素。女性在绩效评估中收到的反馈和个性批评更多是模糊的，而男性收到的是与业务结果相关的可行建议(这更有用)。女性经常被排除在更具创造性和创新性的职位之外，没有得到对晋升有帮助的可观的“延伸”任务。一项研究发现，人们认为男性的声音比女性的声音更有说服力、更有事实依据、更有逻辑，即使是在阅读相同的剧本时也是如此。

据统计，接受导师辅导对男性晋升有帮助，但对女性没有帮助。这背后的原因是，当女性接受导师辅导时，这是关于她们应该如何改变和获得更多的自我认识的建议。当男性接受指导时，这是公众对他们权威的认可。猜猜哪个对升职更有用?

只要合格的女性不断退出技术领域，教更多女孩编程就不能解决困扰该领域的多样性问题。尽管有色人种女性面临着许多额外的障碍，但多元化举措最终往往主要关注白人女性。在对60名从事STEM研究的有色人种女性的采访中，100%的人都经历过歧视。

The hiring process is particularly broken in tech. One study indicative of the dysfunction comes from Triplebyte, a company that helps place software engineers in companies, conducting a standardized technical interview as part of this process. They have a fascinating dataset: the results of how over 300 engineers did on their exam, coupled with the results of how those engineers did during the interview process for a variety of companies. The number one finding from Triplebyte’s research is that “the types of programmers that each company looks for often have little to do with what the company needs or does. Rather, they reflect company culture and the backgrounds of the founders.”

This is a challenge for those trying to break into the world of deep learning, since most companies' deep learning groups today were founded by academics. These groups tend to look for people "like them"—that is, people that can solve complex math problems and understand dense jargon. They don't always know how to spot people who are actually good at solving real problems using deep learning.

This leaves a big opportunity for companies that are ready to look beyond status and pedigree, and focus on results!

科技行业的招聘流程尤其不完善。Triplebyte公司的一项研究表明了这一功能障碍。Triplebyte是一家帮助为公司安排软件工程师的公司，该公司在招聘过程中会进行标准化的技术面试。他们有一个有趣的数据集:300多名工程师的考试结果，以及这些工程师在不同公司面试过程中的表现。Triplebyte研究的第一项发现是，“每个公司寻找的程序员类型通常与公司需要或做什么没有什么关系。相反，它们反映了公司文化和创始人的背景。”

对于那些试图进入深度学习领域的人来说，这是一个挑战，因为如今大多数公司的深度学习小组都是由学者创立的。这些群体倾向于寻找“像他们一样”的人——也就是说，能够解决复杂的数学问题和理解密集术语的人。他们并不总是知道如何识别那些真正擅长使用深度学习解决实际问题的人。

这给那些愿意超越地位和血统，关注结果的公司留下了一个巨大的机会!

Fairness, Accountability, and Transparency

公平、问责和透明

The professional society for computer scientists, the ACM, runs a data ethics conference called the Conference on Fairness, Accountability, and Transparency. "Fairness, Accountability, and Transparency" which used to go under the acronym FAT but now uses to the less objectionable FAccT. Microsoft has a group focused on "Fairness, Accountability, Transparency, and Ethics" (FATE). In this section, we'll use "FAccT" to refer to the concepts of Fairness, Accountability, and Transparency.

FAccT is another lens that you may find useful in considering ethical issues. One useful resource for this is the free online book Fairness and Machine Learning: Limitations and Opportunities by Solon Barocas, Moritz Hardt, and Arvind Narayanan, which "gives a perspective on machine learning that treats fairness as a central concern rather than an afterthought." It also warns, however, that it "is intentionally narrow in scope... A narrow framing of machine learning ethics might be tempting to technologists and businesses as a way to focus on technical interventions while sidestepping deeper questions about power and accountability. We caution against this temptation." Rather than provide an overview of the FAccT approach to ethics (which is better done in books such as that one), our focus here will be on the limitations of this kind of narrow framing.

One great way to consider whether an ethical lens is complete is to try to come up with an example where the lens and our own ethical intuitions give diverging results. Os Keyes, Jevan Hutson, and Meredith Durbin explored this in a graphic way in their paper "A Mulching Proposal: Analysing and Improving an Algorithmic System for Turning the Elderly into High-Nutrient Slurry". The paper's abstract says:

计算机科学家专业协会ACM举办了一个名为“公平、问责和透明会议”的数据伦理会议。“公平、问责和透明”这个词以前用的是FAT的缩写，但现在用的是不那么令人反感的FAccT。微软有一个专注于“公平、责任、透明和道德”(FATE)的小组。在本节中，我们将使用“FAccT”来指代公平、问责和透明的概念。

在考虑道德问题时，FAccT是另一个有用的视角。Solon Barocas、Moritz Hardt和Arvind Narayanan撰写的免费在线书籍公平和机器学习:限制和机会是一个有用的资源，该书“提供了一个关于机器学习的视角，将公平视为核心问题，而不是事后补充。”然而，它也警告说，它“故意缩小范围……对技术人员和企业来说，机器学习伦理的狭隘框架可能很有吸引力，可以让他们专注于技术干预，同时避开有关权力和责任的更深层问题。我们要警惕这种诱惑。”我们在这里将重点放在这种狭窄框架的局限性上，而不是提供对道德的FAccT方法的概述(在这样的书里写得更好)。

考虑道德视角是否完整的一个好方法是，试着举出一个视角和我们自己的道德直觉给出不同结果的例子。Os Keyes, Jevan Hutson和Meredith Durbin在他们的论文"一个覆盖建议：分析和改进一个将老年人变成高营养浆液的算法系统"中以图形的方式探索了这一点。论文摘要说:

: The ethical implications of algorithmic systems have been much discussed in both HCI and the broader community of those interested in technology design, development and policy. In this paper, we explore the application of one prominent ethical framework - Fairness, Accountability, and Transparency - to a proposed algorithm that resolves various societal issues around food security and population aging. Using various standardised forms of algorithmic audit and evaluation, we drastically increase the algorithm's adherence to the FAT framework, resulting in a more ethical and beneficent system. We discuss how this might serve as a guide to other researchers or practitioners looking to ensure better ethical outcomes from algorithmic systems in their line of work.

:算法系统的伦理影响已经在人机交互和对技术设计、开发和政策感兴趣的更广泛的社区中进行了大量讨论。在本文中，我们探讨了一个突出的道德框架——公平、问责和透明度——在一种拟议算法中的应用，该算法解决了围绕粮食安全和人口老龄化的各种社会问题。通过使用各种标准化形式的算法审计和评估，我们大大提高了算法对FAT框架的遵守程度，从而产生了一个更加道德和有益的系统。我们讨论了如何将其作为其他研究人员或从业者的指南，以确保算法系统在他们的工作中产生更好的道德结果。

In this paper, the rather controversial proposal ("Turning the Elderly into High-Nutrient Slurry") and the results ("drastically increase the algorithm's adherence to the FAT framework, resulting in a more ethical and beneficent system") are at odds... to say the least!

In philosophy, and especially philosophy of ethics, this is one of the most effective tools: first, come up with a process, definition, set of questions, etc., which is designed to resolve some problem. Then try to come up with an example where that apparent solution results in a proposal that no one would consider acceptable. This can then lead to a further refinement of the solution.

So far, we've focused on things that you and your organization can do. But sometimes individual or organizational action is not enough. Sometimes, governments also need to consider policy implications.

在这篇论文中，相当有争议的提议(“将老年人变成高营养浆液”)和结果(“大幅增加算法对FAT框架的遵守，导致一个更道德和有益的系统”)是不一致的……退一步说!

在哲学中，特别是在伦理学中，这是最有效的工具之一:首先，提出一个过程，定义，一组问题，等等，它是用来解决某个问题的。然后试着举出一个例子，这个明显的解决方案会导致一个没有人认为可以接受的提案。这可以导致解决方案的进一步细化。

在哲学，尤其是伦理哲学中，这是最有效的工具之一：首先，提出一个过程、定义、一系列问题等，旨在解决一些问题。然后尝试提出一个例子，在这个例子中，显而易见的解决方案会导致一个没有人认为可以接受的提案。这可以导致解决方案的进一步完善。

到目前为止，我们关注的是您和您的组织可以做的事情。但有时个人或组织的行动是不够的。有时，政府也需要考虑政策影响。

Role of Policy

政策的作用

We often talk to people who are eager for technical or design fixes to be a full solution to the kinds of problems that we've been discussing; for instance, a technical approach to debias data, or design guidelines for making technology less addictive. While such measures can be useful, they will not be sufficient to address the underlying problems that have led to our current state. For example, as long as it is incredibly profitable to create addictive technology, companies will continue to do so, regardless of whether this has the side effect of promoting conspiracy theories and polluting our information ecosystem. While individual designers may try to tweak product designs, we will not see substantial changes until the underlying profit incentives change.

我们经常与那些渴望通过技术或设计解决方案来解决我们所讨论的各种问题的人交谈;例如，去偏数据的技术方法，或降低技术成瘾性的设计指南。虽然这些措施可能是有用的，但它们不足以解决导致我们目前状态的根本问题。例如，只要创造令人上瘾的技术能够带来令人难以置信的利润，公司就会继续这样做，不管这是否有促进阴谋论和污染我们的信息生态系统的副作用。虽然个别设计师可能试图调整产品设计，但在潜在的利润激励发生变化之前，我们不会看到实质性的变化。

The Effectiveness of Regulation

监管的有效性

To look at what can cause companies to take concrete action, consider the following two examples of how Facebook has behaved. In 2018, a UN investigation found that Facebook had played a “determining role” in the ongoing genocide of the Rohingya, an ethnic minority in Mynamar described by UN Secretary-General Antonio Guterres as "one of, if not the, most discriminated people in the world." Local activists had been warning Facebook executives that their platform was being used to spread hate speech and incite violence since as early as 2013. In 2015, they were warned that Facebook could play the same role in Myanmar that the radio broadcasts played during the Rwandan genocide (where a million people were killed). Yet, by the end of 2015, Facebook only employed four contractors that spoke Burmese. As one person close to the matter said, "That’s not 20/20 hindsight. The scale of this problem was significant and it was already apparent." Zuckerberg promised during the congressional hearings to hire "dozens" to address the genocide in Myanmar (in 2018, years after the genocide had begun, including the destruction by fire of at least 288 villages in northern Rakhine state after August 2017).

This stands in stark contrast to Facebook quickly hiring 1,200 people in Germany to try to avoid expensive penalties (of up to 50 million euros) under a new German law against hate speech. Clearly, in this case, Facebook was more reactive to the threat of a financial penalty than to the systematic destruction of an ethnic minority.

In an article on privacy issues, Maciej Ceglowski draws parallels with the environmental movement:

: This regulatory project has been so successful in the First World that we risk forgetting what life was like before it. Choking smog of the kind that today kills thousands in Jakarta and Delhi was https://en.wikipedia.org/wiki/Pea_soup_fog[once emblematic of London]. The Cuyahoga River in Ohio used to http://www.ohiohistorycentral.org/w/Cuyahoga_River_Fire[reliably catch fire]. In a particularly horrific example of unforeseen consequences, tetraethyl lead added to gasoline https://en.wikipedia.org/wiki/Lead%E2%80%93crime_hypothesis[raised violent crime rates] worldwide for fifty years. None of these harms could have been fixed by telling people to vote with their wallet, or carefully review the environmental policies of every company they gave their business to, or to stop using the technologies in question. It took coordinated, and sometimes highly technical, regulation across jurisdictional boundaries to fix them. In some cases, like the https://en.wikipedia.org/wiki/Montreal_Protocol[ban on commercial refrigerants] that depleted the ozone layer, that regulation required a worldwide consensus. We’re at the point where we need a similar shift in perspective in our privacy law.

要研究什么能促使公司采取具体行动，可以考虑以下两个Facebook行为的例子。2018年，一项联合国调查发现，Facebook在持续的罗辛亚种族灭绝中发挥了“决定性作用”。罗辛亚是米纳马尔的少数民族，联合国秘书长Antonio Guterres将其描述为“世界上最受歧视的人种之一，如果不是的话。”当地活动人士早在2013年就警告Facebook高管，他们的平台被用来传播仇恨言论和煽动暴力。2015年，他们被警告Facebook可能在缅甸发挥与卢旺达种族灭绝（100万人被杀）期间电台广播相同的作用。然而，到2015年底， Facebook只雇佣了四名会说缅甸语的承包商。正如一位知情人士所说，“这不是事后诸葛亮。这个问题的规模很大，而且已经很明显了。”Zuckerberg在国会听证会上承诺雇佣“数十人”来解决缅甸的种族灭绝问题（2018年，种族灭绝开始多年后，包括2017年8月后若开邦北部至少288个村庄被大火摧毁）。

这与Facebook迅速在德国雇佣1,200人试图避免根据新的德国反仇恨言论法进行昂贵的处罚（最高5000万欧元）形成了鲜明对比。显然，在这种情况下，Facebook对经济处罚的威胁比对系统地摧毁少数民族更敏感。

在一篇关于隐私问题的文章中，Maciej Ceglowski将其与环境运动相提并论：

:这个监管项目在第一世界非常成功，以至于我们可能会忘记之前的生活是什么样子的。如今在雅加达和德里造成数千人死亡的令人窒息的雾霾是https://en.wikipedia.org/wiki/Pea_soup_fog(曾经是伦敦的象征)[once emblematic of London]。俄亥俄州的凯霍加河过去经常着火http://www.ohiohistorycentral.org/w/Cuyahoga_River_Fire[reliably catch fire]。在一个特别可怕的例子中，汽油中添加的四乙基铅导致了不可预见的后果https://en.wikipedia.org/wiki/Lead%E2%80%93crime_hypothesis[raised violent crime rates], 50年来全世界的暴力犯罪率上升。告诉人们用他们的钱包投票，或者仔细审查他们参与业务的每一家公司的环境政策，或者停止使用有问题的技术，这些危害都不可能得到解决。要解决这些问题，需要跨越司法边界的协调监管，有时还需要很高的技术含量。在某些情况下，比如https://en.wikipedia.org/wiki/Montreal_Protocol(禁止商用制冷剂)[ban on commercial refrigerants] 会消耗臭氧层，这项规定需要全世界达成共识。我们现在需要在隐私法上进行类似的视角转换。

Rights and Policy

权利和政策

Clean air and clean drinking water are public goods which are nearly impossible to protect through individual market decisions, but rather require coordinated regulatory action. Similarly, many of the harms resulting from unintended consequences of misuses of technology involve public goods, such as a polluted information environment or deteriorated ambient privacy. Too often privacy is framed as an individual right, yet there are societal impacts to widespread surveillance (which would still be the case even if it was possible for a few individuals to opt out).

Many of the issues we are seeing in tech are actually human rights issues, such as when a biased algorithm recommends that Black defendants have longer prison sentences, when particular job ads are only shown to young people, or when police use facial recognition to identify protesters. The appropriate venue to address human rights issues is typically through the law.

We need both regulatory and legal changes, and the ethical behavior of individuals. Individual behavior change can’t address misaligned profit incentives, externalities (where corporations reap large profits while offloading their costs and harms to the broader society), or systemic failures. However, the law will never cover all edge cases, and it is important that individual software developers and data scientists are equipped to make ethical decisions in practice.

清洁空气和清洁饮用水是公共产品，几乎不可能通过个别市场决定加以保护，而是需要协调一致的监管行动。同样，由于滥用技术而产生的非预期后果所造成的许多危害涉及公共产品，例如信息环境受到污染或环境隐私恶化。隐私常常被视为一种个人权利，但广泛的监控会产生社会影响(即使有少数人可以选择不参与监控，这种影响仍然存在)。

我们在科技领域看到的许多问题实际上都是人权问题，比如有偏见的算法建议黑人被告被判更长的刑期，特定的招聘广告只给年轻人看，或者警察使用面部识别来识别抗议者。处理人权问题的适当场所通常是通过法律。

我们需要监管和法律的改变，也需要个人的道德行为。个人行为的改变不能解决不协调的利润激励、外部性(企业获得巨额利润，同时减轻其成本和对更广泛社会的伤害)或系统性失败。然而，法律永远不会涵盖所有的边缘情况，重要的是，软件开发人员和数据科学家必须具备在实践中做出符合道德的决定。

Cars: A Historical Precedent

汽车：历史先例

The problems we are facing are complex, and there are no simple solutions. This can be discouraging, but we find hope in considering other large challenges that people have tackled throughout history. One example is the movement to increase car safety, covered as a case study in "Datasheets for Datasets" by Timnit Gebru et al. and in the design podcast 99% Invisible. Early cars had no seatbelts, metal knobs on the dashboard that could lodge in people’s skulls during a crash, regular plate glass windows that shattered in dangerous ways, and non-collapsible steering columns that impaled drivers. However, car companies were incredibly resistant to even discussing the idea of safety as something they could help address, and the widespread belief was that cars are just the way they are, and that it was the people using them who caused problems.

It took consumer safety activists and advocates decades of work to even change the national conversation to consider that perhaps car companies had some responsibility which should be addressed through regulation. When the collapsible steering column was invented, it was not implemented for several years as there was no financial incentive to do so. Major car company General Motors hired private detectives to try to dig up dirt on consumer safety advocate Ralph Nader. The requirement of seatbelts, crash test dummies, and collapsible steering columns were major victories. It was only in 2011 that car companies were required to start using crash test dummies that would represent the average woman, and not just average men’s bodies; prior to this, women were 40% more likely to be injured in a car crash of the same impact compared to a man. This is a vivid example of the ways that bias, policy, and technology have important consequences.

我们面临的问题是复杂的，没有简单的解决办法。这可能令人沮丧，但我们在考虑人们在整个历史上解决的其他重大挑战时找到了希望。一个例子是提高汽车安全的运动，在Timnit Gebru等人的"为数据集的数据表"和设计播客99% Invisible中有一个案例研究。早期的汽车没有安全带，仪表盘上的金属旋钮在碰撞时可能会嵌进人的头骨，普通的平板玻璃窗会以危险的方式破碎，不可折叠的方向盘立柱会刺穿司机。然而，汽车公司难以置信地拒绝讨论他们可以帮助解决的安全问题。人们普遍认为，汽车就是这样，是使用它们的人造成了问题。

消费者安全活动人士和倡导者花了几十年的时间才改变了全国性的讨论，认为汽车公司可能有一些责任，这些责任应该通过监管来解决。当可折叠的转向柱被发明时，由于没有经济上的激励，因此几年都没有实施。大型汽车公司通用汽车公司雇佣私人侦探试图挖掘消费者安全倡导者Ralph Nader的丑闻。对安全带、碰撞试验假人和可折叠的转向柱的要求是重大的胜利。直到2011年，汽车公司才被要求开始在碰撞测试中使用代表普通女性的假人，而不仅仅是代表普通男性的身体;在此之前，女性在同样的撞车事故中受伤的可能性比男性高40%。这是偏见、政策和技术产生重要影响的生动例子。

Conclusion

结论

Coming from a background of working with binary logic, the lack of clear answers in ethics can be frustrating at first. Yet, the implications of how our work impacts the world, including unintended consequences and the work becoming weaponized by bad actors, are some of the most important questions we can (and should!) consider. Even though there aren't any easy answers, there are definite pitfalls to avoid and practices to follow to move toward more ethical behavior.

Many people (including us!) are looking for more satisfying, solid answers about how to address harmful impacts of technology. However, given the complex, far-reaching, and interdisciplinary nature of the problems we are facing, there are no simple solutions. Julia Angwin, former senior reporter at ProPublica who focuses on issues of algorithmic bias and surveillance (and one of the 2016 investigators of the COMPAS recidivism algorithm that helped spark the field of FAccT) said in a 2019 interview:

: I strongly believe that in order to solve a problem, you have to diagnose it, and that we’re still in the diagnosis phase of this. If you think about the turn of the century and industrialization, we had, I don’t know, 30 years of child labor, unlimited work hours, terrible working conditions, and it took a lot of journalist muckraking and advocacy to diagnose the problem and have some understanding of what it was, and then the activism to get laws changed. I feel like we’re in a second industrialization of data information... I see my role as trying to make as clear as possible what the downsides are, and diagnosing them really accurately so that they can be solvable. That’s hard work, and lots more people need to be doing it.

It's reassuring that Angwin thinks we are largely still in the diagnosis phase: if your understanding of these problems feels incomplete, that is normal and natural. Nobody has a “cure” yet, although it is vital that we continue working to better understand and address the problems we are facing.

One of our reviewers for this book, Fred Monroe, used to work in hedge fund trading. He told us, after reading this chapter, that many of the issues discussed here (distribution of data being dramatically different than what a model was trained on, the impact feedback loops on a model once deployed and at scale, and so forth) were also key issues for building profitable trading models. The kinds of things you need to do to consider societal consequences are going to have a lot of overlap with things you need to do to consider organizational, market, and customer consequences—so thinking carefully about ethics can also help you think carefully about how to make your data product successful more generally!

来自使用二元逻辑的背景，在伦理学上缺乏明确的答案会让我一开始感到沮丧。然而，我们的工作如何影响世界，包括意想不到的后果和工作被坏人武器化，是我们可以(也应该!)考虑的一些最重要的问题。尽管没有任何简单的答案，但要朝着更合乎道德的行为迈进，还是有一些明确的陷阱需要避免，一些实践需要遵循。

许多人(包括我们!)都在寻找更令人满意的、可靠的答案，来解决技术的有害影响。然而，鉴于我们所面临的问题的复杂性、深远性和多学科性，没有简单的解决办法。ProPublica前高级记者Julia Angwin专注于算法偏见和监视问题(也是 2016 年帮助激发 FAccT 领域的 COMPAS 累犯算法的调查员之一)在2019年的一次采访中说:

:我坚信，为了解决问题，你必须诊断它，而我们仍然处于诊断阶段。如果你想想世纪交替和工业化，我不知道我们有30年的童工，无限的工作时间，糟糕的工作条件，这需要很多记者揭发丑闻和倡导来诊断问题，并了解它是什么，然后行动主义来改变法律。我觉得我们正处于数据信息的第二次工业化…我认为我的职责是尽可能明确缺点是什么，并真正准确地诊断它们，以便它们能够得到解决。这是一项艰巨的工作，需要更多的人来做。

令人欣慰的是，Angwin认为我们在很大程度上仍处于诊断阶段:如果你对这些问题的理解不完全，这是正常和自然的。虽然我们继续努力更好地理解和解决我们面临的问题是至关重要的，但还没有人有“治愈”的方法。

本书的一位审稿人Fred Monroe曾在对冲基金交易领域工作。他告诉我们，在阅读了这一章之后，这里讨论的许多问题(数据的分布与模型所训练的内容有很大的不同，一旦部署并规模化，对模型的影响反馈循环，等等)也是建立盈利交易模型的关键问题。考虑社会后果所需要做的事情将与考虑组织、市场和客户后果所需要做的事情有很多重叠——所以仔细考虑道德问题也可以帮助您仔细考虑如何使数据产品更普遍地获得成功!

Questionnaire

问卷调查

1. Does ethics provide a list of "right answers"?
2. How can working with people of different backgrounds help when considering ethical questions?
3. What was the role of IBM in Nazi Germany? Why did the company participate as it did? Why did the workers participate?
4. What was the role of the first person jailed in the Volkswagen diesel scandal?
5. What was the problem with a database of suspected gang members maintained by California law enforcement officials?
6. Why did YouTube's recommendation algorithm recommend videos of partially clothed children to pedophiles, even though no employee at Google had programmed this feature?
7. What are the problems with the centrality of metrics?
8. Why did Meetup.com not include gender in its recommendation system for tech meetups?
9. What are the six types of bias in machine learning, according to Suresh and Guttag?
10. Give two examples of historical race bias in the US.
11. Where are most images in ImageNet from?
12. In the paper "Does Machine Learning Automate Moral Hazard and Error" why is sinusitis found to be predictive of a stroke?
13. What is representation bias?
14. How are machines and people different, in terms of their use for making decisions?
15. Is disinformation the same as "fake news"?
16. Why is disinformation through auto-generated text a particularly significant issue?
17. What are the five ethical lenses described by the Markkula Center?
18. Where is policy an appropriate tool for addressing data ethics issues?

1. 道德是否提供了一份“正确答案”的清单?
2. 在考虑道德问题时，与不同背景的人合作如何有帮助?
3. IBM在纳粹德国扮演了什么角色?为什么公司会这样参与?工人们为什么要参与?
4. 在大众柴油丑闻中被判入狱的第一人的角色是什么?
5. 由加州执法官员维护的犯罪团伙嫌疑人数据库有什么问题?
6. 为什么YouTube的推荐算法会把不穿衣服的孩子的视频推荐给恋童癖者，即使谷歌的员工没有编写这个功能?
7. 度量标准的中心性存在什么问题?
8. 为什么Meetup.com没有将性别纳入其技术聚会推荐系统中?
9. Suresh和Guttag认为，机器学习中的六种偏见是什么?
10. 举两个美国历史上种族偏见的例子。
11. ImageNet中的大多数图像来自哪里?
12. 在"机器学习是否能自动化道德风险和错误"一文中，为什么鼻窦炎被发现可以预测中风?
13. 什么是表示偏差?
14. 机器和人在做决定方面有什么不同?
15. 虚假信息和“假新闻”是一样的吗?
16. 为什么通过自动生成文本传递虚假信息是一个特别重要的问题?
17. Markkula 中心描述的五种伦理视角是什么?
18. 政策在哪些方面是解决数据伦理问题的合适工具?

Further Research:

进一步研究：

1. Read the article "What Happens When an Algorithm Cuts Your Healthcare". How could problems like this be avoided in the future?
2. Research to find out more about YouTube's recommendation system and its societal impacts. Do you think recommendation systems must always have feedback loops with negative results? What approaches could Google take to avoid them? What about the government?
3. Read the paper "Discrimination in Online Ad Delivery". Do you think Google should be considered responsible for what happened to Dr. Sweeney? What would be an appropriate response?
4. How can a cross-disciplinary team help avoid negative consequences?
5. Read the paper "Does Machine Learning Automate Moral Hazard and Error". What actions do you think should be taken to deal with the issues identified in this paper?
6. Read the article "How Will We Prevent AI-Based Forgery?" Do you think Etzioni's proposed approach could work? Why?
7. Complete the section "Analyze a Project You Are Working On" in this chapter.
8. Consider whether your team could be more diverse. If so, what approaches might help?

1. 请阅读这篇文章“当一个算法削减你的医疗保健时会发生什么”。将来怎样才能避免这样的问题呢?
2. 通过研究了解更多关于YouTube的推荐系统及其社会影响。你认为推荐系统必须总是有负面结果的反馈循环吗?谷歌可以采取什么方法来避免它们?政府呢?
3. 请阅读"在线广告投放中的歧视"一文。你认为谷歌应该为Sweeney博士的遭遇负责吗?什么才是合适的回应?
4. 跨学科团队如何帮助避免负面后果?
5. 请阅读《机器学习是否会自动化道德风险和错误》一文。你认为应该采取什么行动来处理本文中确定的问题?
6. 阅读文章《我们将如何防止基于人工智能的伪造?》你认为Etzioni提出的方法可行吗?为什么?
7. 完成本章中的“分析你正在做的项目”部分。
8. 考虑你的团队是否可以更加多样化。如果是这样，哪些方法可能会有帮助?

Deep Learning in Practice: That's a Wrap!

实践中的深度学习：那是一个总结！

Congratulations! You've made it to the end of the first section of the book. In this section we've tried to show you what deep learning can do, and how you can use it to create real applications and products. At this point, you will get a lot more out of the book if you spend some time trying out what you've learned. Perhaps you have already been doing this as you go along—in which case, great! If not, that's no problem either... Now is a great time to start experimenting yourself.

If you haven't been to the book's website yet, head over there now. It's really important that you get yourself set up to run the notebooks. Becoming an effective deep learning practitioner is all about practice, so you need to be training models. So, please go get the notebooks running now if you haven't already! And also have a look on the website for any important updates or notices; deep learning changes fast, and we can't change the words that are printed in this book, so the website is where you need to look to ensure you have the most up-to-date information.

Make sure that you have completed the following steps:

• Connect to one of the GPU Jupyter servers recommended on the book's website.
• Run the first notebook yourself.
• Upload an image that you find in the first notebook; then try a few different images of different kinds to see what happens.
• Run the second notebook, collecting your own dataset based on image search queries that you come up with.
• Think about how you can use deep learning to help you with your own projects, including what kinds of data you could use, what kinds of problems may come up, and how you might be able to mitigate these issues in practice.

In the next section of the book you will learn about how and why deep learning works, instead of just seeing how you can use it in practice. Understanding the how and why is important for both practitioners and researchers, because in this fairly new field nearly every project requires some level of customization and debugging. The better you understand the foundations of deep learning, the better your models will be. These foundations are less important for executives, product managers, and so forth (although still useful, so feel free to keep reading!), but they are critical for anybody who is actually training and deploying models themselves.

恭喜你!你已经读完了这本书的第一部分。在本节中，我们试图向您展示深度学习可以做什么，以及如何使用它来创建真正的应用程序和产品。在这一点上，如果你花点时间去尝试你所学到的东西，你会从书中获得更多。也许你已经这样做了——在这种情况下，很好!如果没有，那也没有问题……现在是开始尝试的好时机。

如果你还没有访问过这本书的网站，现在就去看看。准备好运行这些笔记本是非常重要的。要成为一名有效的深度学习实践者，关键在于练习，所以你需要训练模型。所以，如果你还没准备好，现在就去拿笔记本吧!也可以在网站上查看任何重要的更新或通知;深度学习变化很快，我们无法改变书中内容，所以你需要查看网站，以确保你拥有最新的信息。

确保你已经完成了以下步骤:

• 连接到本书网站上推荐的GPU Jupyter服务器之一。
• 自己运行第一个笔记本。
• 上传你在第一个笔记本上找到的图片;然后尝试一些不同类型的不同图片，看看会发生什么。
• 运行第二个笔记本，根据您提出的图像搜索查询收集您自己的数据集。
• 考虑如何使用深度学习来帮助您完成自己的项目，包括您可以使用哪种类型的数据，可能会出现哪种类型的问题，以及您可能如何在实践中缓解这些问题。

在本书的下一部分中，您将了解深度学习的工作原理和原因，而不仅仅是了解如何在实践中使用它。理解如何以及为什么对实践者和研究人员都很重要，因为在这个相当新的领域中，几乎每个项目都需要一定程度的定制和调试。你越了解深度学习的基础，你的模型就会越好。这些基础对于高管、产品经理等人来说不那么重要(尽管仍然很有用，所以请继续阅读!)，但是对于任何实际训练和部署模型的人来说，它们都是至关重要的。