Amazon built a program to read job applications and give each one a score. Then its engineers noticed what it had taught itself about women.
Ten items. Say your answer out loud before you reveal it.
Read the sentence, guess the meaning of the word in bold, then reveal.
Three terms: rely on · a black box · reproduce a pattern. Say the right one, then reveal.
One missing word fits every phrase in the group. Say it, then reveal.
Read each part. After it, answer the quick check from memory, then discuss one question.
In 2014, a team of machine-learning engineers at Amazon began work on a program that would read applications. Amazon receives a very large number of CVs, and reading them takes a very large number of recruiters. The idea was easy to state: give the program a pile of CVs and it picks out the best five, the way a shopper picks a product with five stars.
The tool did exactly that. It scored each candidate from one to five stars. To teach it what a good candidate looks like, the team fed it the CVs that had been sent to the company over the previous ten years, together with what had happened to them.
The method is a sensible one, and it is how many such programs are built. The difficulty is in what it assumes: that the past was fair.
1. What was the program supposed to do?
2. How did it score candidates?
3. What data was it trained on?
1. Read a large number of CVs and pick out the best ones, so recruiters would not have to.
2. From one to five stars, like a product rating.
3. CVs that Amazon had received over about ten years, with the outcomes.
By 2015, the engineers saw that the scores were not gender-neutral, at least for software jobs and other technical roles. Most of the CVs in the ten years of data came from men, which reflects how male the technology industry was, and the program drew the obvious conclusion. Male CVs looked like success. So it marked down CVs that included the word "women's", as in "captain of the women's chess club", and, according to people who spoke to Reuters, it downgraded graduates of two women's colleges.
The engineers corrected those particular signals. The trouble was that they could not be certain what else the program had learned. It was looking for patterns they could not see, and removing one pattern did not guarantee the next one had not taken its place.
Recruiters had looked at the tool's recommendations, though, according to the sources, they never relied on its ranking alone. That helped a little. It did not make the scores meaningful.
1. What did the engineers find in 2015?
2. Why did the program penalise some CVs?
3. Why could the engineers not simply fix the problem?
1. That the scores for technical jobs were not gender-neutral.
2. Most of the CVs it learned from came from men, so it treated male CVs as the pattern of success.
3. They could remove the signals they found, but they could not be sure the program had not learned other, hidden ones.
Amazon never launched the tool as the only way of choosing people. By the start of 2017, according to Reuters, executives had lost hope in the project and the team was disbanded. Amazon did not comment on the story when it appeared in October 2018, but it said that it was committed to diversity and equality in its workplace.
Nobody was fined and there was no court case, which is part of why the story is useful. It shows a mistake found early, by the builders, and corrected by stopping. Most companies are less lucky, and a screening tool bought from a supplier may give its users much less to see.
The lesson for anyone who recommends AI to a client is not that these tools never work. It is that you should ask what a tool learned from, and whether anyone can explain what it does. If the answer to the second question is no, the recommendation has to be much more careful.
1. What happened to the team by the start of 2017?
2. Why is the story still useful if nobody was fined?
3. What two things should a consultant ask about an AI tool?
1. It was disbanded, because executives had lost hope in the project.
2. It shows a problem found early by the builders and corrected by stopping, which is rarer than a problem found afterwards.
3. What the tool learned from, and whether anyone can explain what it does.
When a tool offers to rank, score or sort people, two questions decide how far to trust it. The first is whether it saves real time. The second is whether you could explain, to the person it affects, why it chose as it did. Put the two answers on a grid and the recommendation almost makes itself.
Each line is part of a recommendation about an AI screening tool. Is it giving a reason for it, a reason against it, a condition, or a hedge that says nothing? Choose, then check.
1. For — a clear benefit: real time saved.
2. Against — a specific risk, with a reason.
3. Condition — a rule you could check, which makes the tool safer.
4. Empty hedge — it says nothing a listener could use or argue with.
5. Against — if you cannot explain a decision, you cannot defend it.
6. Empty hedge — which factors? The sentence avoids the work of naming them.
7. For — consistency is a genuine strength, as long as the rules are fair.
8. Condition — a named check, with a time, that you could carry out.
Not every tool is a black box. A simple filter, such as "has the required licence", is easy to explain. The test matters most for tools that rank or score people.
Humans are biased too. A fair recommendation says so. The question is not whether the tool is perfect, but whether you can see and correct its mistakes more easily than a person's.
The law can matter. Hiring decisions are regulated in many countries, and some places now require employers to tell candidates when automated tools are used. Check the local rules before recommending anything.
A good recommendation shows both sides before it chooses one. The grammar of balance is mostly small connecting words, and each one changes how fair you sound.
Choose the best connector from the box. Use each once: although · whereas · even so · on balance.
1. the tool was fast, its scores could not be explained.
2. Recruiters read a few CVs a day, the program could read thousands.
3. Humans are biased too. , a person can be asked to explain a decision.
4. , I would recommend using it for sorting but not for ranking.
1. Although — introduces the point being set aside
2. whereas — compares two things
3. Even so — accepts the point, then continues
4. On balance — gives the decision after weighing both sides
Rewrite each flat sentence with the connector in brackets so that it sounds balanced.
1. The tool is quick. It cannot explain its choices. (Although)
2. It is consistent. It may be consistently unfair. (While)
3. A person reads slowly. A program reads fast. (whereas)
4. It saves money. We would still keep a person in the loop. (Even though)
1. Although the tool is quick, it cannot explain its choices.
2. While it is consistent, it may be consistently unfair.
3. A person reads slowly, whereas a program reads fast.
4. Even though it saves money, we would still keep a person in the loop.
Now give a balanced view of one tool you use at work, in four sentences. Use two different connectors and finish with on balance.
Three tasks about judging an AI recommendation. Read the situation, take a moment, and speak. Aim for about a minute each.
Retell the Amazon case in about a minute to someone who has never heard of it. Say what the tool was for, what it learned, and why the project ended.
A client who runs a recruitment business asks whether AI should screen the first thousand CVs for a vacancy. Give the case for it in thirty seconds, the case against it in thirty seconds, and then your own recommendation with one condition. Use the grid to place the tool.
You are advising a company that used an automatic tool to reject a candidate. She calls and asks why. The person you are speaking with plays the candidate. Explain what you can, say plainly what you cannot, and say what happens next.