Bias in Search Engines And Algorithms

Elizabeth Brico -- a Cuban mental health blogger -- believes Florida's use of predictive analytics in their child welfare system determined her risk status and influenced her investigator's recommendation to take her children away from her. Brico and her children have become distressed in the wake of their separation. Her situation is hardly an exception. Brico is one of many mothers of color who are unfairly judged by an algorithm and the humans attempting to make decisions from the data it provides.

Olson's article reveals how a popular data-backed model, called "word embedding," has picked up on deeply entrenched social biases which leads to services like Google Translate spitting those biases back out. Olson cites an example of translating Turkish phrases (which include gender-neutral pronouns) to English. "O bir muhendis" becomes "he is an engineer" while "o bir hemsire" translates to "she is a nurse."

In 2017, ProPublic analyzed a computer program called Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) used in courthouses to help assign criminal sentences. The software flagged black defendants as twice as likely to be misclassified as higher risk as compared to their white counterparts. They found numerous other instances of biases within the program as well.

Beauty.Ai was the first software algorithm to be used in an international beauty contest judged by “machines.” It should have used objective factors such as facial symmetry and wrinkles to identify the winners. Approximately 6,000 people from more than 100 countries submitted photos for the contest. Out of 44 winners, nearly all were white, a handful were Asian, and only one had dark skin. However, many people of color submitted photos to the contest; including large groups from India and Africa.

For-profit colleges admit the highest amount of African-American students, and it isn't because of a push for more diversity in higher education. The recruiters at these universities are trained to prey on "isolated, impatient” individuals with low self-esteem" who have few people in their lives who support them and a grim future outlook. Some research shows that degrees from these schools are worth about as much as a high school diploma in the workplace, thus accruing more debt for already underprivileged groups. The weapon that pulls these students in? Google's targeted advertising. The University of Phoenix is one of Google’s largest advertiser, spending up to $200,000 a day on advertising. In comparison, it spends as little as $700 per student per year.

PredPol, a predictive policing software, is designed to help police stop crimes before they occur by analyzing locations where previous arrests occurred. It's used by over 60 police forces in the US. However, the American Civil Liberties Union (ACLU), the Brennan Center for Justice, and other civil rights organizations have raised questions about the risk of bias. Historical data from police practices can create a feedback loop where police decisions, instead of actual reported crimes, inform the algorithm. This trend could lead to inappropriately labeling certain neighborhoods as "bad" and others as "good." Another possible issue could be that police may be more inclined to use more force in neighborhoods they believe are "bad."

O'Neil's article follows the story of a Vanderbilt graduate named Kyle Behm. A year and a half after receiving treatment for bipolar disorder, Behm applied for a job at Kroger supermarket. He had a friend who could put in a good word for him and seemingly had a strong chance of landing a job. Behm never got called back for an interview. His friend explained to him that a personality test -- created by Kronos, a workforce management company based outside Boston -- screened him out of the job. Behm recalled the test was similar to the "five factor model" tests he took during his treatment for bipolar disorder. Behm applied to other minimum-wage jobs and encountered the same situation. He filed an ongoing class-action lawsuit against seven companies, including Home Depot and Waglreens, alleging the use of the personality test was unlawful.

Karlis' article reveals how Amazon's deep-learning software, called Rekognition, used in many facial recognition algorithms contains gender and racial bias. MIT Media Lab researchers found the technology had trouble discerning gender and skin tone when the individual was a woman or had darker skin. More specifically, Rekognition misidentified women for men 19 percent of the time, and darker-skinned women for men 31 percent of the time. It made no mistakes when identifying men with light skin.