Weapons of Math Destruction: How Big Data Increases Inequity and threatens Democracy was written by Cathy O’Neil. It is a very important book on the current state of our democratic republic. It explores how powerful new mathematical algorithms can be used to exacerbate existing wealth gaps and widen cultural gaps. O’ Neil also examines how these same algorithms can be used to demote populations and to create false majorities. The implications go far beyond this and reveal how algorithms can be misused for political influence at the local, national, and international levels.
Weapons of Math Destruction: How Big Data Increases Inequacy and Threatens Democracy examines how algorithms have been misused to create policies that benefit only the economically well-connected and to push political agendas that benefit only those with advanced educational degrees. O’ Neil shows how poor policies that benefit the well-connected can sometimes become popular policies. She explains how using statistical techniques to detect and measure the connections between economic drivers and popular opinion creates the opportunity for policy makers to override popular sentiment and create policies that serve their narrow interest groups. Additionally, she describes in detail how using mathematics to predict price discrimination can lead to mass discrimination against minorities and women. These policies may increase inequality in access to health care, affordable housing, and jobs; lower average IQ scores for men; increased incarceration rates; increased poverty and hunger; increased racial and gender gaps; and widening gaps in educational opportunities.
While all of these problems are important, we may not even know they exist. O’ Neil describes how algorithms can classify data so that classifications are politically motivated, scientifically valid, or even irrelevant. Classification may even be based on partial information or erroneous data. Further, she demonstrates how this affects our understanding of how markets and prices work. Further, she describes how changing how we classify problems creates a feedback loop that makes it harder to solve problems because of the artificial limitations placed on our ability to classify. Further, she demonstrates how even our attempts to measure and classify correctly can produce inaccurate measurements.
Weapons of Math Destruction by Rebecca O’ Neil makes a valuable point about how even using the most sophisticated algorithms and mathematical techniques can lead to false solutions. In fact, she points out that sometimes the best solutions are the simplest. She shows how sometimes using complex formulas that optimize over a large number of parameters can actually solves more problems than if the algorithm had been used to search for a single variable separately. Further, she points out how even simple techniques can lead to unintended consequences that make the problem worse. For example, if an algorithm is used to find the minimum value of a floating point number, and that number changes as time passes, then in the future that same number might no longer be considered small enough. This could create volatility in financial markets, or cause investors to lose confidence in the currency that is tied to that number.
Using tools such as the weapons of math destruction demonstrates how even well-intended algorithms can have undesired consequences. In addition, it illustrates the danger in relying too heavily on advanced mathematics and numerical techniques, such as mathematical recidivism. In her paper, O’ Neil makes the case that relying on too much recidivism can lead to the development of “black holes,” whereby the very properties that made a system valuable in the first place are rendered meaningless because of an unexpected shock to its mathematical workings. Such a black hole may suck the entity out of an equilibrium, leaving an empty landscape with all of its properties damaged. If a problem goes too deep, a black hole is able to swallow up the entities of a system, leaving no recourse but to restart the whole process from scratch. As this happens, humans are left without any protection against a weapons of math destruction that were designed to destroy their world.
The dangers of relying too heavily on advanced mathematics are clear. However, even if we take advanced mathematics at its face value and use it only to calculate future metrics based on past performance, we still run into problems. The biggest problem is that even though algorithms will solve some of the simpler problems, they often break down complex problems, particularly for non-programmers. Even if you have access to an algorithms designer, the designer is not likely to be familiar enough with your business or your industry to make changes to the algorithms that he designs without your authorization. You could ask, but you really don’t want to.
One of the most telling things about how the weapons of math destruction threaten democracy is that even the political scientists who specialize in studying how technology affects society don’t seem to fully understand what’s going on. Whereas the general public took hold of the technology of the early 1960s, most political scientists remained skeptical. The reason was that it was difficult to see how the use of computers and information systems would actually change people’s behavior. Today we know how the effects of big data are changing everything from how we communicate with one another to how we behave in government agencies, but not all of us have the instincts to see what’s going on.
In the case of policing, the outcome of bad law enforcement decisions is not just a matter of technology. It’s a matter of recidivism, which is the number of people who return to crime once they’ve been in trouble. While many jurisdictions have reduced their maximum recidivism rates, many courts are still setting arbitrary rates that are arbitrary. When a judge or a prosecutor gives the go-ahead for a case, no matter how badly the algorithm worked, the same goes for how the formula was used. A remedy for bad algorithms might be as simple as giving judges and prosecutors more discretion when making their judgments.