Federico Botta, Warwick Business School, University of Warwick

What information can be extracted from online data? How do some private companies exploit our digital traces?

Cathy O’Neil uses her experience working as a data scientist to shed light on some of the grey areas of big data. Each chapter focuses on a different topic in which the author feels that data analytics has contributed to increasing inequality and discriminating against minorities. Using a detailed series of examples, she argues that too often big data has focused on raising profits and efficiency for companies, to the detriment of poor people. From the schoolteacher who’s been fired for ranking low in a new scoring system based on largely unknown algorithms, to the jobseeker whose application is rejected because they fail a non-transparent personality test, Cathy O’Neil raises interesting questions and concerns on the application of statistics to the analysis of large behavioural data sets.

This is a nice and easy-to-read book, which is very well written and suitable for a general audience with an interest in the subject. However, as the title may suggest, it is very clear from the beginning that the author intends to present a very negative view of the topic. Speaking from her own personal experience, and using a great deal of anecdotal evidence, she presents many case studies in which algorithms have resulted in unfair, biased and potentially even discriminatory results. After reading this book, the reader may find it difficult to believe that big data can also offer huge positive opportunities to our society. The book would definitely benefit from a more complete and objective overview of all sides of data science, even just in the form of a final chapter which could help the reader reach a more balanced view of the topic.

In a world increasingly influenced by big data, books like this are necessary and extremely timely. Cathy O’Neil places much emphasis on the lack of transparency typical in several algorithms, which I think is a crucial point in this topic. There is indeed a need for increased transparency, as well as an increased education of the general population that should be aware of the potential use (and misuse) of big data. In fact, academic research in this field has started to consider these issues in recent years. Reproducibility, i.e. the idea that everyone should be able to reproduce the results of your work, is now a key requirement in the publication of scientific papers, and this will increase the transparency of the algorithms behind data analysis. I found the points raised by O’Neil on this subject to be very interesting.

At times, the criticism of the author seems to shift towards a more general criticism of modern society and capitalism, which I think only generates confusion and provides a biased perspective on the matter. This is particularly evident in certain sections, where the criticism of some algorithms is rather generic and oversimplified. Several of the case studies considered in this book are extremely complex and no obvious easy solution is available, so I would argue that any criticism should at least acknowledge the complexity of the situations under consideration.

Overall, I found this to be an interesting book to read, which I would recommend to everyone who wants to have a better picture of our society. I find that a key take-home message of this book is that we need more ‘big data literacy’, a more informed population who has a better understanding of what may happen to their digital traces. It is the role of politicians, academics, and private companies to be more transparent about big data and to make sure that it is used for public good. This is particularly important since several of the critical points of big data raised in this book derive from algorithms that were not carefully designed, which did not include key factors in their models, or which did not take into account any feedback on their accuracy.

All in all, Cathy O’Neil provides an interesting perspective on this topic, which I think would have been strengthened by a more objective presentation. An additional point to keep in mind is that almost all examples are American-based, so the text may be harder to follow for a non-US reader.

Sean Malcolm, School of Mathematical Sciences, Monash University

Cathy O’Neil has a clear message: The increase of big data analysis and mass algorithmic processing in industry and society has had a profound negative impact on many facets of life. In Weapons of Math Destruction, O’Neil hopes to convince her readers of the genuine threat posed by unthinking machines, and the humans who – knowingly or unknowingly – wield them in a way that threatens the social fabric of society, introducing previously inconceivable issues and exacerbating inequality and social stratification.

This doesn’t make for a particularly cheerful read, but O’Neil succeeds in making the book engaging, successfully conveying the seriousness and severity of the misuse of algorithms. Reading this book is very much like learning about climate change for the first time, as O’Neil portrays the detrimental algorithms as huge and unstoppable, and the problems they create as being impossible to solve, too far entrenched in society to be remedied. That said, the issues discussed are intriguing in an intellectual context, both from a political and social justice perspective, and from a STEM-oriented data-driven one.

Each new chapter describes an aspect of life, such as college, employment or advertising, and outlines how a particular harmful algorithm or model has become integral to that area, but has resulted in a plethora of often unintended but negative consequences, many self-perpetuating. For example, a model designed to determine whether or not criminal offenders would reoffend was found to be unfairly unbiased against ethnic minorities (p. 24). While by the later chapters this repetitive structure can be a little monotonous, it does keep the book organised. Each chapter echoes the sentiments of the previous, rhyming like poetry, and O’Neil uses this to illustrate that the same flaws and oversights in algorithmic design manifest themselves in similar ways in a multitude of places.

Despite the rather foreboding image of the world she paints, O’Neil manages to avoid coming across as needlessly alarmist, backing up her claims with sources and logical arguments. She also leverages her experience in both quantitative finance and in data science to convince the reader that she is an authority on the topic, opening the second chapter with her first-hand perspective of the global financial crisis, and the role mathematics played in it (p. 32). While the more sceptical reader may question a few of the specific conclusions she reaches, O’Neil highlights so many examples of large-scale damaging algorithms that the general message and overarching theme of the book is hard to dismiss.

While ostensibly the book may appear anti-technology, O’Neil is certainly no Luddite. She never blames algorithms themselves, but the human implementations and interpretations of them, in particular the tendency to over-value the outputs of some algorithms and use them as a substitute for nuanced decision making. She often mentions that harmful algorithms are created in the pursuit of ‘laudable goals’ (p. 3), and it is human incompetence or apathy that leads to unintentional consequences. She outlines a teacher evaluation algorithm implemented to improve education standards in Washington DC, which eventually resulted in the unfair termination of completely capable teachers. However, she takes issue not with the algorithm itself or the decision to implement one, but with the humans who used it to justify their actions without knowledge of how it worked or how valid its conclusions were.

There are no tedious descriptions of the precise mechanics of these algorithms; instead O’Neil focuses on their severe ramifications, making sure to tell stories in each chapter and emphasising the relevance of destructive models in day-to-day life. For instance, she describes the teacher example mentioned above in the context of an actual affected teacher. This human element is preserved throughout the book, capturing the attention of even those with no interest in maths or data science.

While O’Neil does draw on a variety of different examples, the book is very American-centric, and seems to assume its audience is American; it once refers to the American national motto as ‘Our national motto’ (p. 200). As such, there is little discussion pertaining to the consequences of algorithmic overuse in an international context, and the reader may be left wondering if the world at large faces the same threats as America. While many concepts can be generalised, some ideas, such as social security numbers, are irrelevant to a more a global audience. Other ideas may be relevant, but in different ways that may cause confusion. Additionally, omitting accounts of how unfavourable algorithms affect the world on an international level feels like a missed opportunity.

Two large trends in recent years have been the rise of big data in industry and a clarification of social injustice faced by those less privileged. Weapons of Math Destruction sits at the intersection of these phenomena, and describes the short- and long-term trajectories of both in a compelling and fascinating way.

---

 

To cite either of these reviews please use the following details: Botta, F OR Malcolm, S. (2017), O'Neil, Cathy (2016), 'Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy', Reinvention: an International Journal of Undergraduate Research, Volume 10, Issue 2, http://www.warwick.ac.uk/reinventionjournal/archive/volume10issue2/botta-and-malcolm Date accessed [insert date]. If you cite these reviews or use them in any teaching or other related activities please let us know by e-mailing us at Reinventionjournal at warwick dot ac dot uk.