This summer, I joined a data science & society-themed book club sponsored by the Harvard Data Science Initiative. Our first book was The Alignment Problem: Machine Learning and Human Values by Brian Christian (2020). It turned out to be one of my favorite nonfiction books. Someone in the group also let us know about a free virtual event, the Symposium on Interpretable and Explainable Artificial Intelligence and Machine Learning, hosted by the National Academies of Science Committee on Applied and Theoretical Statistics, which I ended up attending. (If you're interested in this Symposium, a recording is available here.)
Between the reading, the Symposium, and our book club's weekly discussions, there are a plethora of topics about which I could wax lyrical. For the sake of a bite-sized post, however, I will stick to a few overarching themes that stuck out to me: (1) the development of artificial intelligence (AI) giving us insights into our own brains, (2) the ability (or lack thereof) for individual people or minority groups to pursue justice for maltreatment at the "hands" of an algorithm, and (3) the perpetual need for humility (for both humans and AI).
“Our successes and failures alike in getting these systems to do 'what we want,' it turns out, offer us an unflinching, revelatory mirror.”
- Brian Christian
Similarities and Differences Between Machine and Human Learning
The Alignment Problem steps through many key developments in AI, with lots of fun anecdotes to help readers peek behind the publication curtain to see what researchers were thinking and struggling with. This progression really helped me to see why, for instance, animals (including us) require both extrinsic and intrinsic motivations (e.g. hunger and curiosity) to thrive in a variety of environments. I'm also now able to articulate both the benefits and potential consequences of imitation: you can learn things that would otherwise be really difficult to explain or craft incentives for, but it may not work out if you attempt to copy something that you're not equipped to do and/or don't understand the motivations for. Also, it's hard to ever surpass the teacher if you're just trying to match them.
In retrospect, I see a lot of parallels to my personal experience learning to play the clarinet, which is by far my most structured extended learning experience. When you start out, you cling to the rules (of instrument technique and music in general) for dear life in an attempt to get off the ground: you try to do exactly what your teacher tells you, use a metronome to keep time and a tuner to check pitch. Eventually, it helps to have less of this internal rigidity and to start paying more attention to what feels right, trying to follow the "flow" of the music. Throughout this process, having a teacher to imitate and/or listening to recordings of other people playing can help you progress by leaps and bounds. At the same time, you likely won't be able to play a piece as quickly or with as great a dynamic range as your teacher, and it will often sound worse if you try to mimic them in these ways right off the bat.
Another cool development in AI has been the emergence of techniques to train AI against themselves, which I eventually recognized as a version of my clarinet teacher's motto that the goal of taking lessons was to become my own best teacher (either in real-time or by recording and then listening to myself). A final memorable example of AI-human similarities in The Alignment Problem detailed how the development of an algorithm that allowed a computer to learn from changes in its expectation about the future (termed Temporal Difference Learning) ended up closely describing how our brains produce dopamine (a "happy hormone"), which was a huge breakthrough for neuroscience.
In addition to all these similarities, there are also many ways that machine and human learning differ. For instance, "saliency" methods have shown that when some AI are trying to classify images as dogs or not-dogs, they are often looking at features like the blurriness of the outline of the object/subject in the middle of the image, as opposed to humans tending to look at the overall shape. This causes problems if we want to be able to interpret the output from algorithms: forcing computers to use our ontologies (language/thinking structure) might facilitate our understanding but prevent superhuman inference and/or be stymied by competing human ontologies**.
Further, regarding the earlier point about Temporal Difference Learning: although encountering observations that conflict with our mental models offers great learning opportunities, people often resist changing our opinions, sometimes even ignoring new data instead of doing the hard cognitive work of updating our expectations. In this way, we probably don't want AI to be too similar to human cognitive behavior. On the flip side, while certain strategies developed for guiding the behavior of AI might be applicable to humans, they may not necessarily be good for us. For example, in the book club we discussed how gamifying various aspects of our lives can in some cases be helpful (e.g. rewarding ourselves for getting work done) but in other cases are harmful (e.g. becoming addicted to video games because they are more immediately satisfying than real life).
Actionable Recourse for Algorithmic Injustice
Over the last six years, there has been an explosion of research and discussion about algorithmic bias against certain groups of people. Before reading The Alignment Problem, I actually hadn't realized how young this field is because I started college in 2016, so I've heard about it my whole academic career. Now, in 2022, it seems like there are hundreds of different definitions of algorithmic fairness, along with mathematical theory showing that you can't obtain them all at the same time, except possibly under very specific (unrealistic) conditions.
One of the really interesting issues that my partner (who just graduated from law school) and I were talking about, and that then came up in the Symposium, is the lack of legal infrastructure for dealing with decisions made (or strongly informed) by algorithms. For instance, the US legal system currently allows appeal for decisions made by a biased jury, but not by a biased algorithm. Financial institutions are somewhat more regulated in how they may use models to inform lending decisions. However, as Patrick Hall (a professor at George Washington University) noted: even if, for instance, people are entitled to some notice of a few key features that factored into why they were denied a loan, this may be wildly insufficient for explaining an individual prediction from a (potentially proprietary) high-dimensional, nonlinear machine learning model that was designed to make the company profits when the results are averaged across all their clients. This is to say, we have a long way to go in truly providing individuals with actionable recourse against algorithms. If you're interested in learning more on this topic, I'm linking some resources at the end of this post.
Cultivating Uncertainty(?)
During the panel discussion at the Symposium, a common scientific concept was raised: the more independent teams/models arrive at the same conclusion, the more reason to trust in that conclusion. This intuition motivates a general technique that researchers have started using to quantify the uncertainty of AI: generate a lot of results from different models or by perturbing the same model in different ways (e.g. cutting off different parts of the model from contributing to each prediction), and see how similar all the results are to each other. More spread indicates more uncertainty in the model prediction. (Side thought: time will tell whether quantification of model uncertainty ends up playing a role in people's recourse from AI-based decisions.)
However, this approach really only works if all the models / sub-models are sufficiently distinct or independent of one another. In The Alignment Problem, Christian describes how the Effective Altruism movement (see this post) struggles with maintaining moral uncertainty (i.e. for what global causes to focus attention and resources on) while simultaneously distributing educational resources and encouraging discussion between people all around the world. Similarly, while the push for open science / reproducibility and transparent decision making has many benefits, including increased methodological scrutiny and less duplicated work, it seems that there may be some downsides to encouraging everyone to do things the exact same way, using the exact same tools.
In data science, for instance, using the same packages in R or Python streamlines analyses, but could also potentially produce less evidence across multiple studies on the same topic because the work is not being done independently. This would of course be most concerning if a commonly-used package had a mistake in it, but even if the methods were all correct, the convenience of using a ready-made package might result in the community missing out on other ways to think about a problem that would provide different insights. Similarly, if the vast majority of publicly available datasets are biased towards white males, a consensus across these datasets is not necessarily a better representation of the population overall. (Note: I'm certainly not arguing for a return to siloed and/or poorly documented workflows -- just trying to articulate a point that I haven't heard mentioned as much compared to the huge push for open science.)
Even if we thought we had a robust uncertainty estimate, maintaining a heightened sense of humility is especially important for AI because we want to avoid a situation in which the machines are no longer amenable to human input. Private and/or public sector creators of AI systems might also want machines to be even more careful than human actors would be for many practical applications, as they will often be judged more harshly by the public (and potentially, in the future, by the law). For an example of this, I'll draw from The Alignment Problem's description of the relationship between uncertainty and impact: if a self-driving car is unsure whether there's a person in the road in front of it, then the best course of action is to slow down so as to minimize adverse outcomes.
To summarize: on her concluding presentation slide at the Symposium, Been Kim (a research scientist at Google Brain) wrote, "One of the biggest challenges [in interpreting/explaining AI] is to resist building trust and be skeptical." I believe she meant this in the context of humans interpreting a single model, but the sentiment could also easily be applied to humans interpreting ensembles of dependent models and/or data, as well as to AI continuing to make decisions after being deployed for a while.
Conclusion
I am far from being an expert in these topics, but I hope that my rambles have left you with some interesting thoughts and/or whetted your appetite to go learn more from people who've studied these issues in depth. My overarching takeaways from all of this are that (1) there's a lot of interesting work being done in this field, (2) there's a lot more to do to ensure AI systems live up to our ideals, (3) we have the opportunity to learn a lot about ourselves along the way, and (4) as whizzy as our tech gets, humanity will need to stay humble to avoid losing ourselves.
** Here is a 2022 article that describes how researchers are using team psychology to help humans learn to collaborate with robots faster and more effectively.
Here are some interesting resources regarding AI standards / regulation / actionable recourse against AI: A United States Fair Lending Perspective on Machine Learning (2021); Challenging Decisions Made by Algorithms (2021); NIST: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022); https://incidentdatabase.ai