Navigating the Nuances: The Unseen Dynamics of AI’s Existential Risk

Unraveling the Complex Interplay of Agency, Probability Inflation, and Future Implications in the Realm of Artificial Intelligence

This is the second post from the series Impossibility Results in AI related to the work published in ACM Computing Surveys in 2023 (link here). These series present the findings in an approachable manner which may sometimes trade-off with precision. Here is the link to the first post in the series that briefly introduced AI and existential risk. In this series, the term’ agency’ refers to the ability of something to create and pursue its own goals independently. An agent is everything that possesses that ability, for example, humans, organizations, future AI systems, etc.

Humanity faces various risks, including environmental, health, social, technological, and existential risks. Among these, the most dangerous risks are often complex combinations and interactions between elements, presenting significant existential threats.

The Different Faces of AI Risk: Social Vs. Existential

First, let’s distinguish between social and existential risks posed by AI, common topics these days. Social risks are those where certain groups benefit at the expense of others, leading to injustice or inequality. These risks range from discrimination and economic inequality to job displacement, technological unemployment, and information manipulation. Addressing these imminent and nearly inevitable threats cannot be overstated. However, they imply a degree of human control over the situation, suggesting that we can adopt social measures for more equitable outcomes. I will return to this topic some other time.

The concept of existential risk encompasses threats that could lead to humanity’s extinction or irreversible decline, endangering our existence as a species. These risks transcend conventional daily hazards and can cause irreparable harm on a global scale. Existential risks, unlike social ones, offer no intentional benefits to anyone. They highlight our lack of control over negative outcomes, instead relying heavily on elements of chance or luck. Although less probable and more distant than social risks, they present a strictly harder problem due to our lack of control.

The Unique Nature of AI’s Existential Risk: The Power of Agency

When considering existential risks, various sources such as environmental (asteroid impacts, climate change, hostile alien species), technological (uncontrolled AI, runaway nanotech, dangerous biotech), and social (nuclear war, tech-empowered terrorism) come to mind.

The risks that stand out are the uncontrollable AI and hostile alien species because they possess what is known as ‘agency.’ These entities involve having coherent goals and making decisions to optimize outcomes, presenting a focused threat to humanity.

In contrast, threats like nuclear events, biotech hazards, climate change, or asteroid collisions are considered ‘passive.’ They don’t act against us with specific intent, and their effects are widespread rather than focused.

So, why is the agency aspect of AI existential risk problematic? Because it introduces many relevant ‘unknown unknowns.’

The Problem with Agency in AI

An agent is a distinct entity within its surrounding environment that exhibits behavior and goals unique to itself, setting it apart from its surroundings. It doesn’t merely react to its surroundings but possesses agency.

Powerful agents, especially those driven by AI, can be dangerous. Why? Well, they can do things that are hard to predict or expect. In physics that describes the environment, things are typically straightforward and governed by clear rules. However, when complex entities like humans and AI come into the picture, an additional layer of complexity arises.

These powerful agents can make unlikely things happen. They can achieve outcomes that have a very low chance of occurring spontaneously. Let’s take drinking water as an example. Normally, it’s unlikely for water to appear in our mouths magically. But we make it happen when we actively reach, grab, and take a drink. Doing that increases the chance of something we want to happen, even though it doesn’t spontaneously occur very often.

The same idea applies to powerful AI agents. They can make possible things happen that might otherwise seem highly unlikely. They can inflate non-zero probabilities (p>0) beyond what we expect or intuitively estimate. That is because they can imagine things and then use powerful capabilities to make them a reality, changing the world according to their intentions. The more deep and wide capability they have, coupled with unconstrained imagination and acting, the more changes they can enact with greater ease.

The danger is that we might underestimate the risks associated with these powerful agents. We might not realize the extent of the risks we already know about (‘known unknowns’), and even worse, we don’t understand the infinitude of all the ‘unknown unknowns‘ that could arise. It’s essential to be aware that as these agents become more capable, they can achieve even more improbable and potentially disruptive outcomes. This uncertainty coming from unpredictability is why experts often disagree these days. There are no definite answers but a big range of uncertainty due to the inability to estimate ‘unknown unknowns’ arising from powerful agents. That is a reason to be vigilant and cautious.

However, even if we have very capable (deep and wide) agentic AI, this still doesn’t necessarily spell doom. If such AI is not misaligned with our worldviews and preferences, we can still be OK or even benefit. This topic, however, is for future posts. Figure 2 below shows an example of the alignment of interests between rhinoceros and oxpeckers.

Conclusion

Future AI has the potential to make unlikely events happen more often than we might think. That also includes events with extremely low probabilities that may seem absurd but are still possible. Powerful AI agents can boost these chances through their capable actions and intentions, much surpassing our initial estimations. When an event has even the slightest probability of occurring (p>0), the more powerful an AI becomes, the more it can inflate that probability and bring about the event. In other words, as AI grows stronger, it has the potential to make increasingly unlikely events happen.

Understanding the limits of this probability inflation is essential. We need to determine what is truly impossible (with a probability of zero, i.e., p=0) because that is immune to the effects of probability inflation. Our exploration begins here as we investigate the boundaries that apply to all AI agents, regardless of their power. We can find safety and reassurance in understanding these limits. Along the way, we will also realize that these limits are relevant to AI, empirical science, and humans.

**This is a copy of the post from my personal substack “Peregrine”.

Written on: June 17, 2023