Thanks to Adam Shimi, Alex Turner, Noa Nabeshima, Neel Nanda, Sydney Von Arx, Jack Ryan, and Sidney Hough for helpful discussion and comments.

Abstract

We analyze agents by supposing a Cartesian boundary between agent and environment. We extend partially-observable Markov decision processes (POMDPs) into Cartesian world models (CWMs) to describe how these agents might reason. Given a CWM, we distinguish between consequential components, which depend on the consequences of the agent's action, and structural components, which depend on the agent's structure. We describe agents that reason consequentially, structurally, and conditionally, comparing safety properties between them. We conclude by presenting several problems with our framework.

Introduction

Suppose a Cartesian boundary between agent and environment:^[1]

There are four types: actions, observations, environmental states, and internal states. Actions and observations go from agent to environment and vice-versa. Environmental states are on the environment side, and internal states are on the agent side. Let $A, O, E, I$ refer to actions, observations, environmental states, and internal states.

We describe how the agent interfaces with the environment with four maps: $observe$ , $orient$ , $decide$ , and $execute$ .^[2]

$observe : E \to Δ O$ describes how the agent observes the environment, e.g., if the agent sees with a video camera, $observe$ describes what the video camera would see given various environmental states. If the agent can see the entire environment, the image of $observe$ is distinct point distributions. In contrast, humans can see the same observation for different environmental states.
$orient : O \times I \to Δ I$ describes how the agent interprets the observation, e.g., the agent's internal state might be memories of high-level concepts derived from raw data. If there is no historical dependence, $orient$ depends only on the observation. In contrast, humans map multiple observations onto the same internal state.
$decide : I \to Δ A$ describes how the agent acts in a given state, e.g., the agent might maximize a utility function over a world model. In simple devices like thermostats, $decide$ maps each internal state to one of a small number of actions. In contrast, humans have larger action sets.
$execute : E \times A \to Δ E$ describes how actions affect the environment, e.g., code that turns button presses into game actions. If the agent has absolute contro

...

The 2021 Review