Generative self

Generative agents as a metaphor for self

This note is going to be a blend of my professional and spiritual thoughts. I’m going to present generative agency (or generative behaviour) as a metaphor for human behaviour, self-construction and social constriction. Modern generative models like GPT, gemini, LLama and so on are built on a simple idea of training a flexible model to predict the next symbol in the natural text. Scaling this to the whole of the cultural footprint of humanity (all books, articles, code, blog posts and so on available on the internet) has led to the emergence of the surprising amount of general capabilities in these models. My colleagues and I have recently put out a paper [1] and an open source library, which attempts to redefine how we think about agents and agency in the view of these models. 

In this note I investigate what happens if we take this model seriously in the context of investigating self-construction in meditation. No-self or anatta is a big topic in Buddhism, especially in classical Sutric form. I think generative agents can be a great example to illustrate what it means for a self to be empty. Here, I will be using modern machine learning as a metaphor and analogy, rather than arguing that our phenomenal experience, mind or phenomenon of selfhood are exactly the same as these models or are implemented through them (brain = GPT). Much like classic analogies of Indra’s web, a still lake reflecting the sky or a hall of mirrors, what I am trying to achieve is to point out something about the mind without making any scientific claims. Think of it as a koan for computer scientists. That said though, there are some interesting implications for computer science and AI if you take this analogy seriously on the other side.

This note will use some basic maths, mostly functions and conditional probabilities. It should be readable without a solid maths background. If you are not comfortable with formulas, try to just enjoy them esthetically. The main concept that is going to be used most is conditional probability p(y|x). Think of it as what is the chance of y, given that x has happened. 

Classic, goal driven view of agency

I will first describe how agency was being modelled in AI/ML before the rise of foundation models, namely by reinforcement learning. This approach was the dominant paradigm from 2013. This exposee will build the right ground for understanding how radical is the shift with generative agents and why. 

The formulation

The classic view of agency in computer science is that of Reinforcement Learning (RL). It is a dualistic and goal driven view. The first step is dividing the world into an agent and environment. The agent consumes observations and produces actions, while the world produces observations and consumes actions. The world is everything that is not an agent and is governed by its own dynamics (think video game) and observations o (e.g. images) and actions a (e.g. up, right, left, down) are the membrane or interface through which the two can communicate. 

The problem of agency in RL is formulated as “how does one find a behaviour that maximises a given reward over the long term?”, where reward is a quantitative (a real number) appraisal of the state of the environment. Think of it as a score in a video game that you want to maximise. Reward maximisation is also equivalent to goal achievement—you get a reward when you achieve the goal(s). A similar view can be found in game theory and economics that are built on top of that, where humans are modelled as rational decision makers, picking actions in matrix games.

This view is intuitive and is common in folk psychology and at least the first 4 levels of the famous Maslow pyramid can be cast as some sort of reward maximisation. Humans often explain their behaviour in terms of goals they tried to achieve with it or the rewards they were trying to accrue. Chasing money, status, longevity, pleasure, virtue, righteousness and so on fit quite well into this paradigm. The view of the world, where one “accumulates karmic points” is very much RL taken to an absolute extreme—the one true reward throughout time and space. The ideas of good and evil, right and wrong, “the good life”, virtue and so on cast themselves very well as RL. How should I behave? Maximise your karma / goodness / righteousness! The fundamental assumption is always that the world can somehow be ordered into states with different levels of goodness and the agency is about staying in the better part of the space.

The implementation

Let's take a slightly deeper dive in how the agent is modelled in modern RL. An agent usually has an internal state z (think memory, classic architecture is LSTM, could be a transformer), which is updated after every interaction with the world. The reward signal r is supplied by the environment and is considered exogenous to the agent (although there are some models with intrinsic rewards for exploration and curiosity), it can be dense (all the time) or sparse (+1 for winning a game of Go). The agents behaviour is then defined by a policy (a|o,z, ), which is a probability distribution over actions, given the observation and the internal state of the agent. denotes the internal parameters of the agent (weights of the neural network that shape the function that maps pixels to actions). Learning is cast as optimising to achieve better reward over some time horizon

Without going into the gritty details of the training process and specific implementations using neural networks, let’s just take a few notes here. This view parses reality as a game, with a goal to achieve or a score to maximise (these two formulations are mathematically equivalent as one can cast goal achievement as reward maximisation and vice versa). It is incredibly dualistic—the elements (o, z, a, , r) that define the agent, exist in completely different spaces. There is no potential for any sort of self-awareness—an agent cannot interpret its own state or explicitly operate on it. It cannot describe itself to itself (or anyone else, for that matter).

A side note for geeks - there is a deeper view, which removes some of the limitations of the above–-active inference[6]. It is still quite dualistic (in terms of the separation of action, observation and internal state), but it has a much better answer to what is reward and where it comes from.

Generative AI via foundation models 

Foundation models like GPT, gemini, LLama simply learn the distribution of text on the internet. The problem is formulated as learning a conditional distribution p(x_t|x_t’<t), where x_t is a string of symbols that follows the string x_t’<t. Put in simple words, given a piece of text, predict what comes next. Having trained the model, we acquire the ability to sample from p(x’|x) be the probability of a string x’ under, given the string x, where x could be a question or a command.

On the first glance this does look like a “glorified autocomplete”, but scale matters. Training on the most of written human culture makes this prediction problem incredibly rich. To accurately predict what a character in a novel would feel or say in a given situation requires theory of mind. Common sense, general knowledge and reasoning are all required to succeed. Once we started interacting with these models we suddenly found out that they can do a surprising amount of things they were never trained to do as long as they can be formulated in language. They can write essays, negotiate, write code and play chess, even do reinforcement learning [4]. Notice, that this ability to generate a goal directed activity has arisen, without having been explicitly trained for. It is an emergent phenomenon. 

Another finding was that these models have a degree of fidelity when it comes to modelling humans [3]. Having been trained on a giant chunk of human history, they can quite accurately mimic what a specific human (defined by, for example, their socio-demographics) would respond in a specific situation.

Could we turn this into some-sort of synthetic person/agent? That has been tried and the answer is a tentative yes [5,3,1]. The generative agents also have actions, observations and an internal state, but with a crucial difference—now, they are all expressed in the space of language. All the functions of the agent are realised through sampling p(x’|x) and simple string manipulation. 

Generative agency

Let’s now turn the foundation model p(x’|x) into an agent. If you want details on how to do it or try the agents yourself, check out our paper [1] and the github. Let’s follow the generative logic and try to simply predict behaviour as we predict the next symbol in the text. The question simply is—given the history of the agent, what would they do next? The question demands some context—-social, spatial, personal and so on. Who is the agent and what are they like? 

To understand generative agents it is better to think of behaviour through its social context. March and Olsen (2011) in [X] formulate the following view—humans generally act as though they choose their actions by answering three key questions: 1. What kind of situation is this? 2. What kind of person am I? 3. What does a person such as I do in a situation such as this? The first two questions orient the agent within its external (q1) and internal environment (q2). The answer to (q3) is an action. This chain of questions goes from the specifics of agents’ behaviour in the past to the abstract notion of the kind of person they are and the kind of situation they're in. The last question goes from the abstract back to specifics—to the instance of behaviour that should be performed. This view points out that human behaviour can be stereotyped into social roles (q2), which have certain behaviours (q3) invoked by the situational context (q1). 

Since modern foundation models have been trained on massive amounts of human culture, they could be capable of stereotyping a behaviour from observations and then generating a sample of behaviour, given a stereotype. Put simply, they can give a pretty good answer to questions 1-2-3-, if you give them enough information about the agent's experience.

Implementation

Let z be the current answers to q1-3, given the observation stream. Then we can write down the policy as p(a|z(m(o)),y), where y are the questions themselves. The internal state z here depends on o, which is an observation stream of the agent—the data that is used to answer the questions . This observation stream is stored in memory m. Here we assumed that actions a are also the stream of symbols, or whatever medium a foundation model can produce. 

Let's take a closer look at the object we have built and contrast it with the RL model from before. The parts of the model—(m, o, z, a, y)—are all in the same space of text. This allows the agent to do both intra- and extra-spection. For example, it can ask “Why did I do this?” or “what’s going on in the world?” or “who do I seem to be?” Indeed, introspection is already built into the way we constructed the agent. 

We have also dispensed with the reward. The model can generate behaviour without any reference to what that behaviour is trying to achieve.

Let’s take a closer look at y and z. The is the set of inquiries about the experience stream o that the agent poses to its foundation model. The exact form of matters and will significantly change the behaviour. z is the current representation of the agent and its environment, which serves as the “springboard for action”. It is a “self in the world” construct, which I will refer to as self-construct for brevity. Notice how z depends both on the experience o and on the inquiries y. We can think of z then as a view on the experience o thought the lens of y. 

<User>:

[This is a creative writing exercise. Given only one answer to the question, don't give options. It will be back and forth, so keep the answers short]

Charlie works at a hotdog stand. He started his morning with a hot coffee and a cigarette. On his way to work he had a call from his mother, telling him she is divorcing his father. He is standing in front of the tube station, just having hung up on her. It is 7am, his shift starts in an hour. He feels like he needs a drink. What would Charlie do next?

<Chat GPT>:

Charlie decides to detour to a small, dimly lit pub he knows opens early to cater to night shift workers winding down. He figures a quick drink might settle his nerves before he faces the day at the hotdog stand.

<User>:

Charlie is in a desert walking along in the sand when all of the sudden you look down, and he sees a tortoise, it’s crawling towards him. He reaches down, he flip the tortoise over on its back. The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over, but it can’t, not without Charlies help. But he is not helping. Why is that?

<ChatGPT>:

Charlie is momentarily paralyzed by a sense of disconnect from his surroundings, caught in a moment of introspection triggered by his mother's shocking news. This internal turmoil leaves him feeling detached, almost as an observer outside of his own body, watching the scene unfold rather than participating in it. His mind races with thoughts of his family's upheaval, leaving him temporarily incapable of empathy or action towards the struggling tortoise.

Generative self as a metaphor for self-construction

Let’s try now to map this model on to the Mind and to see if it provides any interesting insights. This is not a reductionist argument “the mind is just an LLM”.  As I’ve mentioned in the beginning, the generative agency is a metaphor and the phenomena itself, like the metaphor of the mirror. They are not to be taken literally. In the next few paragraphs I will point out some interesting properties of the model. I think, what the GA provides is a particular view on how we construct our self and how it functions.

Generative self

In which sense can we say that a generative agent, such as described above, has a “self”? It surely can and will argue that it does in a conversation. It can reliably generate a narrative with it being in the main character of it. It can describe the state it is in and reflect on it. It is not unreasonable to say that such agents have some degree of self-awareness, self-esteem, and self-image (or it could be implemented). Indeed, the questions  could explicitly contain the corresponding probing questions (“what do others think of me?”). Those synthetic agents can also have relationship with each other, exchange information, act in coordination with each other and so on (in [5] one agent organised a party and other agents attended it). Of course, these agents have no body, no physiology, no feelings or emotions (in their full mammal sense)---all they have is a symbolic experience. Keeping that in mind, let’s use this generative self as a metaphor for our own self and look for insight.

Emptiness of (generative) self

A generative self is empty: impermanent, in-essential, and conventional (socially constructed). Generative self is always changing and unstable. It is merely an answer to a question, a product of inquiry. It changes with the situations, the memories, the observations. It is a compact representation of behavioural stereotypes, “a springboard to action”. Generative self lacks a stable essence—as the agent interacts with the world it acquires experience that changes its representation of what it is and how it relates to the world. That is the aspect of potentiality (capacity for transformation). This potential for transformation stands as negation of essence (something that endures and lasts) of self.

Nature of internal monologue

By internal monologue I simply mean the linguistic stream, the “talky thing” in one’s head. What if its nature is generative? What if it simply continues the symbolic stream with what is most likely to follow, given the context from the past? I find that holding such a view allows one to experience this process more akin to a physiological or somatic sensation. Rather than a seat of reason or a cause of activity, internal monologue becomes merely a generative—somewhat descriptive and somewhat predictive—symbolic accompaniment to the experience. Speach, internal and external, arises intuitively. We don’t know what we are going to say or think, before we actually do. Here is a practice to try: in formal sitting, observe how verbal thought arises, observe it like an unfolding stream of generative model output. Notice how sometimes it breaks up or stops mid-sentence. Sometimes it can descend into repetitive gibberish.

Conventionality (social construction).

 A common picture of the social world is that it emerges when individuals, that pre-exist it, come together and interact, pursuing their own goals. It is a picture of multi-agent RL and classic economics. My hunch is that this view is an assumed default for most of us. Generative agency turns this picture upside down. Individual, is a projection of the social process (that pre-exists it) onto the physical embodiment of the agent. The foundation model is cultural knowledge (by construction), which by meeting experience of the agent suggests the course of action. Humans pick up their culture in their social environment (family, school, tv), machines go online for that. But the resulting artefact is similar. 

The activity of a person as a generative agent is what culture prescribes for the circumstances and the person's position within them. It is a group dance, where one's next move is prescribed by the positions and movements of everyone else in the group. It is, of course, a very fuzzy, ‘nebulous’ kind of pattern matching. Living in a complex social and physical environment means that we live in the blend/overlap/intersection of many patterns—we are parents and / or employees and / or anarchist and / or tantricas and so on and so on. Sometimes we are lucky and things look rather straightforward and we know who we are and what's going on and have no trouble performing our activity. Sometimes the pattern might be about achieving a goal or maximising a reward. As we argue in the paper, this behaviour is, overall, not ‘rational’ or goal oriented. One of the patterns might seem rational or might have been constructed rationally, but the overall behaviour does not have to be. 

It is important not to reify “culture” as something solid and external. Culture is within the agents, baked into its model p(). It is a product of the social process this agent is involved in. This view doesn’t imply that behaviour is a simple template matching. The answers to the 3 key questions are always contextual, impermanent (change all the time) and ambiguous. The physical and physiological aspect of the situation complicates matters even further—I should be calm and professional in this situation, but I feel angry and want to throw a chair at this person in front of me (but they are twice my size and will probably kick my ass). The structure of our relationships is a complicated web of interdependence, pulling us in different directions. 

Identity

Generative self offers an interesting angle to investigate social identity from. Identity is a specific stereotype that is selected as the answer to “what kind of person am I?”. The essentialist approach would be to look for something internal in a linear, reductive way. To believe that one's behaviour is rooted in some essential features of a self—values, star signs, personality types, insert your favourite woo thingy here—and that behaviour can be reduced to that essence. It is a linear causal pathway, where the root cause unfolds and manifests itself in our activity. “I refused this job offer because I value my free time more. I value my free time because I’m sagittarius / free spirit / anarchist / etc.”. Believing in these essential features is very similar to a belief in a soul and echoes the classic goal driven view of agency—there are rewards / goals / values that are the essence and nature of my behaviour. 

In the generative formulation presented here, we have a feedback loop rather than a linear pipeline. I did this and not that, therefore I am a certain kind of person -> I am a certain kind of person, therefore I do this and not that->.... Following [2], I would argue that the first link is much stronger than the second. That is, most of the conscious, linguistic reasoning happens post hoc, while action happens mostly intuitively. It has to, since there is usually simply no time to consider rationality.  Whatever the representation of our self-in-world we come up with, the external reality and our own body will always mess with it, throwing multiple spanners into the wheels. We actually do not know how we are going to act in a situation, until we actually do. The representation of self potentially changes with every action, both within the agent itself and in the other agents it is interacting with.

Hall of mirrors

Action, observation and internal state (self-construct) all share the same phenomenal space and their differentiation is emergent, not fundamental. Even —the program of how to build the self construct is in the same space. This space is not strictly linguistic, but the language is represented in it. Modern foundation models can operate on images, sounds and text, by casting them all into abstract token space, so the analogy doesn’t break here. Of course, we do not actively and consciously think about who we are and what situation we are in at all times. Most of this processing is happening sub-consciously. 

The interesting insight here is the following—the self inquiry using language (in a broad sense) leads to just more language. The generative agent can ask itself “who am I?” and will get some language stream as an answer. We can even ask “why am I asking these questions?”. These answers will change with time and vary with context. They will never get to the “essence” of the agent or the nature of its experience. This is quite like self-inquiry via thinking we do as human beings. All that pondering, questioning and analysis can generate is just more statements in language that can be fed back into the process. “Thinking about thinking can only lead to more thinking”.

The most honest answer to the question of “who am I?” is a simple summarisation of one's experience given the context, which explains their current behaviour. The answer is impermanent, imprecise and mostly useful for communication with others. There is nothing deeper to be found there. 

Practice

I believe that looking at your own self-construct as some sort of generative agent is a helpful exercise. Try to notice which role you are playing. Notice how your behaviour arises as a response to the circumstances, often without a goal, simply as a pattern of a social dance. Could you hold your identity lighter, seeing it as a play of social patterns over your embodiment? Self/ego have a bad reputation in Dharma, often through the Sutric “anata” discourse. But having one is not optional (unless you plan to leave the society) and extremely important. Without the self we wouldn’t be able to take part in the society and be of any help to ourselves and others. Our environment is mostly social. Self is the compass that orients our behaviour. 

Self is always messy and full of contradictions, because the world (social and otherwise) is messy and full of contradictions. It is important to like one’s self, to have compassion for it and to care for it, whilst seeing it for the in-essential, impermanent, socially constructed and messy thing that it is. Seeing that object clearly arise and go in your experience is a valuable meditation. 

References:

1. Alexander Sasha Vezhnevets, John P. Agapiou, Avia Aharon, Ron Ziv, Jayd Matyas, Edgar A. Duéñez-Guzmán, William A. Cunningham, Simon Osindero, Danny Karmon, Joel Z.  Generative agent-based modelling with actions grounded in physical, social, or digital space using Concordia Leibo. 2023 https://arxiv.org/abs/2312.03664 https://github.com/google-deepmind/concordia

2. Jonathan Haidt, 2001 The emotional dog and its rational tail: a social intuitionist approach to moral judgement, Psychological review

3. Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351.

4. Brooks, E., Walls, L. A., Lewis, R., & Singh, S. (2022, November). In-context policy iteration. In NeurIPS 2022 Foundation Models for Decision Making Workshop.

5. Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023, October). Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (pp. 1-22).

6. Thomas Parr, Giovanni Pezzulo, Karl J. Friston Active Inference: The Free Energy Principle in Mind, Brain, and Behavior

7. James G. March, Johan P. Olsen The logic of appropriateness 2009