Theory of Mind in AI

Theory of mind is the ability to work out what someone else believes, wants, knows and feels, and to act on that picture. Psychologists have studied it for fifty years. It is how a child learns that other people can hold false beliefs, and how an adult judges what a listener already knows before choosing what to say.

AI systems now do something that looks similar. A conversational model infers what its user wants, what they already know, and what mood they are in, and it adjusts its answers accordingly. Memory and personalisation features extend that picture across months. Whether this counts as theory of mind, and what a system does with the model of a person it builds, are open questions.

Before those questions can be answered, a simpler one comes first. When the people who design, train and evaluate AI systems talk about theory of mind, what do they mean by it, and how much of the psychological evidence do they have in view?

A system's model of its user is where cognitive security lives. It decides how persuasive the system can be, how well it can tell when a person is vulnerable, and how easily it could mislead them. Guidelines for that model need a shared vocabulary between psychology and AI, and at present there is little evidence about whether one exists.

If theory of mind means one thing in a psychology department and something looser in an AI lab, evaluations built on the term will measure different things. Knowing where the two communities agree, and where they talk past each other, is the first step towards guidelines that both can use.

Julia Shaw and Caroline Catmur are surveying people who work in AI research, development, ethics and evaluation. The survey is short and anonymous, and asks how familiar the concept is, what people take it to mean, and whether they would value knowing more.

The study has ethical approval from King's College London. Findings will inform a larger piece of work at CogGuide on how AI systems model the people they talk to, and may lead to educational resources on theory of mind for the AI community.

Findings will be published here when the study completes.