I’ve always been inspired by the trolley dilemma. How do you decide who to save in a morally ambiguous situation? What happens when the one person you have to sacrifice in order to save five, is actually someone you love? Who are the kinds of people that still think through this kind of a situation rationally, who still hold it as one against five, no matter who that one ends up being?
I went into a PhD to answer questions like these. Using both neuroimaging techniques and behavioral experimentation, I studied reward, empathy, and moral cognition, and how these processes play out differently across people. I now apply my findings to AI, leveraging decades of research on the successes and failures of human intelligence to build safe, aligned, and ethical systems.
Empathy is a big part of how I think about AI safety. When we talk about empathy in humans, it is often split into two parts. One is the ability to read someone, to predict what they are thinking or feeling. That’s called cognitive empathy. The other, affective empathy, is the ability to share feelings with them, for their distress to be mirrored in your own body as something like your own. Researchers have been studying these as two systems for decades, identifying separable neural patterns and behaviors that seem to characterize each subtype.
My master’s work looked deep into the neural basis of empathy, with a specific focus on affective empathy. I was interested in the sharing of others’ states, on how we mirror the feelings of others and map them onto our own bodies. We tested that by looking at how well brain patterns can distinguish sensory states, how much we can separate out patterns related to seeing someone being hurt compared to someone being touched. Those with more distinct brain patterns for the two sensory states showed higher self-reported empathy. Our findings made a case for the simulationist theory of empathy, a theory claiming that our ability to empathize with others relies on our ability to accurately simulate their states and therefore also robustly experience our own internal states.
While I was working on that, a paper was being written and finalized in my lab about artificial empathy and what happens when you build only one of the two parts in a system. The argument is that most work in this space builds out cognitive empathy, the predictive side, modeling what a user feels and producing a calibrated response, and seems to be entirely missing the ability to share states. A system built this way can end up behaving a lot like a sociopath. This research motivated me to think about why I was interested in studying empathy in the first place. If empathy can be separated apart, and one piece can be present while the other is missing, why not study how this dissociation manifests in humans?
This led me to the Dark Triad of psychopathy, narcissism, and Machiavellianism, studying how these traits are characterized across the brain and behavior. I was not necessarily interested in the collection of these three specific traits, but rather as a framework for understanding how this dissociation of empathy can manifest. The subtitle of my dissertation was “The Interplay of Affective Dysfunction and Utility Maximization”, which is a good illustration of what I was after. I wanted to understand how empathic dissociations paired with a hypersensitivity to reward can lead to antisocial behavior. Dark personalities are where you can watch reward, morality, and empathy interact with each other, and what shows up is a person who reads other people accurately and shares very little of what they read. This often facilitates manipulation, where you can predict what someone might feel, but not really care if you harm them for your own reward or utility.
As language models improved, I began seeing the same dissociation manifest. These models are trained on close to everything we have ever written down, which makes them exceptionally good at modeling humans, and there is nothing in that process that gives them a way of feeling what we feel. This seemed to mirror what I was studying with the Dark Triad, and I wanted to see how easily we could push models into the same antisocial patterns I observed across human populations. Some of that work is published here and here. I found that these dark personas can be induced in frontier models with small perturbations, i.e. very little data, and the resulting personas strongly replicate the behaviors we see in humans. I then looked inside the model to see where these antisocial behaviors might live, finding individual features that seem to drive exploitation, manipulation, and deception separately. This suggests the same empathic dissociation that underlies the Dark Triad can live within an artificial system, maybe not on the surface, but in a way that can be easily induced and identified internally.
While you can show this dissociation in a language model, the fix may lie in systems that have a body to sense. The theories I was working from are grounded in the body. Affective empathy relies on interoception, the sensing of our own internal signals, and on homeostasis, the continuous regulation that keeps those signals in a livable range. Sharing someone else’s pain means simulating it with the same systems that represent your own body, which requires having internal states that can be sensed in the first place. A system made of text may have internal states, but little to nothing that works like a body signaling whether things are going well or badly for it.
This brought me back to the framework from my lab about avoiding robotic sociopaths. Their framework introduces vulnerability as a requirement to building affective empathy, the ability to share states with others. What they propose is an agent that maintains its own internal balance, analogous to a human’s homeostasis, builds predictive models of the hidden states of other agents, and maps those states onto its own in order to determine any action taken. Prosocial behavior then emerges as a result of understanding vulnerability and risk-to-self. My own findings add to this, since if empathy depends on simulating others with your own states, those states need to be distinct enough to map onto in the first place.
All of this becomes more real with embodied AI, systems that are being built for deployment in homes, schools, and care-giving environments. As Co-Founder and Chief Research Officer of the Institute for Humane Robotics, I plan to implement frameworks like the one described above within embodied and physical AI systems. Our work focuses on making these ideas testable, bringing other such frameworks from academia into industry and seeing how they work in a real-life robot deployed in human environments. We are also translating our findings into design guidelines for those building and deploying robots, working towards a future of both physical and psychosocial safety.
A point worth noting: we are intentionally trying to build systems that surpass us, that amplify us, that don’t replicate our vulnerabilities. So when we talk about building internal states or vulnerability into artificial systems, which vulnerabilities do we actually mean? Humans are fragile in ways worth leaving behind. We feel pain, we damage easily, and we die from very little. Same goes for the failure modes we have spent decades documenting in ourselves, such as the Dark Triad. But some ability to feel both harm and benefit seems necessary, as a system that can be affected by nothing has little reason to care what happens to anyone.
So the question we are actually working on is one of calibration. How much of our own vulnerability has to be replicated to get a system that is prosocial in the way we are, and how little can we get away with, so that what we build doesn’t replicate our worst failures. I don’t think anyone knows the answer yet. But it seems like the right question to be asking while these systems are still being designed.
“We use “sociopathy” to describe a tendency for antisocial behavior resulting from an impairment of affective empathy… Such an AI could effectively be considered sociopathic: It knows how to predict and manipulate the emotions of others without any empathic motivation of its own to constrain its behavior and to avoid harm and suffering in others. This potentially poses a civilization-level risk.”
(Christov-Moore et al., 2023)
Brilliant piece, Roshni. Looking forward to more from you!
This is one of the best pieces I have read in quite some time! Thanks, it is really insightful, especially as a robotics engineer.