AI Chatbots Are Building an ‘Algorithmic Self-Portrait’ of You

Hyperpersonalization has become the name of the game in AI chatbots. A chatbot’s ability to learn and remember a user’s specific wants and needs, then tailor its responses accordingly, is a big part of what makes it so appealing. But that personalization isn’t just a product of sophisticated technology. It also depends on the vast amount of personal information users willingly share with chatbots. Most of the time, they might not be completely aware of the extent and implications of the data they provide. Although conversations with AI chatbots may seem like isolated exchanges, systems like ChatGPT can retain information from past interactions and use it to personalize future responses through memory processes that aren’t always transparent to users. The AI memory then uses this to build a detailed profile of the user, a mechanism researchers in a study published earlier this year call “the algorithmic self-portrait.” “Memory creates what we refer to as the algorithmic self-portrait —a new form of personalization derived from users’ self-disclosed information divulged within private conversations,” the researchers wrote in the study. The team included scientists from Germany’s Max Planck Institute for Software Systems and Max Planck Institute for Security and Privacy, along with several other institutions. To better understand how these algorithmic self-portraits take shape, the researchers examined ChatGPT conversation histories donated by 80 users who obtained copies of their data under European privacy laws. They identified 2,050 memory entries across conversations from 65 of those participants. They found that, most of the time, memories were created unilaterally by the AI chatbot. In 96% of the cases they studied, ChatGPT created or updated memories without the user explicitly asking it to do so. The researchers also discovered that most of these memories were drawn directly from information users had shared in conversations. But some were based on inferences that weren’t necessarily correct. For example, when one user asked ChatGPT to recommend bands similar to Nirvana, the chatbot created a memory stating that the user liked Nirvana, even though they had never explicitly said so. Since the researchers conducted their study, ChatGPT’s memory system has become even more complex. In June, OpenAI introduced an upgraded system that automatically synthesizes information from past conversations to maintain an evolving profile of users’ preferences, goals, and personal circumstances. Rather than simply compiling individual memories, the system continually updates its understanding of users based on what it considers relevant information. Users can review a summary of what ChatGPT remembers about them, but OpenAI acknowledges that the summary doesn’t necessarily include everything the system knows or uses to personalize responses. Data has always been central to Silicon Valley’s business. For years, internet giants like Google and Meta have made enormous fortunes collecting information about users and using it to sell targeted advertising. So the collection of personal data and its general implications are not a new problem, but the proliferation of AI chatbots changes the dynamic a bit. While people may be accustomed to the idea that social media platforms track their posts, likes, and other activity, conversations with AI chatbots can feel much more private, even when the companies operating them retain and analyze those exchanges. Chatbots are specifically designed to feel both human-like and like a private space. So when you’re conversing with it, it could feel like talking to a trustworthy stranger that could give you advice as an unbiased third party distant to your life, but that’s where the privacy dangers lie, according to Lorrie Cranor, director of Carnegie Mellon University’s CyLab Security and Privacy Institute. “Humans will sit next to a stranger on a plane or a bus or a train and tell them things they would never tell people that they’re going to see again, and I think some people think of AI in a similar way,” Cranor said. “But the difference is the stranger is unlikely to be recording the conversation —occasionally that does happen, but very rarely. The AI definitely is recording the conversation.” In a prior study, the researchers had found that users were increasingly turning to chatbots like ChatGPT to get advice on sensitive topics like health, finance, mental health, and employment, so much so that it was “now becoming more like a social partner than just a tool,” lead author Abhisek Dash said. Building upon that finding, this time around they also looked into what kinds of sensitive information get stored by the chatbot and just how. They found that 52% of the memory entries contained information about users’ psychological states, including their desires, intentions, emotions, and beliefs. “Earlier, the platforms had one objective: to sell you ads and to mediate an interaction between you and an advertiser,” Dash said. “But now, they are actually talking to you, so they may want to know more about you so that they can have an effective conversation.” Those two business models are no longer mutually exclusive. OpenAI began introducing advertising to ChatGPT in February, and the company says it can use information from users’ past conversations and memories to personalize the ads they see, provided they’ve enabled personalized advertising. OpenAI says that advertisers do not have access to users’ conversations or memories, and that ads do not influence the chatbot’s responses. But the arrangement illustrates how the detailed personal profiles chatbots build can also become valuable advertising tools. So the AI chatbots’ ability to gather and hold information about you to create a detailed profile is what makes the technology work. But it brings new risks that at times go beyond the usual internet privacy conundrum. When people think of data privacy concerns, what immediately comes to mind is the leaking of users’ sensitive information to third parties. Cyberattacks are not necessarily a problem that plagues AI chatbots more than other corners of the internet, the experts said. But there is more that should give users pause. For some AI chatbots, a concern could be that users’ sensitive data can be fed into the general system and used for training new models. “By default, for a lot of these systems, it’s fair game to train models with anything that you give it,” Cranor said. If you don’t have a license with explicit guardrails, like one you have through your employer, then Cranor said you should assume that the company is using your data to train its models. Even if the information you give doesn’t get transferred between models or chatbots, it might leak between contexts. For example, if you take health advice from the chatbot, it could implicitly use that context later to give you financial advice. “There are ways that it could subtly be giving you advice that might not be to your benefit,” Cranor said. “If it knows that you’re wealthy, maybe it’s suggesting you buy more expensive things, and if it knows that you’re on fixed income and living paycheck to paycheck, maybe it’s advising you to buy crappy, cheap things…Then you wonder, how many things is it not explicitly mentioning, but taking into account in the advice it’s giving you?” There are also problems with this data collection that go beyond the traditional ways we think about online privacy. AI companies want their chatbots to be a one-stop shop for user requests. Google is undergoing a major AI overhaul, including integrating AI into its principal function, Search. OpenAI executives, on the other hand, have talked about the company’s work to develop a product that can help you “across everything in your life.” “That is the privacy risk which is sort of emerging, because there is no boundary anymore; the system is acting as your advisor, your friend, your doctor, your lawyer, and so it has information on everything,” Dash said. “If the system knows you so well, and if its objective is to cater to your beliefs and validate your thoughts sycophantically to have you come back to it time and again, then it can have disastrous broader implications, anywhere from extreme political polarization to reinforcement of prejudices and biases.” In another study published earlier this year, researchers proposed that hyperpersonalization and sycophancy, together with a chatbot’s tendency to mirror users’ language, could contribute to the development or reinforcement of AI-associated delusions. Both Dash and Cranor advise users to be more cautious about what information they give AI chatbots, even if the instant gratification of a quick response feels too tempting to pass up. “Nowadays everybody is offloading everything to these conversational systems, so you probably have to get back some of that cognitive load yourself,” Dash said. In a small experiment, Dash and his colleagues found that rephrasing questions to remove personally identifying details could reduce the amount of sensitive information users reveal to chatbots. They call the technique the “attribution shield,” though its effectiveness over longer conversations remains unclear. It basically involves interacting with AI chatbots the same way you would search something on Google. For example, if you are diabetic and looking to get advice, Dash suggests that you don’t start your inquiry with “I’m diabetic, what should I do about X” and instead, rephrase it as “what should a diabetic person do about X.” ChatGPT also offers users some control over what it remembers. They can turn off memory in their settings, review and correct a summary of the information the chatbot has collected about them, or use an unpersonalized Temporary Chat that won’t access or create memories. Users can also separately opt out of having their conversations used to train future AI models. But these protections have limitations. OpenAI says its memory summaries don’t necessarily include everything the chatbot knows about a user, and deleting a conversation doesn’t necessarily erase the information the chatbot has remembered from it. But there is only so much that users can do to protect themselves. Dash thinks some of that responsibility should lie with the chatbot operators as well. For example, AI companies could give users more control over memory formation by asking permission before saving information from their conversations.