“I'm trying to spend less this week.” In this small thought experiment, someone has asked an AI assistant for help with dinner. They want a cheaper meal, perhaps something made from what's already in the cupboard. They have not applied to become the sort of person who is always trying to spend less.
That last distinction seems worth remembering.
Suppose the assistant saves a tidy conclusion: prefers budget options. Next month it steers a birthday dinner toward the cheapest restaurants. Nothing has leaked to an advertiser. No stranger has read the conversation. The assistant is simply being considerate toward someone who no longer quite exists, or who never existed in that form to begin with.
The mistake happened when “this week” disappeared.
The appeal of being remembered
There is a good reason to want an assistant that remembers. Repeating a dietary preference every time you ask for a recipe is clerical work disguised as conversation. A useful assistant ought to let you get on with asking about dinner.
OpenAI's Memory FAQ describes a system that, when enabled, draws useful context from chats, files, and connected apps to personalize later responses.[1] That's the promise: less repetition, more continuity. The dinner scenario above is invented, not a report of a failure I observed in ChatGPT.
I don't think the answer is to make every conversation begin with ritual amnesia. For someone who relies on an external aid to keep track of things, continuity can matter much more than the convenience of skipping a sentence.
Andy Clark and David Chalmers gave that intuition a demanding form in their 1998 paper The Extended Mind.[2] Their thought experiment follows Otto, who has Alzheimer's disease and uses a notebook to retrieve information; they argue that the notebook can play a role comparable to biological memory in guiding his actions.[2] You needn't accept their whole account of the mind to take the reliance seriously.
A notebook that helpfully erased itself each morning would be a terrible notebook.
The complication with an AI assistant is that we may ask it to do something a notebook doesn't do on its own: decide what our words reveal about us. Recording “budget dinner this week” and inferring “budget-conscious person” are different jobs. The second gives the system considerably more editorial freedom.
True, but out of place
Privacy scholar Helen Nissenbaum's account of contextual integrity ties appropriate information gathering and sharing to the norms of particular contexts.[3] In that view, privacy involves more than keeping facts secret. Where information belongs and how it moves matter too.[3]
I think a related test belongs inside personalization, even when information never leaves the assistant. Was this detail offered for this task, or for any future task where the system can find a use for it?
Take another invented case. You're helping a friend find wheelchair-accessible venues. Remembering the accessibility requirement while you plan that outing is useful. Turning it into an unqualified fact about your mobility would be a bad inference. Even preserving it accurately as a friend's requirement doesn't make it relevant to every trip you plan afterward.
A fact can be correct and still be in the wrong conversation.
This creates an uncomfortable tradeoff. If an assistant asks permission for every possible inference, dinner becomes an interview about dinner. Automatic personalization saves effort precisely because the person doesn't have to supervise every small decision. “Just ask” sounds better before the twentieth question.
But silence isn't a good general answer either. A mistaken profile can shape which options appear in the first place. In the birthday example, you might never see the restaurant you'd have chosen, so there is nothing conspicuous to correct. The suggestions just seem a little dull. Your celebration has been reviewed by Tuesday's grocery budget.
Leave room for a different answer
My preference would be for memory to keep the scope of a statement, not merely its subject. An explicit “remember this about me” should carry more weight than a passing constraint. A one-off request can guide a task without becoming an enduring description of the person. Those are design preferences, not a claim that present systems reliably make these distinctions.
Visibility helps, although a summary isn't necessarily a complete inventory. OpenAI's FAQ says its memory summary will not include everything remembered from chats, and its displayed sources may not show every factor that shaped a response.[1] It also says that fully deleting something may require removing it from every source where it appears, rather than only editing the summary.[1]
That makes the humble ability to ask “why did you assume that?” important. I'd want a useful answer: which statement, from which situation, led to this recommendation? And I'd want “that was only for that trip” to correct the scope without forcing me to erase a whole useful conversation.
Good personalization should make it easier to be understood, including when you've changed your mind. I would rather an assistant remember that a preference was provisional than remember it very confidently forever.
You asked for a cheaper dinner. You should still be allowed to celebrate on Friday.