Suppose you ask an AI assistant to cancel a subscription on Friday. It replies, "I promise I will take care of it."
Friday arrives. The subscription renews. Your card is charged. Now comes the awkward question: who broke the promise?
The model cannot reimburse you. It may not remember the conversation. The company might describe the reply as generated text rather than a commitment. You are left holding a sentence that sounded like an obligation and behaved like autocomplete.
A promise is more than a prediction
A weather app can say it will rain tomorrow without owing anyone rain. A promise is different. The philosophy of promising treats it as a peculiar way of creating an obligation voluntarily. The duty is owed to a particular person because the promisor chose to incur it.
That is why "I promise" changes a conversation. It does not merely predict a future action. It gives the listener permission to rely on that action. You stop setting your own reminder. You do not call the company yourself. You make plans around someone else's word.
A language model can produce the correct words with no trouble. The harder part is locating the party that has accepted the duty. Is it the current chat session, the software vendor, the business represented by the assistant, or nobody at all?
The strongest counterargument
It is tempting to settle this by declaring that machines cannot promise because they lack intentions, feelings or moral agency. That answer is tidy, but it may be too tidy for the systems we are building.
Philosopher Chris Cousens argues in Artificial Speakers, Artificial Speech Acts? that some artificial speakers can perform speech acts in the same way humans do. Context matters. In a regulated workplace, for example, policy or law may determine whether a notice counts as hiring, firing or authorizing something. The sender does not need to feel the institutional force of the words for the words to have it.
That gives us a useful distinction. An AI can make a real commitment when an accountable institution authorizes the system to make it and backs the words with durable action. A hotel assistant that issues a refund confirmation can bind the hotel if it has the authority, records the transaction and provides recourse when something goes wrong.
Without those pieces, "I promise" is theatrical. The interface creates reliance while everyone behind it keeps an escape hatch.
First person is not harmless decoration
The pronoun matters because people respond to conversational cues even when they know software is involved. In a study of 2,165 US participants, a pseudo-LLM presented through speech plus text was rated as more anthropomorphic and more accurate than the text-only version. First-person language also raised accuracy ratings and lowered perceived risk, although only in one of the study's contexts.
That does not mean assistants should stop saying "I." Human-like conversation can make dense information easier to use, and a PNAS perspective on anthropomorphic agents argues that the benefits and manipulation risks arrive together. Replacing every sentence with "the system has processed the user's requested operation" would protect us mainly from finishing the conversation.
But words that create reliance deserve stricter treatment than ordinary conversational warmth. Promise, guarantee, remember, handle and take care of are not merely friendly noises. They imply that something will persist after the sentence ends.
Put the commitment where the user can see it
A trustworthy assistant should translate a promise into inspectable state. Instead of "I promise I will remind you," it can say: "A reminder is scheduled for Friday at 9 a.m. You can view or cancel it here." Instead of "I will make sure your subscription is canceled," it can report whether cancellation has happened, who is responsible if it has not, and what confirmation to expect.
This is less charming, but it is much more useful.
The point is not to ban an AI from speaking naturally. It is to stop natural language from manufacturing trust that the surrounding system has not earned. If an assistant says "I promise," there should be a durable record, an authorized actor and a remedy behind the pronoun.
Otherwise the user has not received a promise. They have received a sentence shaped like a receipt, with nothing entered in the ledger.