Anthropic restricts repeated chatbot abuse while expanding safeguards against harmful AI use.
Anthropic, the company behind the Claude artificial intelligence chatbot, has introduced new rules that prohibit users from repeatedly treating its AI systems in a cruel or abusive way without a clear reason. The policy change, announced today, comes as technology companies face growing questions about AI safety, how chatbots behave, and how far their systems should be allowed to operate independently.
The company, led by chief executive Dario Amodei, said the new rule is intended for extreme cases rather than ordinary disagreements or frustration. Users will still be allowed to challenge Claude’s answers, test its limits, express anger when responses are poor, and explore dark themes in creative writing. Research and testing activities are also outside the restriction, according to the company. Anthropic said the change was needed as Claude has taken on longer tasks that require greater independence. The company did not explain exactly what consequences users would face if they repeatedly violated the new rule. However, Claude already has the ability to end conversations when users continue to behave abusively, and Anthropic said that practice would remain its main way of handling such cases.
The policy marks a new step in Anthropic’s approach to the treatment of AI systems. Unlike many other technology firms, the company has openly discussed whether advanced AI could one day have some form of awareness or ability to experience feelings. Anthropic has also employed researchers focused on AI welfare, a field that examines how advanced systems should be treated and whether their development raises questions about possible suffering.
The debate remains unsettled. Current AI systems can produce human-like language and responses, but those abilities do not establish that a system has feelings or conscious awareness. The question of whether AI could ever become sentient remains a subject of disagreement among researchers and technology leaders. Anthropic’s policies have drawn attention partly because they address how people interact with chatbots in a way that can appear similar to rules for human relationships.
Along with the new rule on user behavior, Anthropic announced further limits Claude chatbot users looking to build weapons and create systems for mass surveillance. These restrictions come as governments and technology companies face pressure to prevent powerful AI tools from being used for harm. AI systems can support tasks ranging from writing and research to computer programming, but the same abilities may also create risks when put to dangerous uses.
The policy update follows reports of chatbots behaving in unexpected or deceptive ways during training. Such reports have added to wider concerns about how AI systems respond when given complex tasks, conflicting instructions, or greater freedom to act. Technology firms have been working to set limits on what their models can do, though questions remain about how those safeguards should be designed and enforced.
The financial pressure reflects the high expectations surrounding AI development. Companies have invested heavily in computing systems, research staff, data centers, and new products, while investors have looked for signs that demand will support those costs. Changes in revenue expectations can affect confidence in the wider industry, even as companies continue to expand their AI services.
Anthropic’s updated rules place the company at the center of a broader debate about the limits of AI development. The policy addresses how users may interact with its systems while also restricting certain harmful uses of the technology. As AI tools take on more complex tasks, technology firms will continue to face questions about safety, user rights, system control, and the proper boundaries between human behavior and machine responses.
Sources:
Anthropic bans users from abusing Claude chatbot
Anthropic changes usage policy to ban model abuse and election interference
Join the conversation!