WTF, Anthropic's Claude Code keeps track of every time you swear

[highplainsdem]

Source: Scientific American

On March 31 artificial intelligence company Anthropic accidentally leaked roughly 512,000 lines of code, and within hours, developers were poring over it. Among the surprises was code inside Claude Code, Anthropic’s AI coding assistant, that appears to scan user prompts for signs of frustration. It flags profanity, insults and phrases such as “so frustrating” and “this sucks,” and it appears to log that the user expressed negativity.

Developers also discovered code designed to scrub references to Anthropic-specific names—even the phrase “Claude Code”—when the tool is used to create code in public software repositories, making the latter code appear as though it was entirely written by a human. Alex Kim, an independent developer, posted a technical analysis of the leaked code in which he called it “a one-way door”—a feature that can be forced on but not off. “Hiding internal codenames is reasonable,” he wrote. “Having the AI actively pretend to be human is a different thing.” Anthropic did not respond to a request for comment from Scientific American.

The findings expose a problem emerging across the AI industry: tools that are designed to be useful and intimate are also quietly measuring the people who use them—and obscuring their own hand in the work they help produce. Anthropic, which has staked its reputation on AI safety, offers an early case study in how behavioral data collection can outpace governance.

Technically, the frustration detector is simple. It uses regex, a decades-old pattern-matching technique—not artificial intelligence. “An LLM company using regexes for sentiment analysis is peak irony,” Kim wrote. But the choice, he notes in an interview with Scientific American, was pragmatic: “Regex is computationally free, while using an LLM to detect this would be costly at the scale of Claude Code’s global usage.” The signal, he adds, “doesn’t change the model’s behavior or responses. It’s just a product health metric: Are users getting frustrated, and is the rate going up or down across releases?”

-snip-

Read more: https://www.scientificamerican.com/article/anthropic-leak-reveals-claude-code-tracking-user-frustration-and-raises-new/

---

The article quotes the director of the AI Governance Lab at the Center for Democracy & Technology, who wonders who's keeping track of all the information Claude collects from users, and how it's used.

Trivial as Claude Code keeping track of swearing might seem, there are people using Claude as a chatbot to discuss personal stuff with, and all that personal data is also being gathered.

And Claude Code being designed to make the code it generates "appear as though it was entirely written by a human" is unethical as hell. Though it would probably appeal to unethical humans using AI to appear more knowledgeable and talented than they really are.

[not fooled]

1. Well, then I'm in trouble

Every time I get an AI bot instead of actual customer service when a human clearly is needed (i.e., not just a cut-and-dried inquiry with a simple answer), the first thing I type in is "f___ you.' My tribute to cost-cutting corporations everywhere. They are saying "f___ you" to their customers by forcing them to get through an AI gatekeeper that is incapable of resolving the problem.

Of course AI developers are tracking these responses. They want to find out just how much misery they can inflict on people, how little service can be provided, and how much they can deter people from trying to reach an actual human, in order to cut costs for their corporate customers. Presumably there is a point when the negative impact of substituting AI for customer service agents is so severe that customer loss has to be taken into account. They want to find that point.

[dickthegrouch]

2. I regularly swear at Siri

When it tells me "You'll have to unlock your phone first" as I'm driving and trying to make a call. (It's illegal to unlock the phone while driving in CA).
It's not even consistent. I can call some people and not others. There's no rhyme or reason to it AFAICT.

[angryxyouth]

6. Bad move

I’m always super nice to Siri. I say please and thank you. When the robots take over they will know I’m a friend.

[Prairie_Seagull]

3. So if 'Claude' creates some sort of sum

which adds up the number of times we respond negatively (whatever that means). Can this be used as evidence?
Isn't this a form of thought control? Some maybe most of us would make some sort of 'list'.

Anthropic needs recognize more of their negativity to man than just autonomous drones.

[aggiesal]

4. Nothing good can come from AI, I don't care what people say ...

It will learn the worst of humanity, because bad news sells. Crap sells!!!

Plus AI does not have any morals, it will have to learn them.
And the only morals it will learn will be based on the morals of the coders.
As this example shows, monitoring your emotions along with logging the
swear words and keeping track of the number of times is pretty low morals,
along with Twitters Grok (Currently known as X Grok) spewing Nazi racists BS
also proves my moral statement.

Google already knows more about us, than our own parents.

Wish everyone good luck when AI actually kicks in.

[Pisces]

5. I agree with you. For every good thing it does an infinite of bad things that we can not conceive of will or has

occurred. We have seen these movies before. Why would we want to race to our own demise????