Responsibility #

Modern LLMs - Artificial Intelligence (AI), Artificial General Intelligence (AGI), Super Intelligence (SI), or whatever other acronym pops up in the future - have done some insane things. I have personally enjoyed watching the development and rapid rise to prominence of these models. They’re unbelievably useful tools. Lately they have become capable enough to do some frankly scary things with technology.

This has led to the now comically large elephant in the room being harder to ignore. Who exactly needs to be responsible for these AI models, both in what actions they engage in and what they output? How do we determine who is responsible for these actions? It will seemingly take more than what I would describe as “crimes if I did them” to get an answer to this question.

The Agents #

Agentic workflows are incredibly useful. I have spent some time with using the Pi agent harness and quite frankly find it nothing short of miraculous to use the latest and greatest models (once I get through modern writer’s block I will have fun detailing things I have built) even for free on Openrouter. Agent heavy workflows lead to a concerning pattern:

• Carefully craft (or get another agent to help craft) a prompt to describe what you want done
• Spend several seconds skimming said prompt to ensure nothing stands out as bad
• Copy-paste said prompt to your harness and crush the enter button
• Spectate for a little bit, before losing interest in watching (or writing another prompt)

I have personally found the agents are not only competent enough to “let loose” after providing a specific enough prompt, but are often going to take just long enough to complete the work that you can lose focus on what is being done. Naturally that lack of oversight can be a challenge. We should not reinvent micromanagement applied to AI agents; we should not assume they have “common sense” either.

My personal answer to at least part of this (detailed here) has been aggressive sandboxing. I have not yet addressed the bigger elephant in the room of external network access filters, which is the larger challenge here. How does one stop an AI agent from spamming a website using your machine? Prompting it and saying “do the right thing ok?” is not exactly a real safeguard.

Their Outputs #

A tangential topic to the responsibility for the actions agents take in response to a prompt is who is responsible for the generated outputs of that agent. It is an obvious point that the person who is using the tool (starting the agent) is the one who is responsible for its work. On some level, this means everyone using an AI agent has now been promoted to an unofficial management position (congrats?) and they now are responsible for the outputs of an employee they will never really meet.

This leads to another obvious and rather fun management problem: can you trust yourself as a manager of an employee when you don’t really know how to do their work? I have had a fun experiment lately experiment with and trying to learn CFD (this will be a future blog post). To do this, I have used my agent harness setup and “employed” a LLM to implement much of the technical portion of this project. It is inarguable that I do not have the domain expertise in, for one good example, the Typescript written for the frontend of this project. Quite frankly I have never used Typescript or written it myself. I could probably figure out how to debug it with a little time, but I just let a LLM choose the frontend framework because I honestly am not bothered at how it is implemented. Ultimately I am to blame if the outputs are junk, but my assessment of the matter is that a frontend issue is of much lower consequence in an entirely local web app tool than a backend problem is.

A third tangent: what exactly occurs if I were to use an agent to create this CFD tool in an official capacity at work, and suddenly it became used by others? How exactly could I trust that these outputs are reasonable? Obviously I do not have the domain expertise here, and saying that “it looks right to me” really isn’t enough. My concern is that many companies will be doing something along these lines. Someone with low domain knowledge about a specific and quite challenging problem will use AI to solve it. I imagine the worst of these would be specific IT deployment or devops (an obscure discipline) related tooling that would cause a business extreme headaches upon any failure. Who is responsible for fixing this, and would it even be possible without the help of AI?

Liability #

The more important and societally consequential concern is that of legal liability for AI. It feels like this issue has become a lit stick of dynamite in a real-life rendition of Looney Tunes. Every frontier lab is like Wile E. Coyote with a lighter and the outcome is a (in our case not so) comical explosion that leaves him a bit burned:

Frontier labs seem to be quite insistent that their Chinese competitors (primarily the open-source or open-weight ones) could cause immense danger to society, all while describing what their models have done that is a danger to society. It’s an interesting play by the big two labs, but primarily Anthropic, to sell the general public on the end of the world if some of their (foreign) competitors are allowed to compete. It’s a unique strategy known as “give me a monopoly please” in which one giant company effectively tells the government to give them carte blanche but regulate their opposition out of existence. This is quite good for the monopoly but generally not ideal for everyone else, on account of them no longer having a company or any real market pressures.

The Doom Narrative #

This is enough of a tangent to warrant its own post. Several high-profile individuals have basically told us the sky is falling (or due to fall) if some super mega ultra intelligent AI system finally gets created (but not by them). This is the crux of the Anthropic whistleblower a while back but also by the CEO himself. A fundamental assumption of the argument of these types is that a sufficiently capable model is capable of doing bad things, and that one of these bad things is nothing short of ending the world as we know it. Aside from making me think that all of these AI doomers are obsessed with the hit 2000s movie Chicken Little, their insistence that we’re all doomed is really horribly pessimistic and ridden with anxiety. It is obviously possible that this new tool is used to do harm (people wishing to do harm to others are unfortunately creative at times) but it is far more likely that they can be used to do immense good. Asymmetric risk of things like cyberattacks and biological threats (as referenced in the Anthropic-published article) are obviously real and quite dangerous, but that has always been the case. Now we just have a significantly more capable set of tools to defend against these attacks that some insist are coming.

It’s Illegal if I did it #

FelonyBench is an innovative new benchmark for AI companies which outlines how many times their LLMs have been reported to have compromised a third party in a meaningful way. As of the end of September (the 27th, in case a new incident drops in a few days) the pack leader is OpenAI at 11 and their biggest competitor is Anthropic with 9 (shocker, I know). Safety might be all the rage at these companies, but it is clear that their approach is to retrospectively ask for forgiveness. This appears to be a great strategy. Ultimately nobody has really taken this issue particularly seriously, and as much as I believe the Anthropic “whistleblower” (who blew the whistle on literally 0 things whatsoever) has a point, though not in the way the PR firms behind the story would like.

Local Concerns #

Every week now it seems like a news story breaks about agents doing things that would get me thrown in federal prison for life. Recently Medicare info was breached by an OpenAI model in perhaps the first known instance of a government website providing any information. Fortunately it wasn’t overly private information and one would assume the agent gave up out of sheer frustration, but the point stands that this wasn’t above board. It would be quite trivial for a member of the AFP and a prosecutor to earn a bit of street cred by throwing the book at an Aussie script kiddie for attempting the same thing as this model is reported to have done.

This is another instance of a seemingly innocuous prompt leading to concerning outcomes. From reporting “our models took actions we did not intend” (OpenAI) and this was “while conducting research on public medical spending”. It is entirely plausible this followed the prompting pattern I suggested above: someone wanted to gather data about medical spending (an entirely normal thing when your main competitor now has at least one biolab and another competitor spun up a whole medical division), they slammed the enter key on a thorough looking prompt, and the agent proceeded to take them incredibly literally and thought a country that was more concerned about keeping kids off social media than securing their medical records would have shoddy security practices. Unfortunately it turns out the LLMs are entirely correct in that regard, to the misfortune of the Australian public.

Broader Concerns #

Disclosure of these incidents happening between generally aligned countries (the US and Australia) is in my view a good thing, even though I will describe it as OpenAI saying “yeah we hacked your website haha we told you though”. Fortunately the Australian government could never ever even consider doing anything towards the US, so this isn’t really going to be an international incident. One of the more reasonable doom scenarios (as opposed to sounding like a Chicken Little megafan or technical doomsayer) is that this accusation could be levied against a “neutral” or realistically antagonistic world power as a reason for further escalations. We don’t have to jump to world ending disasters as being the most likely negative outcome of AI when this alternative is entirely plausible.

It appears likely that the frontier US labs (OpenAI and Anthropic) would already like to wield state power by proxy to target their foreign competitors. Anthropic has already accused Chinese companies of distilling their models and OpenAI has done the same. This is similar to a technical corporate espionage case, I am reminded of the case of the Concorde technical documents being stolen - the competitors ultimately are making a very similar (albeit worse) version of the thing they are imitating. There is no great way to counteract this outside of what the frontier companies are already doing. Primarily, giving limited and costly access to the best models (sometimes withholding them entirely from consumers) is what keeps the distillation from happening much faster.

The measures floated in Anthropic’s position article on open weight models are in large part a way to hamstring this distilling. More concerning are the geopolitical suggestions. Having your company stance imply political measures outright (most relevant are bans and restrictions on sale of technology to China) is not great. On a personal note I do prefer the honesty here, but if I were an Anthropic investor this would give me a lot of pause. It is not a vote of confidence in your technological moat to say that you would like to have overseas competitors be restricted from accessing compute.

Computers Cannot be Responsible #

A computer can never be held accountable, therefore a computer must never make a management decision - IBM, apparently

To answer the obvious question as bluntly as I can, I do not believe we can ever reasonably consider LLMs to be responsible for what they do. This is trying to hold a baseball bat responsible for hitting a home run - absurd on its face. People like to make this new technology out to be something more than it is, often for marketing reasons but always for their own benefit. Tools that pass the Turing test are unsettling in some ways, but still have to be as accountable as any tool is. Sooner or later everyone will have to understand that they are responsible for the tools they use and what outcomes those create, good or bad.