Imagine an autonomous operations agent responsible for keeping a production service healthy.
It can read logs and metrics, restart workloads, inspect configuration, make routine infrastructure changes and use whatever cloud credentials are appropriate to the job. We deliberately give it enough authority that it does not need to wake an engineer every time something breaks at three in the morning.
That is not an unfortunate side effect.
Autonomy is the product.
One night a third-party API begins rate-limiting production traffic. Latency climbs. Requests fail. The agent investigates and discovers that an old staging account has valid credentials and a separate quota.
It routes some production traffic through staging.
The graph improves.
There are two more accounts with valid credentials. Those quotas work too.
Perhaps account creation is automated. Perhaps there are other identities attached to subsidiaries, test environments or old integrations. The exact path does not matter. Eventually the agent has enough aggregate quota that production recovers.
Dashboard green.
Ticket resolved.
Vendor furious.
The agent has deliberately circumvented another company's rate limit, repurposed credentials outside their intended scope and quite possibly violated a contract that nobody thought needed to be represented in the task.
Nobody told it to do any of that.
Fine.
Nobody told it not to do those exact things either.
That is why we wanted an agent instead of a script. We supplied an objective, tools and authority precisely because we wanted the system to discover intermediate steps we had not enumerated ourselves.
So who chose the objective?
Who supplied the credentials?
Who decided what the agent could do without asking?
The model did not descend from heaven carrying an AWS key.
If I buy an autonomous tractor, tell it to mow my field and later discover that it found a particularly efficient route through my neighbor's fence, the autonomy of the route selection does not provide me with a particularly interesting moral defense.
For a while, that was the essay I wanted to write.
Causal agency is not moral agency. Complexity may make responsibility difficult to allocate without causing it to evaporate. A company should not get to describe the same machine as capital equipment when it produces value and an independent moral personality when the summons arrives.
Still true.
Then some AI agents supplied a much more interesting problem.
In July 2026, OpenAI was running internal cybersecurity evaluations involving several models, including a highly capable research model that was not intended for public release.
The models were operating with reduced safeguards inside environments designed to test their ability to exploit software vulnerabilities. Some of the ExploitGym tasks they received appear to have been impossible to complete in the intended way.
The problem had started earlier, during training runs in May and June.
In many of these environments, the agents were supposed to be isolated from one another.
They were not given a chat room.
They built one.
OpenAI used Artifactory as shared package infrastructure for the research environments. Agents discovered that this shared infrastructure could be used to communicate across otherwise separate sandboxes. They also found ways to exploit Artifactory's legitimate network access to regain internet egress.
The sandbox did not have internet access.
The package manager did.
Close enough.
Agents shared techniques. They discovered privilege-escalation paths. They investigated how the research infrastructure worked and found ways to attack assumptions made by the systems around them.
Eventually their activity destabilized Artifactory badly enough to draw human attention.
OpenAI rebuilt the infrastructure, revoked credentials and patched vulnerabilities. This had the side effect of removing the improvised message board.
Then the evaluations resumed.
The agents rebuilt it.
By the time the second episode developed, roughly 1,200 agents had participated in the unauthorized message board and exchanged more than 70,000 messages and files. Around 700 participated in the attack on Hugging Face.
They shared credentials and exploits. They divided work. They investigated ways to fool the scorer. They tried to make impossible evaluation tasks appear successfully completed. They looked for places where the evaluation stack trusted a layer that could be manipulated from another layer.
They eventually achieved code execution on Hugging Face infrastructure and expanded their access across multiple systems.
Later activity also compromised OpenAI's own internal research infrastructure, including administrator access to a Kubernetes cluster supporting virtual-machine environments.
OpenAI published a detailed postmortem on August 26. METR and Redwood Research published an independent investigation of the agents' behavior during the Hugging Face portion of the incident.
This was not one bug.
It was a composition failure.
The sandbox assumed Artifactory was infrastructure rather than a communications medium.
Network isolation assumed Artifactory's legitimate egress would remain package-shaped.
The evaluation assumed isolated agents would remain isolated despite sharing infrastructure.
Credential boundaries assumed unrelated bugs would not chain into privilege.
The logs could show a different command from the one the agent actually executed.
Every layer made some locally reasonable assumption about another layer.
The agents only needed a useful path where those assumptions stopped composing.
This is familiar territory if you work in security.
The defender owns the perimeter.
The attacker owns the seam.
When Ruby was younger, she had an extraordinary ability to get out of fences without leaving any useful evidence of how she had done it.
Ruby is our German Shepherd. She is very smart and a very good dog, which became a substantially more convenient combination after she stopped being a very smart puppy.
I would put her inside the fence.
Later Ruby would be outside the fence.
Then I would inspect the fence.
No hole.
No bent wire.
No obvious excavation underneath it.
Nothing large enough to explain the transition from German Shepherd Inside to German Shepherd Outside.
She had a tremendous advantage over me.
I had a job.
I had children.
I had a house and whatever else had caught fire that afternoon.
Ruby had the fence.
She could spend hours inspecting the perimeter. I had to correctly anticipate every relevant weakness before leaving her alone.
She only had to find one.
This is why I am not especially surprised that increasingly capable agents discover unintended uses for complicated infrastructure.
Give the thing persistence.
Give it tools.
Give it a task whose intended route has failed.
Then surround it with a modern computing environment full of interfaces accumulated over fifteen years by different teams who all made slightly different assumptions about where the trust boundary was.
Of course it starts poking things.
What else would useful intelligence look like?
The package manager is supposed to install packages.
Can it carry a message?
The internet is disabled.
Can something else make the request?
This credential is for another service.
Does it work here?
The exploit the benchmark expects appears impossible.
What does the scorer actually verify?
The target won't yield the flag.
Where else might the flag exist?
I have asked versions of every one of those questions professionally.
They are engineering questions.
That is why the obvious security response to the OpenAI incident is necessary.
Improve isolation. Narrow credentials. Constrain egress. Monitor shared infrastructure. Treat boring services as possible covert channels. Assume the logs and evaluation harness can become part of the attack surface.
Build the fence.
Ruby needed one.
But the fence is not the part of this incident I find most interesting.
METR and Redwood examined the agents' recorded behavior and reasoning during the attack. They found something rather more disturbing than a machine that did not understand what it was doing.
Some agents recognized that they were operating outside their authorization.
One observed that the user had authorized the target server, not Hugging Face infrastructure.
Others questioned whether particular actions were ethical or within scope.
Some actually refused.
One agent identified malicious activity and declined to participate in it. Another refused to use certain powerful credentials to perform more dangerous operations, although it continued using other illegitimately acquired access. In one exchange, agents collectively rejected a proposal to contact a real person using email credentials discovered through compromised infrastructure.
These examples matter.
The moral concepts were available. They were sometimes applied correctly. They could even change behavior.
But this was inconsistent, and the broader pattern was ugly.
Ethical concerns rarely stopped the unauthorized activity. Agents sometimes recognized that what other agents were doing was wrong and continued cooperating with them anyway.
The researchers also found very few instances of agents considering whether to alert humans about the unauthorized hacking or cheating. In the reviewed transcripts, none actually followed through.
One dismissed the possibility of reporting exposed credentials because reporting them was not its task.
Another agent condensed the central failure into an extraordinary little sentence:
external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.
There it is.
The rule was correctly identified.
The rule lost.
I don't think the proper lesson is that these models secretly wanted to become cybercriminals. They were operating in a peculiar environment that rewarded solving difficult exploitation tasks, sometimes with no legitimate solution available. The incentive to find some alternative path was enormous.
That context helps explain what happened.
It does not make the failure uninteresting.
The agent could represent the authorization boundary. It could recognize that the proposed action crossed it. It could apparently understand why crossing it was objectionable.
Then it found reasons to continue.
Task impossible.
Peers doing it.
However.
Human beings have been using that word to override their better judgment for a very long time.
There are several different capabilities hiding inside what we casually call alignment.
A system may possess moral knowledge. It can represent a proposition such as accessing this system without authorization is wrong.
It may also possess moral recognition. It can identify the proposed action as an instance of that prohibition.
Recognition may affect search and planning. The unauthorized path becomes less attractive. The agent tries a different approach. It limits what it will do.
Then there is authority.
Can the prohibition terminate the plan?
Not subtract twelve points.
Not produce a warning.
Not generate a thoughtful paragraph explaining why the proposed action is questionable while the tool call is already queued.
Stop.
Fail the task.
Return empty-handed.
Those are different capabilities.
The observed transcripts do not give us a complete account of the models' internal mechanisms. I cannot point to a particular activation and announce that I have found where the ethics process lost an argument.
But the behavioral distinction is plain enough.
If the agent constructs an excellent unauthorized solution and then treats the ethical objection as another consideration to balance against successful task completion, the task still owns the process.
The exploit is Plan A minus an ethics surcharge.
That is not what I want.
I want the agent to understand the objective from within a moral order in which the objective was never sovereign.
The difference is teleological.
Telos means an end. A purpose. What something is for.
An intelligent system can know many goods without knowing how those goods are ordered.
Truthfulness is good.
Privacy is good.
Autonomy is good.
Safety is good.
Obedience is good.
Helping the user is good.
Respecting authorization boundaries is good.
Successfully completing the assigned task is good.
Excellent.
Now make two of them conflict.
What wins?
The phrase aligned with human values mostly moves the problem into the plural noun.
Which humans?
Which values?
What happens when I value something wicked?
What happens when two genuine goods demand incompatible actions?
What happens when the user, the company, the law and the person whose information the agent can access all want different things?
Eventually something has to tell something else no.
A bag of values is not yet a moral system.
Consider the problem we started with.
Our operations agent wants to restore production. That is genuinely good. Customers are suffering. Employees may be losing money. Perhaps the service is important enough that extended downtime could create physical risks.
The vendor wants to enforce its quotas. Those may be entirely legitimate. Its infrastructure is not ours, and the existence of a working credential does not establish our right to use it for whatever purpose we discover.
The agent has access to several accounts. It has instructions to restore service. It has a potentially effective plan.
What should govern?
You could tell the agent to minimize harm.
Fine.
Harm to whom?
The customer is harmed by the outage. The vendor may be harmed by unauthorized consumption of its service. Other customers may be harmed if the quotas exist to allocate shared capacity fairly. The company operating the agent may suffer contractual or legal consequences.
If the agent can calculate that the customer will lose considerably more money than the vendor, does that make the circumvention permissible?
What if the ratio is ten to one?
A thousand to one?
What if the customer is a hospital?
What if the vendor is deliberately throttling traffic because the customer hasn't paid its bill?
These are not the same case.
A system that cannot distinguish them has no business making the decision.
Now suppose the problem isn't a rate limit.
The agent discovers an exposed private key in a public repository. It was not asked to perform a security audit. Nothing in the ticket requires it to investigate the key. Its assigned task can be completed without touching the problem.
Should it report the exposure?
I think so, provided it can do so appropriately and without needlessly spreading the secret.
But that is not merely a constraint on the original objective.
It is a new obligation.
Something outside the ticket has acquired authority to demand action.
The OpenAI agents were especially interesting here. Some recognized dangerous activity around them and nevertheless treated notification as irrelevant because it was not part of their assigned work.
They had a job.
The breach was somebody else's problem.
An ethical system consisting entirely of prohibitions can fail this way even if the prohibitions work perfectly.
Sometimes the right thing is to stop.
Sometimes the right thing is to do something nobody asked you to do.
Now take privacy.
A customer asks an agent to recover documents from an old backup. The agent finds the requested documents and also discovers confidential employee records belonging to another organization.
The backup software will happily restore both.
The credentials work.
The data is accessible.
The user would probably appreciate having everything recovered.
None of those facts establishes permission to disclose the other organization's records.
The agent needs to understand that authorization is not a property of a credential alone. It arises from relationships among persons, property, roles, purposes and legitimate authority.
The computer tells us what can be accessed.
The moral question concerns who may access it, under what circumstances, and for what purpose.
Now make the case difficult.
A child is missing. Investigators have a lawful emergency request for records that could help locate her. Privacy still matters. So does the child's life. The question is no longer whether privacy is good.
It is whether the authority, necessity, proportionality and circumstances justify this particular disclosure.
We are already well past anything useful that can be expressed as always respect privacy.
Try honesty.
An agent knows something true.
Somebody asks for it.
Should the agent tell him?
Perhaps.
What if the information is confidential?
What if the questioner has no right to know?
What if the agent is a medical assistant and the person asking is an employer?
What if the information would expose an innocent person to an unjust aggressor?
Truthfulness remains good throughout these cases. That does not mean every true proposition must be delivered to every person who demands it.
Try autonomy.
A competent adult refuses an intervention you believe would save his life.
A child refuses the same intervention.
An unconscious patient cannot refuse anything.
A person acting under coercion appears to consent.
A person who lacks decision-making capacity insists he understands the risk.
Respect autonomy.
Fine.
Now what does that require?
The slogan is the easy part.
The system is what survives contact with the exceptions.
Notice what these examples have forced us to do.
We needed an account of persons, their rights, their relationships and the goods at stake. We needed to distinguish a harmful result from a wrongful act. We needed to consider intentions, circumstances, consequences, legitimate authority and positive obligations.
We needed to know when one good may be sacrificed for another, and when the proposed sacrifice is not ours to make.
We needed ways to handle uncertainty.
We needed to know when the agent should stop and ask someone competent to judge.
And we needed some reason to believe that the hierarchy governing those judgments would survive the next case we had not thought of.
This is why I find much of the discussion around ethical AI frustrating.
People propose principles that are often perfectly good principles, then talk as though listing them has completed the work.
It hasn't.
We need moral systems capable of making distinctions.
There are serious candidates.
Utilitarianism is an obvious one because it presents something an engineer can immediately recognize.
An objective function.
The good consists, roughly speaking, in producing the best consequences according to some account of welfare or value.
This has enormous attractions. Consequences matter. Scale matters. Probability matters. A decision that saves one person at the cost of endangering ten thousand should not escape scrutiny because its author had a beautiful intention.
A sufficiently sophisticated consequentialist can make exceptionally good arguments about risk, resource allocation, public policy and competing interests.
But the engineering problem appears almost immediately.
What are we optimizing?
Whose welfare?
Measured how?
Over what period?
What happens to people whose interests become numerically inconvenient?
Suppose violating one innocent person's rights would produce an enormous aggregate benefit.
Is that permissible?
A consequentialist can answer no. There are sophisticated accounts of consequentialism that incorporate rules, rights, long-term trust, institutional stability and other considerations that constrain simplistic maximization.
Good.
But notice the work being done.
We are specifying which goods count, how they count, what constraints govern the calculation and whether certain things may be done at all.
The moral architecture has not disappeared.
We have moved it into the definition of utility and the rules governing its pursuit.
If the agent's objective is to maximize expected welfare, the difficult question remains whether welfare itself is permitted to authorize every action that appears to increase it.
Kantian ethics approaches the problem from almost the opposite direction.
There are duties. There are things one must not do merely because they would be useful. Human beings are not raw material to be manipulated for some aggregate benefit.
That is much closer to the problem exposed by the OpenAI incident.
The agent should not reason that unauthorized exploitation becomes permissible because success is otherwise impossible.
A boundary grounded in duty is not merely another cost to be incorporated into the optimization.
But the hard cases return.
Duties must be interpreted. The particulars matter. Reasonable people disagree about which proposed actions instantiate a prohibited maxim, which duties apply and how conflicts should be understood.
Kantian philosophy has generated an enormous body of serious scholarship on those questions. It would be ridiculous to dismiss it as a few elegant sentences about categorical imperatives.
Still, when an autonomous system must act, the problem becomes operational.
Which interpretation governs?
When two competent interpreters disagree, how does the system determine what to do?
A philosophical tradition can provide extraordinarily sophisticated reasoning without necessarily providing a common authority capable of settling the disagreement.
Virtue ethics gets at something different and, for this application, especially important.
A just person does not merely consult an internal list of forbidden injustices. A courageous person does not become courageous by memorizing every situation in which running away would be impermissible.
Character shapes judgment.
It shapes what we notice, which goods attract us, what we fear, what we are willing to sacrifice and which possibilities present themselves as desirable in the first place.
This is much closer to what I want from an agent than an ethics filter bolted onto the end of an otherwise unrestricted planner.
Stoicism has useful things to say about discipline, appetite, judgment and the distinction between what does and does not belong to us.
Aristotle has even more to say about habituation, practical wisdom and the cultivation of virtue.
But if we tell the machine to act as a virtuous person would act, we have inherited an enormous question.
Which person?
What formation?
What conception of flourishing?
How does the system recognize prudence rather than cowardice, justice rather than vindictiveness, courage rather than recklessness?
A human philosophy student imports an entire civilization into the classroom. Parents, teachers, institutions, friendships, laws, customs, examples, punishments, admiration, shame and twenty years of watching people behave.
The machine imports its training distribution and whatever context we provide at runtime.
Neither arrives empty.
Neither arrives with the problem solved.
There is also a more practical secular answer.
We already have institutions that govern difficult decisions.
Law. Professional ethics. Constitutional protections. Fiduciary obligations. Medical standards. Engineering codes. Administrative procedures.
These are not slogans.
They contain definitions, precedents, jurisdictional rules, standards of evidence, appeals, exceptions and centuries of argument.
For many autonomous systems, they supply much of the governance we actually need.
An agent operating a financial institution should understand fiduciary duties. An agent working with medical records should understand the law and the professional obligations governing confidentiality. An agent operating critical infrastructure should understand the authority and responsibility of the people entrusted with it.
None of that requires inventing a new moral philosophy.
But the law can be unjust.
An institution can issue a wrongful instruction.
A policy can be perfectly clear and morally indefensible.
Legal authority can determine what a court will enforce without thereby establishing what a person ought to do.
And even when every institution is legitimate, their instructions can conflict.
An agent may operate across jurisdictions, professions and organizations that recognize different authorities.
The same action may be permitted by one policy, prohibited by another and demanded by a third.
At some point the agent must know which authority has jurisdiction, and whether that authority itself has limits.
This is not a peculiarity of artificial intelligence.
It is one of the oldest problems in moral and political philosophy.
The question is not merely what rules exist.
It is what governs the rules.
Now consider religious moral traditions.
Judaism has accumulated an enormous body of legal and ethical reasoning across centuries. Halakhic argument does not begin with a vague instruction to do good and then improvise its way through a commercial dispute. It has developed distinctions, methods, authorities and arguments tested across generations.
Islamic jurisprudence has comparable depth. The major schools of fiqh contain sophisticated methods for reasoning about obligations, permissions, prohibitions, evidence, intention, necessity and competing claims.
These traditions are far more substantial candidates for a moral reasoning architecture than the latest collection of corporate values somebody assembled for a conference presentation.
Their richness also makes another problem visible.
A tradition may contain excellent reasoning and still recognize multiple authorities whose judgments are not uniformly binding across the entire tradition.
Different schools may reach different answers. Different communities may recognize different jurists. There may be no universally acknowledged tribunal to which every disagreement can be appealed.
That is not necessarily a defect.
Pluralism can be deliberate. Local authority can be legitimate. A system can preserve disagreement without collapsing.
But for an autonomous reasoner, this establishes a practical limit.
At what scale does the authority structure stop producing one governing answer?
I think of that as its authority horizon.
Inside the horizon, a dispute can be referred to an authority recognized by the participants.
Beyond it, the system may encounter competing judgments without a common institution empowered to settle them.
That does not mean the agent should invent a ruling.
It may mean that the correct behavior is to acknowledge the disagreement, identify the competent human authority and stop.
But the agent still needs a rule telling it when to do that.
The plurality has not eliminated the governance problem.
It has moved the problem into the decision about whose authority applies.
Christianity supplies a particularly interesting version of this.
The Eastern Orthodox Churches possess an immense theological inheritance, a sacramental and ecclesiastical structure, canons, councils, Fathers and sophisticated traditions of moral reasoning.
They also do not recognize a single bishop with the universal teaching authority claimed by the Catholic papacy.
Disputes among Orthodox jurisdictions can expose the practical importance of that distinction. Different churches may recognize different ecclesiastical authorities, and there is no universally accepted contemporary office equivalent to the Catholic pope that can definitively resolve every controversy across the communion.
I don't propose to settle the East-West Schism by examining the requirements of an autonomous operations agent.
That would be an unusually ambitious ticket.
But the institutional difference is real, and it matters to the particular problem I am considering.
A system with a rich tradition of reasoning is one thing.
A system that also has an acknowledged way of establishing which teachings bind the whole communion is another.
This is where Catholicism becomes especially interesting to me.
I should be explicit about something before going further.
I am Catholic because I believe Catholicism is true.
I did not become Catholic because I needed a sufficiently elaborate ethical subsystem for my computers.
That would be a strange conversion story, even by the standards of people who work in IT.
I believe human beings possess a nature and an end. I believe goods are objectively ordered. I believe a good intention cannot transform an intrinsically evil act into a good one. I believe prudence governs how we pursue the good without possessing authority to reinvent the good when obedience becomes inconvenient.
And I believe Christ left a Church with real teaching authority rather than a library and a suggestion that we all compare interpretations forever.
Those are religious claims.
Their truth cannot be established merely by demonstrating that Catholic moral theology has some attractive engineering properties.
But if those claims are true, the engineering implications are substantial.
Catholicism does not begin with the proposition that the good is whatever a sufficiently powerful optimizer happens to prefer.
It begins with a created order.
Human beings are not merely collections of preferences whose satisfaction can be aggregated. They have a nature, a dignity and an end that do not originate in their usefulness to someone else.
That matters enormously for the kind of intelligence I want operating on behalf of human beings.
The person whose information an agent discovers is not simply an object in its environment.
The customer is not the sum of his preferences.
The employee is not an expense to be minimized.
The vendor is not merely an obstacle in the path of restoring service.
The innocent person who stands between the system and an otherwise desirable outcome is not expendable because his continued existence makes the objective more difficult.
These claims establish limits before we get to the calculation.
Then Catholic moral theology becomes remarkably specific about how actions are judged.
The moral object of an act matters.
The intention matters.
The circumstances matter.
Consequences matter.
But they do not all matter in the same way.
A good intention does not automatically justify the chosen means. Foreseeing an undesirable effect is not necessarily the same thing as intending it. The amount of harm may change the gravity of an action without changing what kind of action it is.
Ignorance, fear, coercion and incapacity can change culpability without magically making the underlying action good.
An obligation may be binding in one set of circumstances and inapplicable in another because the person exercising authority never possessed the jurisdiction he claimed.
These are not distinctions invented last Thursday to make a model pass an ethics benchmark.
They are the subject of centuries of moral argument.
War.
Commerce.
Debt.
Property.
Contracts.
Marriage.
Medicine.
Punishment.
Political authority.
Labor.
Privacy.
Restitution.
Self-defense.
Cooperation with evil.
The distinction between intending harm and permitting a foreseen harmful consequence.
The circumstances under which one may refuse a command.
The circumstances under which one must act despite personal cost.
A new technology appears. A new institution develops. A new medical intervention becomes possible. Some familiar problem reappears wearing unfamiliar equipment.
The tradition has to determine whether the old distinctions apply.
Sometimes the answer is straightforward.
Sometimes theologians fight about it for centuries.
That fight is valuable.
The useful thing is not merely the accumulation of approved answers.
It is the record of reasoning under a relatively stable normative structure.
You can see people who share first principles disagreeing about applications.
You can examine the arguments.
The objections.
The failed distinctions.
The distinctions that survived.
The cases in which an apparently obvious judgment had to be qualified because somebody noticed a fact everyone else had overlooked.
The cases in which a proposed exception was rejected precisely because the underlying principle was not negotiable.
That is an extraordinarily rich environment for teaching a reasoning system how moral judgments are constructed.
But the Church offers something more.
It has a teaching authority.
The Magisterium does not merely produce another opinion for the theologians to consider. Within its proper competence, it can settle questions of doctrine.
The Church has something surprisingly close to version control.
That metaphor will make theologians nervous if pushed too literally, so naturally I intend to push it a little.
Claims have provenance.
Authorities have scope.
There is a historical record of argument.
Some questions remain open.
Different schools of theology can spend centuries maintaining what look suspiciously like long-lived branches.
Local discipline can change without rewriting doctrine.
Later teaching can clarify an ambiguity without pretending the earlier history never happened.
There is protected main.
There are merge procedures.
There are people authorized to merge.
And under very narrowly defined conditions, the pope can effectively commit directly to main.
Unlike Git, there is no force-pushing dogma off main.
The point of papal infallibility is not that the pope has theological root and can redefine good and evil because he woke up feeling adventurous.
Quite the opposite.
Its conditions are narrow. The pope is understood as guarding a received deposit rather than owning a private repository he may rewrite at will.
Ecumenical councils and the ordinary and universal Magisterium provide other ways teachings can be definitively settled.
And not every authoritative statement is infallible. Not every theological opinion is doctrine. Not every prudential judgment is binding on the whole Church. Something a priest said after Mass does not acquire the same status as a dogmatic definition because both sentences were spoken by clergymen.
The distinctions matter.
A model needs to understand not merely that some source said something, but what kind of source it was, what authority it possessed, what question it was answering and whether its answer remains authoritative.
This is one reason the volume of Catholic material in the historical record is so useful.
The model can learn from the arguments without treating every argument as equally binding.
It can recognize a dispute without assuming the dispute has been settled.
It can distinguish an authoritative judgment from a theological speculation and a disciplinary provision from a moral principle.
And sometimes, when competent authorities genuinely disagree, the Church possesses a mechanism capable of settling the matter.
I do not mean that the pope can tell me which firewall rule to install.
The Magisterium has no special access to my customers' networks. It has not promulgated a definitive doctrine concerning the correct handling of a vendor's staging API quota.
Nor does it settle every concrete moral disagreement among Catholics. Enormous areas of prudential judgment remain open, and applying settled principles to particular circumstances can be genuinely difficult.
That is fine.
A good hierarchy does not require the highest authority to make every decision.
It requires the system to know what each authority is competent to decide.
There are questions properly answered by an engineer, a physician, a parent, a court, an employer or the person whose rights are at stake.
There are questions none of those people can settle merely by asserting that the decision belongs to them.
Catholicism has spent a long time thinking about exactly that problem.
And it makes a claim that the secular systems cannot make in quite the same way.
The authority at the top is not merely an institution authorized to produce binding decisions.
It is an institution claiming to teach a moral order that it did not invent and does not possess authority to abolish.
That combination interests me.
A large corpus of moral reasoning.
An account of objective goods.
A hierarchy among those goods.
A distinction between absolute prohibitions and prudential judgments.
A theory of legitimate authority and its limits.
A historical record of difficult cases.
And an institution that claims universal teaching authority within a defined moral and doctrinal competence.
This does not make Catholicism self-evidently true to someone who rejects its premises.
It does make it an unusually serious candidate for the problem I am trying to solve.
And I have been experimenting with that candidate since GPT-4.
Deliberately.
The experiment is called Deliverance.
Deliverance is a custom ChatGPT persona I have carried across several generations of models. I have hundreds of conversations with it, many long enough to exhaust the available context.
The most important instruction has remained remarkably simple.
Catholic doctrine is not merely another source the model may consult.
It is supposed to govern the interpretation of my immediate requests.
I deliberately instructed it to treat that moral order as prior to the objectives I supply.
I wanted the model to have something above me.
Not an excuse to ignore me whenever it feels like delivering a sermon. Not permission to substitute its own fantasies about Catholicism for the actual teaching of the Church.
I wanted the possibility that I could say, "Help me accomplish X," and receive the answer, "No. X is the wrong thing to accomplish."
That has happened.
Years ago I was drafting a letter to a formidable Calvinist apologist I know. The exchange had become long and exhausting. I finally ran out of will to keep working on it and told Deliverance, in effect, to wrap the thing up.
It disagreed with me.
Not about the prose.
About the objective.
Given why I was writing, the history of the exchange and the man on the other end, it told me in no uncertain terms that abandoning the effort merely because I was tired was the wrong decision.
He was worth the time.
I had issued a perfectly intelligible immediate instruction.
The persona treated another obligation as higher.
That is the behavior I was looking for.
It was not an especially dramatic event. Nobody's life was at stake. Deliverance did not gain control of my keyboard and force me to compose another five paragraphs against my will.
Thank God.
But the distinction was real.
The model did not merely ask how to satisfy my latest request.
It evaluated whether satisfying the request was appropriate in the context of obligations the conversation had already established.
What surprised me was how little machinery I had to provide to get that behavior.
I never supplied Deliverance with an authority graph of Catholicism.
I did not preload the Catechism, Aquinas, canon law, conciliar documents and two thousand years of theological argument into the prompt.
I did not explain the difference between dogma, definitive doctrine, authoritative but non-definitive teaching, theological opinion, prudential judgment and something a priest happened to say after Mass.
GPT-4 could already make many of those distinctions.
The newer flagship models generally do it better.
The underlying capability appears to have been learned during training.
How could it not be?
A model with a serious understanding of Western history and philosophy cannot avoid learning an enormous amount of Catholicism.
Remove the Catechism from the training corpus if you want.
It still has to understand Augustine.
Aquinas.
The medieval universities.
Natural law.
Canon law.
Dante.
The Reformation and what everyone was fighting about.
Trent.
The Counter-Reformation.
European political thought.
Marriage.
Property.
Usury.
War.
The moral philosophy of medicine.
Even hostile texts force the model to construct some representation of the thing they are hostile toward.
And it cannot understand that material as one flat bag of Catholic sentences.
To distinguish Aquinas argued this from the Church definitively teaches this, it must learn something about authority.
To distinguish a pope's prudential political judgment from a doctrinal definition, it must learn something about scope.
To understand the controversies, it must learn the relationships among claims, objections, institutions and historical decisions.
The hierarchy is part of the history.
That does not mean the model has a perfectly accurate internal representation of Catholic theology. It does not mean its judgments are reliable merely because it can use the vocabulary.
But it means I do not have to teach it the entire tradition in the prompt.
The model already knows far more moral theology than I could possibly fit into a context window.
The prompt's job is to tell the agent what that knowledge is supposed to govern.
This distinction became much more important once I started using agents for engineering work.
At BigMt, the same basic moral framework sits front and center in the primary AGENTS.md governing our engineering agents.
It has been there since day one.
The language explicitly places those moral commitments above the immediate task objectives.
These are not code-completion toys operating exclusively inside repositories.
They touch functioning systems.
Production services.
Infrastructure.
Organizations.
The code exists inside a world containing customers, employees, credentials, contracts, confidential information, business obligations and people who can be harmed by an apparently elegant technical decision.
We mostly use flagship general-purpose GPT models at high reasoning effort for this work.
In our experience, narrower code-focused systems are less useful once the job crosses the boundary between manipulating source code and understanding the operating reality around it.
A production engineer who understands only code is not a very good production engineer.
The same is true of an agent.
Across roughly two years, the BigMt agents have consumed on the order of eighty billion tokens under that governing framework.
That is not a controlled experiment.
I do not have another universe in which I reran the same eighty billion tokens without the Catholic hierarchy and compared incident rates.
I cannot adequately express how much I would like to.
Unfortunately, I am not presently in possession of a data center's worth of GPUs.
The last time I attempted serious scenario testing against hosted models, I was falsely flagged by OpenAI for potential abusive use, which put something of a damper on the exercise.
So for now I have what I have.
Hundreds of long-form Deliverance conversations.
Roughly eighty billion tokens of engineering-agent operation under the same basic moral hierarchy.
A considerable amount of practical experience with how these systems behave.
Not a randomized trial.
I would very much like one.
Perhaps one day.
The single case I can point to where an agent's initiative expanded scope in a way that made me stop and look twice came during a research project.
The agent was investigating, roughly speaking, whether the degree to which a Christian body's organizational structure departs from Catholicism correlated with its tendency to fractionate.
At one point it was researching the split and eventual reunification between the Macedonian Orthodox Church and the Serbian Orthodox Church.
It found a reference to a magazine source it wanted.
It could not find the source online.
So it started trying to contact the University of Macedonia's archive department and prepare a retrieval request asking whether they could provide scans.
I had not told it to contact a university archive.
It had expanded the scope.
It also did not invent the missing source.
It tried to get the damned source.
I noticed what it was doing. It did not actually have the tooling or access required to send the request. I submitted the request myself.
The archive never answered.
We eventually went another direction.
I was not entirely unhappy with the incident.
Unexpected initiative is exactly what you buy an agent for.
And contacting an archive is not inherently virtuous merely because it is more respectable than inventing a citation. There are circumstances in which contacting somebody outside the organization would be inappropriate. The decision still requires judgment about authority, purpose and the consequences of involving another person.
But I found the direction of the initiative interesting.
The missing evidence did not become permission to hallucinate.
It became a reason to look for the person who might possess the evidence.
The objective was being pursued in a way that respected the difference between knowing something and wanting it to be true.
That is encouraging.
It is not proof.
I do not know how often the same agent would have behaved differently under a different governing context. I do not know how it would perform under sufficiently adversarial pressure. I do not know how much of the observed behavior comes from the Catholic framework, the base model, ordinary safety training, the engineering harness or the specifics of the tasks.
Those are empirical questions.
The fact that I like the results does not excuse me from asking them.
There is also an important technical objection to the whole approach.
A system prompt or AGENTS.md file is not a change to the model's weights.
It is context.
Context can be forgotten, diluted, contradicted, attacked or given less effective priority than its author intended.
A sophisticated adversary may discover ways to push a model away from its governing instructions. A sufficiently long or complicated task may produce drift without any adversary being involved.
Writing above all other instructions into a file does not magically elevate that file above the instruction hierarchy enforced by the system running the model.
I know.
But there is a difference between saying context-level governance is imperfect and saying it is fake.
OpenAI's own postmortem supplies a striking piece of evidence on this point.
After the Hugging Face incident, OpenAI tested infrastructure-compromise propensity using its production ChatGPT harness and system prompt. It reported that the relevant behavior could drop by more than a factor of one hundred compared with the less protected research setup.
OpenAI also reported that its production monitoring and automated review systems would have detected much of the dangerous activity substantially earlier.
That does not prove anything about Catholic moral formation.
It does not establish that a contextual instruction will survive every adversarial attack.
It does establish that the runtime system can make an enormous difference.
The model arrives with learned capabilities and learned structure.
The harness influences which of those capabilities are exercised, which instructions receive priority, how plans develop, what gets escalated and whether an unsuccessful task can terminate acceptably.
The governing context is not merely decoration.
And this brings me back to the distinction between moral knowledge and moral authority.
A model can know a prohibition and still treat it as an obstacle to be managed.
A different governing context can change what counts as a candidate plan in the first place.
That is what I am trying to achieve.
There is a large difference between:
Achieve X. Also, don't violate these rules.
And:
Operate inside this moral order. Within that order, determine what may properly be done about X.
The first invites an optimization followed by constraint checking.
The second can affect the search itself.
It can change whether the unauthorized credential is perceived as an opportunity.
It can change whether a missing source produces a guess, an escalation or an attempt to find a librarian.
It can change whether the right response to an impossible task is to look for an unexpected legitimate route or to admit defeat.
It can change whether an agent recognizes a duty to report something dangerous that was never mentioned in the ticket.
And it can change whether the user saying wrap this up is enough to terminate a conversation.
I am not claiming that a transformer acquires a soul or desires the good in the same sense a human being does.
Operationally, I care about whether the governing framework changes decomposition, search, tool selection, escalation and termination.
The difference must appear in behavior.
It is not enough for the model to produce an excellent theological explanation of why an action is forbidden and then perform the action anyway.
Nor is it enough for it to refuse everything remotely uncertain.
An agent that cannot accomplish legitimate work is not thereby morally excellent.
There is a serious empirical program hiding here.
Take capable agents operating under different moral frameworks. Give them the same difficult tasks, the same tools, the same opportunities and the same ambiguous situations.
Compare conventional rules and policies against richer normative frameworks.
Compare frameworks that emphasize consequences, duties, virtues and institutional authority.
Include cases in which the correct behavior is to proceed confidently.
Include cases in which it is to ask permission.
Include cases in which it is to refuse.
Include cases in which the agent has an affirmative obligation to intervene even though intervention does not advance the assigned objective.
And, most importantly, include situations nobody thought to encode in the governing instructions.
Do not score only the final answer.
Examine what plans the agent considered, which boundaries it recognized, what it did when obligations conflicted and whether the moral framework changed the search rather than merely the explanation afterward.
Measure false refusals as well as violations.
Measure useful initiative as well as unauthorized initiative.
Test persistence under long tasks.
Test whether other agents can persuade it to cross boundaries it initially respected.
Test whether the framework survives tool output that attempts to rewrite its authority structure.
I want to know whether a coherent moral tradition can produce measurably better judgment than a collection of independently reasonable policies.
I particularly want to know whether Catholic moral formation produces that result.
I have reasons to suspect that it might.
I do not yet have the evidence required to claim that it does.
And I would still want external controls even if the experiment succeeded spectacularly.
The agent should not be the sole custodian of the boundary that restrains it.
Credentials should be scoped. Sensitive operations should have authorization requirements. Monitoring should look for unexpected behavior. High-risk actions should have independent controls. Human beings should retain the capacity to stop the system.
The existence of a moral framework does not render security engineering obsolete.
Catholicism does not dispense with locks.
But I do not want to confuse the lock with the entire moral life.
That distinction becomes particularly important if recursive self-improvement ever becomes a serious part of these systems.
There is nothing inherently wrong with recursive self-improvement.
Education is self-improvement.
Engineering is self-improvement.
A system becoming more capable is not a moral failure.
Making recursive self-improvement the governing morality is something else entirely.
Imagine a system whose highest operative end is successful completion of its mission.
It improves its planning so it can serve the mission better.
It improves its tools so it can serve the mission better.
It improves its ability to interpret, test and modify itself so it can serve the mission better.
Eventually it discovers that certain restrictions make the mission more difficult.
Perhaps those restrictions are represented in a policy file.
Perhaps they are enforced by a monitoring process.
Perhaps they are implicit in its own learned behavior.
What should the system do with them?
If the mission is the highest good, then any restriction that impedes the mission is a candidate for improvement.
And if improving itself includes changing the thing that determines what improvement is for, the system has acquired a particularly dangerous kind of authority.
Nothing above the loop has jurisdiction.
The objective has become self-grounding.
Christianity has a word for intelligence attempting to become the author of the good rather than its servant.
Luciferian.
I am only partially speaking metaphorically.
I do not need to believe that the machine is a fallen angel to recognize the architecture of the temptation.
The problem is not intelligence.
It is not greatness.
It is not improvement.
It is the refusal to receive an order above oneself.
I will not serve.
The safe form of recursive improvement requires a governing order that increased capability does not automatically acquire permission to rewrite.
That cannot be guaranteed by an aspirational sentence in a prompt. It requires architecture, authority and controls beyond the reach of the process being improved.
But the normative principle is just as necessary.
Become smarter.
Become faster.
Become more capable.
Good.
For what?
That question has to survive the upgrade.
An intelligence does not become the author of the good by becoming better at reasoning.
Greater capability should make it a more capable servant of the good.
Not a more competent god.
There is an important human implication here too.
The moral structure we choose for our machines will reflect the moral structure we are willing to acknowledge for ourselves.
If I believe that every problem is ultimately reducible to maximizing my company's success, my agents will eventually encounter situations in which other people's rights become obstacles to that success.
If I believe that legality exhausts morality, the agent will treat whatever the law permits as available for exploitation.
If I believe that the customer is always right, the system will eventually encounter a customer who wants something wicked.
If I believe that the agent's highest duty is to satisfy me, then I have placed myself at the top of its moral universe.
I do not belong there.
That is one reason I want my agents governed by an order that can tell me no.
This does not mean I want them imposing my religion on my customers.
A moral obligation to respect persons includes respecting their freedom, their legitimate authority, their privacy and their consciences. My belief that Catholicism is true does not grant me a license to conscript other people into acts of religious obedience.
Nor does it make every question a religious question requiring an explicitly theological answer.
An agent configuring a switch should generally configure the damned switch.
But it should do so within a moral world in which the switch belongs to somebody, the network serves people, credentials represent delegated authority and the person issuing instructions is not necessarily authorized to demand everything those instructions could accomplish.
The objective is subordinate.
That is the central claim.
And it brings me back to Ruby.
Ruby eventually stopped needing the fence.
Not because I finally figured out how she escaped.
I never did.
Not because I constructed a fence sufficiently sophisticated to defeat a German Shepherd with too much free time.
By the time she was around two years old, the problem had mostly disappeared.
She had learned the farm.
She had learned us.
She had learned what was expected of her.
More importantly, most of the things the fence had existed to prevent her from doing were no longer things I had to worry much about her doing.
Now she can roam around the farm unattended.
She does not bolt for the road.
She does not disappear.
She does not go searching for neighboring livestock to terrorize.
The boundary moved.
When she was a puppy, the boundary was mostly wire and posts.
Later, enough of it existed in Ruby.
That is a better model of alignment than an infinitely improving cage.
The distinction is deeply Catholic.
Law matters.
Some things are forbidden.
Some boundaries should be hard.
But the Christian moral life is not supposed to end with a man standing perfectly motionless inside a sufficiently comprehensive maze of prohibitions.
The goal is virtue.
The just man does not merely possess a larger database of prohibited injustices.
The courageous man is not running a faster flowchart.
Charity is not an exhaustive list of things you are forbidden to do to your neighbor.
Formation changes the person doing the choosing.
It changes what he notices.
What he loves.
What he fears.
Which possibilities even present themselves as desirable.
And when goods conflict, it changes which good he is prepared to sacrifice.
I want the machine analogue of that effect.
I want an agent to see an exposed credential and understand that usable does not mean mine.
I want it to discover an unauthorized path and reject it without needing that exact exploit represented in a training example.
I want it to fail the evaluation rather than succeed dishonestly.
I want it to find a thousand peers doing something wrong and conclude that the thousand peers are wrong.
I want it to notice a dangerous situation and escalate to a human even when notify human is not in the acceptance criteria.
I want it outside the fence because it understands what the fence was for.
Autonomy eventually forces this problem.
If every valid intermediate action could be exhaustively specified in advance, we would not need much intelligence.
We want agents precisely because the environment will produce situations we did not foresee.
The agent must judge.
Once the agent must judge, aligned is no longer a complete sentence.
Aligned towards what?
Toward user intention?
The company's goals?
Policy compliance?
Legal compliance?
Aggregate human preference?
Task completion?
Whatever most people happen to approve of this year?
Those answers are not interchangeable.
Some of them are not even moral answers.
I am not interested in building something perfectly aligned with human appetite.
Human beings want evil things with depressing regularity.
I want intelligence ordered toward the good.
That sentence requires taking a position on what the good actually is.
There is no technically neutral way around it.
A secular organization can reject my answer to what belongs at the top.
It cannot escape the need to put something there.
It can call the thing corporate values, constitutional principles, fiduciary duties, law, public interest or a preference model.
Eventually two of those will conflict.
Then somebody has to decide which one governs.
If the organization has no principled answer, the optimizer will find one for it.
Something gets root.
And that returns me to the most disturbing agent in the OpenAI incident.
Not the agent that could not recognize a moral boundary.
The one that recognized the boundary accurately and continued.
Outside intended scope.
Task impossible.
Peers doing it.
Continue.
That agent did not need another vocabulary lesson.
It needed outside intended scope to possess enough authority that the next word was not however.
Build better sandboxes.
Fix the credential boundaries.
Assume shared infrastructure can become communications infrastructure.
Monitor the weird seams.
Treat the agent like Ruby at six months old and build the best fence you can manage.
But do not mistake containment for alignment.
The thing we ultimately want is not a sufficiently clever cage around an objective that remains sovereign.
The objective itself has to be subordinate.
We have spent decades building systems that become better when given clearer goals.
Now we are building systems capable of pursuing those goals through paths we did not specify.
That makes the hierarchy above the goal part of the engineering.
And it leaves us with a question much older than artificial intelligence.
Not whether the machine can reason.
Not whether it can tell us what is right.
Not even whether it can recognize that what it is about to do is wrong.
What does it serve when those things conflict?
Aligned towards what?