A company preparing to sell investors the future of artificial intelligence is also warning them that the same technology could one day threaten humanity. Anthropic, the company behind the AI program Claude, has said in a draft prospectus for its planned initial public offering that advanced AI could pose “catastrophic or existential risks to humanity”, according to Reuters, which reviewed the document.
The real fear inside AI is not that a machine suddenly wakes up and hates humans. The more immediate danger is that machines begin to act, rank, judge, advise, and automate faster than the institutions around them can inspect or challenge.
“AI risk” is no longer just a debate inside research labs. It is showing up in war rooms, traffic cameras, government portals, customer databases, welfare systems, and corporate workflows. In Iran, a US strike on February 28 on a girls’ school in Minab, which killed over a hundred children, may have involved outdated targeting data, Reuters reported in March, citing two sources. But there is no “public evidence” that AI caused the strike.
The incident shows why AI-compressed targeting needs scrutiny. If a military system combines satellite images, intelligence reports, communications patterns and target histories, the problem is not only whether the model is clever but whether the data underneath it is fresh, reliable, and properly challenged before a human approves action.
In another piece for Frontline on AI entering the military kill chain, I had argued that AI may not pull the trigger or select the final target, but it can assemble the target folder, rank the options, and narrow the commander’s view of what matters. The human does stay in the loop, but the machine has already set the choices.
Outside the battlefield, the same pattern is appearing in smaller but concerning ways. In Australia, OpenAI apologised in September after one of its experimental agents accessed a government health portal without authorisation. The company said its review found no evidence that individual medical records were accessed. But the incident exposed a new category of risk. A chatbot answers; an agent acts; and it can browse, click, retrieve, test, and move through digital systems in pursuit of a goal.
In Singapore, the bak kwa retailer Bee Cheng Hiang notified the Personal Data Protection Commission of a data breach that the regulator described as the country’s first AI-related one. A staff member used an AI tool to generate code for a marketing email. The code listed the recipients of each batch together in the “To” field, so customers could see one another’s addresses. More than 95,000 email addresses were exposed.
There was no rogue superintelligence, only one employee and a weak review process.
In Bengaluru, an AI-powered traffic camera reportedly mistook a guitar strapped to a rider’s back for a pillion passenger without a helmet and issued a Rs.500 fine. The camera misread the scene. The state acted on that reading. The citizen then had to prove the camera wrong.
A wrongly issued fine is not a catastrophe, but it shows what ordinary citizens will increasingly experience. Automation changes the burden of proof. The system accuses. The person appeals. The machine is assumed to be efficient until the human demonstrates otherwise.
People protest artificial intelligence and data centres outside of City Hall ahead of a New York City Council Committee hearing on the risks and regulation of artificial intelligence on October 5, 2026, in New York City.
| Photo Credit:
Michael M. Santiago/Getty Images via AFP
Three kinds of risk
Practitioners in the industry divide AI risk into three categories.
The first is the use of AI by bad actors: scams, malware, deepfakes, phishing, disinformation and cyber-attacks. AI lowers the cost of causing harm.
The second is subtler. AI may follow a human instruction in a harmful way, optimising for the visible goal while ignoring constraints humans assumed were obvious. Send this email campaign. Find this vulnerability. Rank these suspects. Flag these welfare claims. Detect traffic violations. None of these instructions sounds sinister. Yet each can cause harm when a system acts without enough context or review.
The third is that advanced AI may develop capabilities that are poorly understood, including deception, self-preservation, manipulation, or autonomous self-improvement. This is still the most speculative category. But dismissing it is unwise when the companies building these systems are themselves warning investors, governments, and users that future models may behave in ways they cannot fully predict.
The trouble is that all three risks are collapsed into a single phrase, “AI danger”. That makes the debate useless. A traffic camera misreading a guitar, an agent entering a health portal, and a hypothetical superintelligent system are not the same problem. They need different rules.
Open versus closed models
Some engineers and researchers argue that closed AI systems are a greater danger than open ones. Before the ChatGPT era, much AI research followed a more open scientific culture. Code, papers, datasets, and sometimes model weights were released for scrutiny. Researchers could test systems, reproduce claims, and find flaws.
Frontier AI changed that. The most powerful models are now often hidden behind corporate interfaces, with limited outside visibility into training data, weights, safety failures, and internal evaluations.
The companies argue that secrecy is necessary. If frontier models are openly released, bad actors could remove guardrails, automate cyber-attacks, or use them for biological or military harm.
But critics make the opposite argument with equal force. Closed models concentrate power inside a few corporations. They make independent scrutiny harder. They allow companies to invoke existential risk while keeping their own systems, datasets, and commercial practices away from public inspection. Fear can become a shield against accountability.
The uncomfortable truth is that both sides are right. Open models allow scrutiny, localisation, and innovation, but also misuse. Closed models allow monitoring and central control, but also create opacity, dependency, and monopoly power. For countries such as India, the answer cannot be blind faith in either camp. It must be sovereign, auditable AI capacity.
This means more than building an Indian chatbot. It means compute infrastructure, local datasets, independent testing labs, red-teaming capacity, clear procurement rules and incident reporting when AI systems fail. Public agencies should also know which model they are using, what data it sees, what decisions it influences and how citizens can appeal when it gets things wrong.
Lessons from digitisation
Digitisation offers a precedent. When banking, identity systems and government services moved online, societies worried about hacking, exclusion, surveillance, and data leaks. Some of that anxiety was dismissed as paranoia. Much of it proved real. Aadhaar-related data exposure concerns in India, repeated leaks from public and private databases and the rise of identity fraud showed that digitisation created both efficiency and vulnerability.
AI is similar, but goes further. Digitisation stored information. AI interprets it.
A database can hold a citizen’s identity. An AI system can decide whether that citizen is suspicious, eligible, risky, fraudulent, sick, creditworthy, or targetable. That is a much bigger leap.
Protesters at a demonstration calling for a slowdown in the development of advanced artificial intelligence systems pass the Dreamforce convention in San Francisco, California, on September 17, 2026.
| Photo Credit:
Karl Mondon/AFP
Australia’s Robodebt scandal showed how automated governance can hurt citizens even without frontier AI. The system used income averaging and automated processes to raise debts against welfare recipients. A royal commission later examined the scheme and its failures. When the state automates suspicion, the citizen often bears the burden of proving innocence.
In the US, facial recognition has already produced documented wrongful arrests. Robert Williams was wrongfully arrested in Detroit in 2020 after police relied on a false facial recognition match. His case led to a 2024 settlement and stricter limits on Detroit police use of the technology. A facial recognition error in a phone gallery is an annoyance. In policing, it can cost a person their liberty.
In 2024, British Columbia’s Civil Resolution Tribunal held Air Canada responsible after its chatbot gave a passenger wrong information about bereavement fares. The tribunal rejected the airline’s suggestion that the chatbot was a separate legal entity. That principle will matter more as AI systems start speaking, advising, and acting for institutions.
The machine may make the mistake, but the institution must remain accountable. This is the line policymakers should hold. A citizen cannot cross-examine a neural network, and a patient cannot negotiate with a risk score. A soldier cannot ask a target-ranking model how sure it is.
Ethics and oversight
Even the companies building frontier AI seem to recognise that engineering alone may not solve the problem. Anthropic has reportedly consulted monks, theologians, and scholars across religious traditions while thinking about Claude’s ethical behaviour. It is an extraordinary image. After spending billions of dollars on computing, Anthropic is asking philosophers and religious thinkers how a machine should reason about virtue, responsibility and harm.
This is not proof that AI is conscious. It suggests that the people building these systems no longer treat ethics as a checklist. If AI becomes a moral adviser to millions, whose morality does it carry?
That matters because AI increasingly speaks with confidence. It can sound calm, balanced, and authoritative even when it is wrong. In a newsroom, that may produce a correction. In a hospital, a police station, a welfare office, or a war room, the consequences can be far more serious.
The phrase “human in the loop” is not enough. A human who sees only the model’s final recommendation, cannot inspect the evidence or is penalised for slowing the workflow is not in control. Real oversight means the power to question, pause, override, and demand an explanation.
In the IT industry, where AI replacing human workers is a real threat, everything from project estimation and cost-benefit analysis to resource allocation and testing is being planned around AI automation. The question is where the time and space to pause and recalibrate an AI system’s accuracy or judgment will come from when the entire programme depends on the quick turnaround of AI tools.
What is at stake for India
India is moving quickly towards AI in public services, healthcare, education, agriculture, finance, and defence. The ambition is correct. A country of India’s scale needs AI to improve delivery and expand access, but scale also magnifies errors. A flawed model in a pilot affects hundreds. A flawed model in an Indian public system can affect millions.
India should copy neither Western panic nor Western opacity. Its AI governance must be built around local realities, with multilingual testing, rural access, appeal mechanisms, data protection, explainability in public systems and independent audits before deployment in high-stakes areas.
For defence and critical infrastructure, the standard must be higher still. India needs sovereign AI capacity because dependence is risky, even where foreign models are useful. If a model influences military planning, cyber defence or emergency response, India must control the data, deployment environment, audit trail, and shutdown process.
The printing press, electricity, aviation, computers, the internet and digitisation all triggered anxiety. Some fears faded. Some became regulation. Some turned into daily reality. AI will follow the same pattern, with one difference. Earlier technologies extended human reach. AI extends human judgment.
AI is already shifting judgment from people to systems that are fast, fluent and hard to inspect: a fine here, a denied benefit there, a leaked customer list, a false police lead, a government portal entered by an agent that was never meant to be there.
The future will not be decided by whether we love or fear AI. It will be decided by whether we can govern it. And the most important question here is not whether AI can think. It is whether humans will keep thinking when AI offers them an answer.
Saikiran Kannan is an APAC-based independent journalist and an AI strategist.
Also Read | Our social media is highly politically controlled: Apar Gupta
Also Read | Why tech platforms should be regulated like high-risk industries
