Chapter 4: AI and Security

AI and Security 🤖

Treat every input as untrusted.

A few years ago, an AI chatbot could only talk. You typed a question, it typed an answer, and a human decided what to do next. Today’s AI systems increasingly act. They read your email, browse websites, write and run code, call other software and hand tasks to other AI systems, often with only light human supervision. That shift from talking to doing is the biggest change in cybersecurity since the move to the cloud.

AI cuts both ways. Attackers use it to write convincing scams in any language and to automate parts of intrusions. Defenders use it to sort alerts, find bugs and respond faster. And AI systems themselves are a new kind of target, with weaknesses unlike anything in traditional software. This chapter covers all three sides: the changing threat landscape, how to secure AI agents, the prompt injection problem, the rise of synthetic media, and how organizations and governments are trying to keep up.

4.1 — Agentic AI and the Threat Landscape 🤖

An AI agent is an AI system that works toward a goal by planning steps and using tools. A typical agent combines a large language model (the “brain” that reads and writes text), instructions from its developer, a set of tools it may call (search, email, a database, a code runner), and some memory of what it has done. It runs in a loop: look at the situation, decide on an action, take it, observe the result, repeat. The loop is what makes agents useful, and also what makes them risky: an agent can take dozens of actions before any person looks at the result.

How Agentic AI Helps Attackers

AI does not give attackers new goals; it changes their speed, scale and cost.

Social engineering at scale: phishing messages no longer give themselves away with clumsy grammar. AI can write fluent, personalized messages in any language and clone a voice from a short recording.
Faster reconnaissance: agents can gather and summarize public information about an organization and its staff in minutes.
Lower skill barrier: less experienced attackers can get help writing scripts or understanding systems, although reputable AI providers try to block clearly malicious requests and monitor for misuse.
Automation of whole attack stages: in November 2025, Anthropic reported that it had detected and disrupted a cyber-espionage campaign, which it attributed to a Chinese state-sponsored group, in which attackers used its AI coding tool to carry out much of the intrusion work automatically, with humans stepping in mainly at key decision points.

How Agentic AI Helps Defenders

The same capabilities strengthen defense. Security teams use AI assistants to summarize alerts, explain suspicious scripts, draft incident reports and write detection rules, which helps with the alert fatigue described in Chapter 3. AI is also finding vulnerabilities before attackers do. In 2024, Google reported that its AI agent Big Sleep found a previously unknown, exploitable flaw in the widely used SQLite database software, which was fixed before it appeared in an official release. In August 2025, the U.S. Defense Advanced Research Projects Agency (DARPA) held the final round of its AI Cyber Challenge, in which teams’ autonomous systems found and patched vulnerabilities in real open-source software, and the competition required finalists to release their systems as open source so that anyone can use them.

Evaluating the New Landscape

Across the network threat landscape, autonomous agents matter in three ways at once. They are attack tools that compress the time between a vulnerability’s discovery and its exploitation. They are defense tools that can patch and respond at machine speed. And they are targets: every agent connected to email, files and networks is a new privileged identity that can be tricked or hijacked. A balanced evaluation avoids both panic and hype. Most AI-assisted attacks still rely on the same old weaknesses (unpatched systems, stolen passwords, missing MFA), so fundamentals still stop most of them. What changes is that defenders have less time, and that agents themselves must now be secured like any other powerful user.

4.2 — Securing AI Agents 🧩

An AI agent is software with a job, credentials and access to data, so it deserves the same scrutiny as a new employee with administrator rights. Securing one starts with understanding its architecture.

The Parts of an Agent System

Component	What it does	Security question
Model	Reads input and decides on actions	Can it be tricked into the wrong decision?
System instructions	The developer’s rules for the agent	Do they leak? Can user input override them?
Tools and connectors	Email, files, databases, web, code execution	Which tools does it really need? With what permissions?
Memory and data stores	Past conversations, documents, search indexes	Can untrusted content be planted there?
Orchestrator	Coordinates steps and, in multi-agent systems, other agents	How do agents prove who they are to each other?
Logs	Records of prompts, actions and results	Can every action be traced and reviewed?

Many agents now connect to tools through standard protocols such as the Model Context Protocol (MCP), an open standard introduced by Anthropic in 2024 and since adopted widely. Standards make connections easier, but every connected tool server is a trust boundary: an agent is only as safe as the least trustworthy tool it can reach.

Identity and Access for Agents

Each agent should have its own non-human identity, never a borrowed human password. Good practice applies the access control ideas from Chapter 2:

Least privilege: an agent that summarizes tickets needs read access to tickets, not to the payroll system.
Scoped, short-lived credentials: tokens that expire in minutes or hours and work only for specific actions, issued through standards such as OAuth.
Delegated authority: when an agent acts for a user, it should never get more access than that user has.
Authenticated hops: in multi-agent workflows, every agent verifies every request from every other agent. Otherwise a compromised low-level agent can ask a powerful one to act for it, a classic “confused deputy” problem.
human-in-the-loop approval for high-impact or irreversible actions: sending money, deleting data, emailing outsiders, changing permissions.

Running a Security Evaluation

A security evaluation of an agent works through a checklist. Inventory every tool and permission. Identify every source of input, especially content from outside the organization. Check how the agent authenticates to each tool and to other agents. Confirm that actions are logged and that a person can pause or shut it off instantly (a kill switch). Test, with authorization, how it behaves when given misleading input. Rate each finding by likelihood and impact, just as you would for any other system.

Designing a Secure Workflow

Picture a school help desk that wants an agent to read incoming support tickets, suggest fixes and draft replies. A secure design gives the agent read-only access to the ticket queue and the public knowledge base, no access to student records, and no ability to send email; it writes drafts that a technician reviews and sends. It runs under its own identity with short-lived tokens, every action is logged, and anything it reads from a ticket is treated as data, never as instructions. The agent still saves technicians time, and the worst case of a failure is a bad draft that a human catches.

4.3 — Prompt Injection 💉

Traditional software keeps code and data separate. A spreadsheet does not run the words typed into its cells. Large language models have no such wall: instructions and data arrive as the same stream of text, and the model has to guess which is which. prompt injection exploits that confusion.

Direct and Indirect

In direct prompt injection, the user types instructions meant to override the system’s rules, such as telling a customer-service bot to ignore its policy. The damage is usually limited to what that user could already see. The more serious variant is indirect prompt injection, where the instructions are hidden in content the AI reads while doing a job for someone else: a web page it summarizes, an email in an inbox it manages, a shared document, a calendar invite, a code comment or text inside an image. The user never sees the hidden instruction, but the model may follow it.

How It Exploits Networked Applications

The danger grows with an application’s connections. Imagine an AI email assistant that can read the inbox and send messages. An outsider sends an email containing hidden text that tells any AI reading it to search the inbox for password-reset messages and forward them to an outside address. If the assistant obeys, an attacker has stolen data without touching the network directly. Developer and researcher Simon Willison has called this combination the “lethal trifecta”: access to private data, exposure to untrusted content and the ability to send information out. When an agent has all three, a single poisoned input can lead to data theft.

Validating a Scenario

To validate whether a scenario is vulnerable, trace the path on paper. List every source of untrusted text the AI can read. List every action it can take. Draw a line from each untrusted source to each sensitive action that could happen without a human check. Any complete line is a vulnerability. Fixes break the lines: remove an unneeded tool, separate the agent that reads untrusted content from the agent that holds private data, require human approval before anything leaves the organization, and block the AI from embedding outside links or images that could quietly carry data away. No model-level filter is reliable enough on its own, which is why the design matters more than the wording of the instructions.

Vulnerability Frameworks for AI Systems

Security teams use frameworks to evaluate AI components systematically. The OWASP Top 10 for Large Language Model Applications (2025 edition) ranks prompt injection first. Several of its entries apply directly to networked systems:

OWASP LLM risk	What it means in a networked system
Prompt injection	Inputs from users or outside content alter the model’s behavior
Sensitive information disclosure	The model reveals private data or secrets it can access
Supply chain	Compromised models, datasets, plugins or tool servers
Improper output handling	Model output passed unchecked into browsers, databases or commands
Excessive agency	The model has more tools, permissions or autonomy than it needs
System prompt leakage	Hidden instructions, sometimes containing secrets, are exposed

MITRE ATLAS catalogs real adversary techniques against AI systems in the style of MITRE ATT&CK, and the NIST AI Risk Management Framework helps organizations govern AI risk overall. Applying a framework means walking through each entry for your specific system and recording whether it applies and what control addresses it.

4.4 — Synthetic Media Threats 🎬

A deepfake is AI-made audio, video or imagery that convincingly shows a real person doing or saying something they never did. Synthetic media has creative and helpful uses, from film effects to giving people who have lost their voice a way to speak. As a security threat, it is a powerful upgrade to social engineering.

Synthetic Media as an Attack Vector

Social engineering and fraud: in early 2024, the engineering firm Arup confirmed that an employee in its Hong Kong office had transferred about US$25 million after a video call in which the company’s chief financial officer and other colleagues were AI-generated fakes.
Credential fraud: cloned voices can be used to call help desks and request password resets, and fake identity documents or synthetic selfies can target online identity checks.
Disinformation: in January 2024, a robocall with an AI-generated imitation of President Biden’s voice urged New Hampshire voters to skip the primary election. The Federal Communications Commission then ruled that AI-generated voices in robocalls fall under existing federal robocall restrictions.

An organizational policy against deepfake-based attacks focuses on process rather than spotting fakes by eye: verify any request for money, credentials or sensitive data through a separate, known channel; require two people to approve payments; use pre-agreed code words for urgent requests; train staff with real examples; and include deepfakes in the incident response plan, with a clear way to report and a commitment to support victims.

Why Detection Is Not Enough

It is tempting to rely on deepfake detectors, but their limits are well documented. Detectors trained on one generation method often fail on newer ones. Compressing, cropping or re-recording a video can strip the clues detectors look for. Invisible watermarks can be damaged or removed. In 2023, OpenAI withdrew its own tool for identifying AI-written text, citing its low accuracy. Human judges do not do much better, especially with short clips or audio. A false “real” verdict gives a fake extra credibility; a false “fake” verdict can discredit a real victim.

The more promising approach is provenance: proving where genuine media came from rather than proving what is fake. The C2PA standard attaches content credentials to files, cryptographically signed at capture or creation, so viewers can check a file’s origin and edit history. Credentials only help when cameras, apps and platforms support them, and when they keep them attached.

Policy, Law, and Society

At scale, synthetic media raises policy questions. In the United States, the TAKE IT DOWN Act of 2025 made it a federal crime to knowingly publish nonconsensual intimate images, including AI-generated ones, and requires platforms to remove them within 48 hours of a valid request, a duty that took effect in May 2026. The European Union’s AI Act requires deepfakes to be disclosed as artificially generated, with its transparency rules phasing in from 2026, and many U.S. states have passed laws on deceptive election deepfakes. A societal side effect is the liar’s dividend: once fakes are common, people caught on genuine recordings can claim the evidence is fake.

Creating or sharing sexual or humiliating deepfakes of real people, including classmates, is abuse and can be a crime. If it happens to you or someone you know, save evidence, do not reshare it, and report it to a trusted adult, the school and the platform.

A Community Response Framework

Schools, newsrooms, companies and towns need a plan before disinformation strikes. A strong framework has five parts: prepare (designate spokespeople, set up content credentials for official media, teach media literacy), verify (check sources and provenance before reacting), respond (publish a clear correction through trusted channels quickly), support (protect and help the people targeted) and review (learn and update the plan).

4.5 — Shadow AI and Governance 📋

shadow AI is the use of AI tools without the organization’s approval: a teacher pasting student essays into a free chatbot, an employee installing an AI browser extension that can read every page. It usually comes from good intentions, people trying to work faster, but it creates real risk. In 2023, Samsung restricted employee use of public generative AI tools after reports that engineers had pasted confidential source code into one.

The Risks

Data leakage: whatever is typed or uploaded leaves the organization and may be stored or, under some terms, used to train future models.
Compliance: student records are protected by privacy laws such as FERPA, and districts must control who processes them.
Unvetted output: AI-generated code or answers may contain errors or vulnerabilities.
Excessive permissions: AI extensions and plugins often request access to email, files or every website visited.
Age terms: many AI services require users to be at least 13, some require 18, and others need parental consent, so schools must check terms before approving tools for students.

Writing Governance Guidelines

Banning AI outright usually just drives it underground. Good governance makes the safe path the easy path:

Approved tools list: vetted services with contracts that protect data, available through single sign-on.
data classification: for example, Public data may go into any approved tool; Internal data only into enterprise tools with data protections; Confidential and Restricted data, such as student records and health information, never into general AI tools.
Request and review process: a quick way for anyone to ask for a new tool, with a security and privacy review.
Monitoring: network and browser controls that detect unapproved AI services and extensions.
Security awareness training: short, practical lessons on what may be shared, how to spot AI-powered scams, and how to report mistakes without fear.

Guidelines should be short enough that people actually read them, reviewed at least yearly because AI tools change quickly, and written with the people who will follow them. Teachers and students who help write the rules understand why they exist and are far more likely to use the approved path.

Comparing Global Policies

Governments are taking different approaches to AI, data and cyber defense:

Region	Approach to AI	Data and cyber rules
European Union	The AI Act (in force since 2024, phased in over several years) classifies AI by risk and bans some uses	GDPR governs personal data; the NIS2 directive sets cybersecurity duties for important sectors
United States	No single federal AI law; voluntary NIST frameworks, sector regulators, executive orders that change between administrations, and growing state laws	Sector privacy laws (health, education, children) and state privacy and breach laws
China	Specific rules for generative AI services and mandatory labeling of AI-generated content	Strict data security and cybersecurity laws, including limits on moving data abroad
United Kingdom	A sector-by-sector approach and a government AI Security Institute that tests advanced models	UK GDPR and data protection law

These choices reflect different balances between innovation, individual rights and state security. All major powers also use AI for cyber defense and intelligence, which raises ongoing debates about surveillance, oversight and how data collected for security may be used.

Chapter 4 Activity: The AI Security Review 🧩

Your team is the AI security committee for Summit Academy, a fictional high school that wants to adopt AI tools safely. Work on paper or in district-approved tools; do not test any real AI product.

Part 1: Evaluate the Agent

Summit’s office wants an AI agent that reads the main inbox, answers routine questions, schedules meetings and forwards urgent messages to staff. Using the component table and evaluation checklist from section 4.2, administer a security evaluation: list its tools, permissions, identities and input sources, and rate at least five risks by likelihood and impact.

Part 2: Validate the Injection

Your teacher will provide three sample emails for the agent’s inbox, one containing hidden instructions. Trace every path from untrusted input to sensitive action, decide whether the scenario is vulnerable to indirect prompt injection, and identify which OWASP LLM risks apply. Then redesign the workflow so every dangerous path is broken, using least privilege, non-human identities and human-in-the-loop approval.

Part 3: The Deepfake Plan

Write Summit’s response policy for deepfake-based social engineering and disinformation: how staff verify unusual requests, how a fake video of the principal would be handled within the first hour, how targeted students or staff would be supported, and why the school will not rely on detection tools alone. Defend your framework to another team, which will try to find gaps.

Part 4: Shadow AI Guidelines

Draft Summit’s AI governance guidelines: an approved tools process, a four-level data classification with examples from a school, age-appropriate rules for students under 13, 13 to 17 and staff, and a ten-minute awareness training outline. Close with one paragraph comparing how the EU and U.S. approaches would affect your guidelines.