Chapter 4: AI, Agents, and New Threats

AI, Agents, and New Threats 🤖

Verify the source, then trust the content

Artificial intelligence can now write an essay, answer a question in a friendly voice, make a picture of something that never happened, and even take actions on its own, like sorting email or booking a meeting. Those abilities are genuinely useful. They also hand new tools to scammers, and they create brand-new kinds of weaknesses.

This chapter explains how AI agents are built, so you can see why the same design helps both defenders and attackers. You’ll learn how AI makes old tricks more convincing, how hidden instructions can hijack an AI tool, how to weigh evidence about a suspicious video, and why using unapproved AI tools can leak private data. The goal isn’t to fear AI. It’s to use it with your eyes open.

Use only AI tools your school has approved, and follow your teacher’s directions. Many AI services require users to be 13 or older, so younger students should use them only through teacher-led school accounts.

4.1 — How AI Agents Work 🤖

Most AI you’ve seen is a chatbot: you type, it answers. An AI agent goes further. Give it a goal, and it plans steps and carries them out. Many agents are built around a large language model (LLM), the kind of AI that reads and writes language.

The Four Components

Every AI agent, simple or advanced, has four parts working together:

Component	What It Does	Example: A Homework Planner Agent
Information	The inputs it receives	Your assignment list and class calendar
Decision process	How it reasons and plans next steps	Decides what’s due first and how long each task takes
Context or memory	What it remembers from earlier	Remembers you work slowly on essays
Actions or tools	What it can actually do	Adds study blocks to your calendar, sends you a reminder

The fourth component is what makes agents powerful, and risky. An AI that can only talk can give bad advice. An AI that can act can send, delete, buy, or share things.

Same Architecture, Two Sides

Because the four components are neutral, they can be pointed at defense or offense.

Component	Defensive Agent	Offensive Agent
Information	Reads security alerts and login records	Gathers public information about targets
Decision process	Decides whether activity looks like an attack	Decides which message will most likely fool each person
Context or memory	Remembers what normal activity looks like	Remembers which targets replied
Actions or tools	Blocks a suspicious login, opens a help ticket, alerts a person	Sends messages, creates fake accounts

A defensive agent can watch thousands of events per second and respond in moments, something no human team could do alone. An offensive agent can do the same thing for an attacker. That’s why defenders study how agents work: to use them well, and to recognize them when they’re used against us.

4.2 — AI-Powered Attacks 🎣

Most cyberattacks still begin with social engineering: tricking a person instead of breaking a machine. AI doesn’t invent new tricks so much as it makes old ones faster, cheaper, and more convincing.

What AI Changes

Attack Step	Before AI	With AI Agents
Research	An attacker reads profiles one at a time	An agent gathers public details about hundreds of people quickly
Writing	Scam messages often had spelling mistakes	Messages are polished, personal, and written in any language
Voice and video	Hard to fake	A short clip of someone’s voice can be used to clone it
Conversation	A person had to answer replies	A chatbot can keep a scam conversation going around the clock
Scale	Limited by the attacker’s time	Thousands of tailored messages at once

The old advice “look for bad spelling” doesn’t work anymore. The new advice focuses on what the message asks for: urgency, secrecy, money, passwords, gift cards, or a change in payment details.

Real-World Examples

These are documented cases and warnings:

Individuals: In 2023, the U.S. Federal Trade Commission warned families about scammers using AI voice cloning to imitate a relative in an “emergency” and ask for money.
A school: In 2024, an audio clip spread online that seemed to be the principal of Pikesville High School in Maryland making hateful remarks. Police found it had been made with AI, and the school’s athletic director was arrested and charged. The principal and the school community were harmed by a lie that sounded real.
An organization: In 2024, an employee at the engineering firm Arup in Hong Kong joined a video call with what appeared to be senior colleagues. They were deepfakes, and the employee transferred about 25 million U.S. dollars to criminals.
Government: In 2025, the FBI warned that criminals were sending AI-generated voice messages pretending to be senior U.S. officials.

Defending Against AI-Powered Scams

Verify through a different channel. If a message claims to be from your coach, call or ask them in person using contact information you already have.
Slow down. Urgency is the attacker’s favorite tool.
Use a family code word for real emergencies.
Report it. Tell a teacher, a parent, or your school’s tech team. Reports protect everyone else.

4.3 — Prompt Injection 💉

An LLM follows instructions written in plain language. That’s what makes it easy to use. It’s also its biggest weakness: the model can’t reliably tell the difference between instructions from its owner and text it’s only supposed to read.

prompt injection takes advantage of that. Security experts rank it as one of the top risks for AI applications.

Two Kinds

Direct prompt injection: a user types instructions meant to make the AI break its own rules, such as telling it to ignore its earlier instructions.
Indirect prompt injection: the instructions are hidden inside content the AI reads on someone’s behalf, such as a web page, an email, or a shared document. The person using the AI never sees them.

Indirect injection is especially dangerous for agents, because agents read untrusted content and have tools.

Scenario Analysis: The Inbox Helper

A company gives employees an AI agent that summarizes their email and can send replies. An attacker emails an employee a message with hidden text telling the AI to forward the employee’s recent messages to an outside address. When the employee asks for a summary, the agent reads the hidden text and follows it.

Which security controls failed?

Failure	Why It Mattered
Too much access	A summarizing tool didn’t need permission to send email
No human approval	The agent sent messages without asking the employee
Untrusted input treated as instructions	Email from strangers was handled like a command from the owner
No monitoring	Nobody noticed messages leaving to a new address

Recommended Controls

No single fix stops prompt injection, so defenders stack several:

Least privilege for tools. Give an agent only the tools its job needs. A summarizer reads; it doesn’t send.
Human in the loop. Require a person to approve risky actions like sending, deleting, sharing, or paying.
Separate instructions from data. Mark outside content as untrusted so the system treats it as material to read, never as commands.
Filter inputs and outputs. Scan for hidden text and block outputs that contain private data or new destinations.
Log and monitor. Record what the agent did so unusual actions are caught quickly.

4.4 — Deepfakes and Synthetic Media 🎬

synthetic media is content made or heavily changed by AI. A deepfake is synthetic media that makes a real person seem to say or do something they never did.

How Synthetic Media Creates Risk

Use	Example	Risk
Impersonation	A cloned voice of a boss asks for a payment	Fraud; security checks fooled
Misinformation	A fake image of a disaster spreads during a storm	Panic; people make bad decisions
Harassment	Fake images of a classmate	Serious harm to a real person
Doubt	A real video is dismissed as “probably AI”	Truth becomes easier to deny

That last risk has a name: the liar’s dividend. When everyone knows fakes exist, people who are caught on real recordings can claim the evidence is fake. Synthetic media threatens not only what we believe, but whether we can believe anything at all.

Weighing the Evidence

No single test proves a video is real or fake, and detection tools make mistakes. Investigators weigh several kinds of evidence:

Source: Who posted it first? Is it from an account with a history, or a brand-new one?
Provenance: Some cameras and editing apps attach Content Credentials, a record of where media came from and how it was changed. Missing credentials don’t prove a fake, but present ones help.
Context: Are trusted news organizations reporting the same event? Does a reverse image search show the picture years earlier, somewhere else?
Technical clues: Odd hands or teeth, blurry edges around a face, lip movements that don’t match words, flat or robotic voices, text that turns to nonsense. These clues are getting harder to spot as tools improve.
Behavior: Does the person in the video act or ask for things that are out of character?

Synthetic media isn’t automatically bad. It dubs films into other languages, helps people who’ve lost their voice to illness speak in their own voice again, and powers creative projects. The ethical questions are about consent, honesty, and harm: Did the real person agree? Is it labeled? Could it hurt someone? Laws are catching up. In 2025, the federal TAKE IT DOWN Act made it a crime to publish private intimate images of real people without consent, including AI-made fakes, and requires platforms to remove them quickly when a victim asks.

4.5 — Shadow AI and Automated Defense 🌑

shadow AI means using AI tools your school or workplace hasn’t approved. It usually comes from good intentions: someone wants to work faster. But anything typed into an unapproved tool may be stored, reviewed by the company, or used to train future models, and the school has no agreement protecting it.

Data Leakage and Governance Failure

In 2023, engineers at Samsung pasted confidential source code into a public AI chatbot while trying to fix problems. The company then restricted employee use of outside AI tools. Nobody was attacked, yet private data left the company. That’s data leakage. It’s also a governance failure: the organization didn’t know which tools people used or where its data went, so it couldn’t keep its own promises about protecting it. For a school, that could mean breaking privacy laws like FERPA.

Choosing the Right Control

Say a student pastes a classmate’s name, grades, and a teacher’s comments into an unapproved AI app to “rewrite a report.” Possible controls include:

Control	How It Helps	Limit
Provide an approved AI tool	Gives people a safe option with data protections	Only works if people know about it
Clear policy and training	Explains what never goes into any AI tool	People forget or ignore rules
Data classification	Labels what’s private, like grades	Needs everyone to apply the labels
Technical blocking and data loss prevention	Blocks unapproved sites or sensitive data	Can be bypassed on personal devices

For this incident, the most effective first step is usually an approved tool plus training, because blocking alone pushes people to personal phones. The best answer always depends on the specific incident.

Automation That Defends

Automation also works for defenders. Spam filters block millions of phishing emails, systems install patches overnight, and security software can automatically isolate a laptop that starts acting like ransomware. Automation can create risk too: a rule that’s too strict can lock out every real user, and an over-powered agent can delete the wrong files.

Here’s a simulated log from a school network. Which actions are defensive and which are offensive?

Time	Agent	Action
02:14	Unknown script	Tried 400 passwords on staff accounts
02:15	Login protection	Locked the targeted accounts and alerted IT
02:40	Unknown device	Scanned the network for computers missing updates
03:00	Update service	Installed security patches on 120 laptops

The unknown script and the scanning device are offensive; the login protection and update service are defensive. Watching logs like this is how security teams spot attacks in progress.

Chapter 4 Activity: AI Threat Response Lab 🧪

Your Mission: Run a paper-based security lab on AI threats. Use only school-approved tools, with your teacher leading any use of AI.

Part 1: Build an Agent on Paper

Design a helpful AI agent for your school (a library helper, a lunch-line scheduler). List its four components. Then write how an attacker could misuse the same design, and which tool you would remove or limit.

Part 2: Scam or Not?

Your teacher will share sample messages. For each, list what it asks for, the pressure it uses, and how you would verify it through a separate channel.

Part 3: Injection Investigation

For the inbox helper in section 4.3, or a new scenario your teacher provides, identify every failed control and recommend three controls in priority order. Explain your top choice.

Part 4: Evidence Board

Examine two images or short clips your teacher provides. Fill in an evidence table (source, provenance, context, technical clues, behavior) and give a verdict: likely real, likely synthetic, or not enough evidence. Finish with one sentence on the ethical issue each raises.

Key Concepts Checklist

I can explain the four components of an AI agent
I can show how the same agent design supports defense and attack
I can explain how AI makes attacks more convincing and describe real examples
I can identify control failures and evaluate prompt injection in a scenario
I can recommend controls that reduce prompt injection risk
I can analyze how deepfakes enable impersonation and misinformation
I can evaluate evidence about whether media is real or synthetic
I can explain shadow AI risks and tell defensive from offensive automation in a log