Running Defender for Office 365 Attack Simulation Training: Payloads, Automations, and Repeat Offenders

Attack Simulation Training (AST) is the part of Defender for Office 365 Plan 2 that most teams turn on, run once for a compliance checkbox, and never operationalize. That is a waste. Run correctly, it gives you a measured, per-user click rate, a predicted-compromise model that tells you which simulations are realistic, and an automated loop that assigns just-in-time training to the people who actually fall for things. This is the playbook for running it as a program, not an event.

A few hard prerequisites before anything else. AST requires Defender for Office 365 Plan 2 (E5, the E5 Security add-on, or Microsoft 365 Business Premium, which includes a limited AST capability). The portal lives in the Microsoft Defender portal at Email & collaboration > Attack simulation training. There is no full Exchange Online PowerShell surface for authoring simulations — creation and reporting are driven from the portal and the Microsoft Graph security endpoints. We will use both, and I will be explicit about which is which so you do not go hunting for a cmdlet that does not exist.

1. Understand what the simulator measures (and what it does not)

A real phishing campaign measures whether an attacker got in. A simulation measures behavior under a controlled, first-party-delivered lure. The distinction matters because it changes how you read every number.

The core events AST records per recipient:

Event	What it means
Message sent / read	Delivery and open. Open tracking uses a tracking pixel and is noisy — treat read rate as soft.
Clicked link	User clicked the simulated URL. This is your primary signal for most techniques.
Supplied credentials	User typed into the harvest landing page. The strongest compromise signal.
Attachment opened / macro	For attachment and link-in-attachment techniques.
Reported (good)	User reported the message via the Report button. This is the metric you want to increase.
Out of office	Suppressed from results so OOF auto-reads do not pollute click data.

The single most important architectural fact: native AST messages bypass your own Defender filtering automatically. Microsoft injects them post-filter, so Safe Links does not rewrite the simulated URL and Safe Attachments does not detonate the payload. You do not need to build Advanced delivery (Set-PhishSimOverridePolicy) overrides for native AST — that mechanism is only for third-party phishing-sim vendors. If you find yourself adding sim-override entries for native AST, stop; you are masking a misconfiguration.

What it does not measure: lateral movement, MFA fatigue against your real Conditional Access, or whether a credential would actually have worked. AST harvests nothing usable — the landing page captures the fact of a submission, never the password.

2. Choose the technique deliberately

AST exposes five social-engineering techniques. Pick based on the behavior you are trying to measure, not on which looks scariest in a demo.

• Credential Harvest — link to a sign-in lookalike page. The default starting point; cleanest click-to-submit funnel. Best for baselining.
• Malware Attachment — an attachment with a (benign) payload that runs on open. Measures attachment-open discipline.
• Link in Attachment — a URL embedded inside an attached document. Defeats users who trust attachments more than inline links, and tests whether attachment-borne links get scrutiny.
• Link to Malware — a link to a file in a well-known service. Tests download-then-open behavior.
• Drive-by URL — a link that “compromises” on visit, no further interaction. Measures pure click discipline with no submit step.

There is also an OAuth Consent Grant technique, which presents a consent prompt for a malicious-looking app. This is the one most teams skip and the one most worth running, because illicit consent grants are a real, MFA-proof attack path. Note its quirk: it requires user interaction with a real consent dialog scoped to read-only permissions, so plan your messaging — a help-desk that does not know it is coming will generate tickets.

Rule of thumb: baseline the whole org with Credential Harvest, then rotate Link-in-Attachment and OAuth Consent for your higher-risk cohorts (finance, HR, IT admins). Do not run the most aggressive technique against everyone on day one — you will train people to fear the simulator, not to report.

3. Build payloads and landing pages

AST ships a large global payload catalog (graded by predicted compromise rate and tagged by theme), but the catalog is what attackers have already burned. The realistic lures are the ones that look like your tenant: your branding, your travel tool, your benefits portal.

In the portal: Attack simulation training > Content library > Payloads > Tenant payloads > Create a payload. The authoring flow:

1. Pick the technique (must match the simulation technique you will pair it with).
2. Choose theme and a sender (display name + from address). Use a name that is plausible but external — payroll-noreply@contoso-hr.com, not your real domain, so SPF/DMARC do not interfere and so the lure is honest about being external.
3. Author the HTML body. AST gives you dynamic tags — substitution tokens injected per recipient:

${userDisplayName}      First + last name of the target
${userFirstName}        First name only
${userEmailAddress}     Target SMTP address
${date|MM/dd/yyyy|+5}   Computed date, here today + 5 days
${managerDisplayName}   The target's manager (great for BEC-style lures)

${managerDisplayName} is the dangerous one — manager-impersonation lures consistently out-perform generic ones, which is exactly why you should measure your exposure to them.

1. Insert the phishing link placeholder; AST owns the URL and the click-tracking redirect. Pick the indicated phishing domain from the rotating Microsoft-owned set so the link is plausibly external.
2. For Credential Harvest, attach a login page. Author a tenant login page under Content library > Login pages that mirrors your real sign-in (logo, “Stay signed in?” prompt). Keep a generic Microsoft-style page for broad baselining and a branded one for targeted runs.

The payload editor previews how the message renders and flags missing required elements (a harvest payload with no phishing link will not save). Author once, reuse across simulations and automations.

4. Target selection, exclusions, and Safe Links

Targeting is where programs quietly go wrong. Three rules:

Scope by group, not by hand-picked list. Use a dynamic Entra ID group (department, role) or address-book filtering so the cohort stays current as people join and leave. Hand-curated CSVs rot within a quarter.

Exclude the people who must not be tested. The SOC analysts triaging the simulation, executive assistants who will escalate to legal, and any account in an active incident. Exclusions are set per-simulation under the target step.

Do not fight Safe Links. Because native AST is injected post-filter, Safe Links leaves the simulated URL alone — there is nothing to exclude. The mistake to avoid is the inverse: someone adds the AST phishing domains to a Tenant Allow/Block List allow entry or a transport-rule bypass “to make the sim work.” That permanently allow-lists Microsoft’s rotating sim domains for real mail too. Confirm you have no such entries:

Connect-ExchangeOnline -UserPrincipalName admin@contoso.onmicrosoft.com

# There should be NO manual allow entries for AST phishing domains.
Get-TenantAllowBlockListItems -ListType Url |
  Format-Table Value, Action, ExpirationDate

# Advanced delivery phishing-sim override is for THIRD-PARTY vendors only.
# It should be empty unless you also run KnowBe4/Proofpoint sims.
Get-PhishSimOverridePolicy | Format-List Name, Enabled
Get-PhishSimOverrideRule  | Format-Table Name, Domains, SenderIpRanges

If Get-PhishSimOverrideRule returns Microsoft’s own AST domains, someone has confused first-party and third-party simulation — remove them.

5. Simulation automations and payload harvesting

A one-off simulation is a snapshot. Simulation automations turn it into a moving baseline. Under Attack simulation training > Simulation automations > Create automation, you define a technique, a payload selection, a target group, and a schedule (randomized send windows over a date range so everyone does not get hit at 9:00 a.m. Monday and warn each other).

Two automation features that change the program economics:

• Randomized payload selection — the automation rotates across a set of payloads, so users cannot memorize “the phishing email.” This is the single biggest lever for keeping repeat-offender data honest over time.
• Send randomization across a window — spreads delivery over days/weeks. Critical above a few thousand users; a synchronized blast tanks your data the moment the first person posts in Teams.

The feature most teams miss is payload harvesting. Under Payload automations, AST can mine real, user-reported phishing mail in your tenant and turn the genuine lures your users are receiving into simulation payloads (stripped of live links/attachments). This is the closest thing to a self-updating, threat-intel-driven simulation program — you are testing people against what is actually landing in their inbox this month, not a 2019 template.

You can enumerate automations through Graph to keep an inventory in source control:

Connect-MgGraph -Scopes "AttackSimulation.Read.All"

# List configured simulation automations
Get-MgSecurityAttackSimulationAutomation |
  Format-Table DisplayName, Status, LastRunDateTime, NextRunDateTime

# List completed/running simulations and their high-level state
Get-MgSecurityAttackSimulation |
  Format-Table DisplayName, Status, AttackTechnique, LaunchDateTime

Treat the automation inventory as config: which cohort, which technique, what cadence. When a new business unit onboards, adding them to the dynamic target group is the only change required.

6. Assign training, track completion, and keep a control group

The simulation is the diagnostic; the training is the intervention. AST’s value is that it assigns training conditionally based on the simulated outcome.

In the simulation’s training step, choose Assign training based on user’s risk (Microsoft picks modules from the content library matched to who clicked or submitted) or curate a specific course set. Set a due date — without one, completion drifts toward zero. AST sends the training notification, reminders, and a completion notice from configurable end-user notification templates (use the Microsoft default or author tenant templates with your security team’s branding).

The deliberately counterintuitive move: keep a no-training control group. AST lets you carve out a slice of the population that gets simulated but receives no assigned training. Over two or three cycles, the gap between the trained and untrained cohorts’ click rates is the only honest measure of whether your training is doing anything. If the control group’s click rate falls as fast as the trained group’s, your improvement is regression to the mean or awareness of the program — not the courseware — and you should change the training, not celebrate.

Track completion through the Simulations > [simulation] > Training assignments view, or pull user-level coverage via Graph (security/attackSimulation exposes training events and the operation/user reports). Wire the completion data into your existing LMS or compliance dashboard rather than letting it live only in the Defender portal.

7. Read the reports: predicted compromise rate and repeat offenders

This is where the program earns its keep. Three numbers matter.

Predicted Compromise Rate (PCR). AST models, per payload, the percentage of similar users likely to be compromised by it, learned from anonymized cross-tenant signal. PCR is how you tell a realistic simulation from an unfair one. If your actual click rate massively exceeds PCR, the lure was too easy for your population (or too obviously internal); if it is far below, the lure may have been implausible. Use PCR to grade payloads, not just users — a good program retires payloads whose PCR no longer matches reality.

Actual click / compromise rate vs. the org benchmark. The simulation report shows your rate against an anonymized benchmark of comparable organizations. A rate above benchmark for a common technique is your headline finding for leadership.

Repeat offenders. The portal surfaces users compromised across multiple simulations. This is the population that matters: a one-time clicker is noise; someone who submits credentials in three consecutive quarters is a standing risk that training alone has failed to fix. Build the repeat-offender cohort as a dynamic membership target and escalate it — more frequent simulations, mandatory live training, and (section 8) tighter Conditional Access.

You can pull the recipient-level detail from Graph for your own reporting pipeline:

Connect-MgGraph -Scopes "AttackSimulation.Read.All"

# Per-user outcomes for a given simulation (clicked, submitted, reported, etc.)
$sim = (Get-MgSecurityAttackSimulation |
  Where-Object DisplayName -eq "Q2 Credential Harvest - All Staff")

Invoke-MgGraphRequest -Method GET `
  -Uri "https://graph.microsoft.com/v1.0/security/attackSimulation/simulations/$($sim.Id)/report/simulationUsers" |
  Select-Object -ExpandProperty value |
  ForEach-Object {
    [pscustomobject]@{
      User        = $_.simulationUser.email
      ClickCount  = $_.clickCount
      Submissions = $_.compromisedCount
      Reported    = $_.reportedCount
    }
  } | Sort-Object Submissions -Descending

Export that on a schedule, join it across simulations on UPN, and count() > N gives you your repeat-offender list as data you control rather than a portal screenshot.

8. Close the loop into Conditional Access and SecOps

A click rate that does not change anything is theater. Two real closures:

Feed repeat offenders into risk-based Conditional Access. AST does not write to Entra ID risk, so do the join yourself. Maintain the repeat-offender cohort as an Entra security group (populated from your exported Graph data) and target it with a stricter CA policy — for example, require phishing-resistant MFA and block legacy auth for that group, and tighten session controls. You are not punishing them; you are reducing the blast radius of the next successful lure against a known-susceptible population.

# Add this period's repeat offenders to the high-risk CA target group.
Connect-MgGraph -Scopes "GroupMember.ReadWrite.All","User.Read.All"

$groupId = (Get-MgGroup -Filter "displayName eq 'Phishing-Repeat-Offenders'").Id
$repeatOffenders = Import-Csv ./repeat-offenders.csv     # UPN column from section 7

foreach ($u in $repeatOffenders) {
  $user = Get-MgUser -UserId $u.UPN -ErrorAction SilentlyContinue
  if ($user) {
    New-MgGroupMember -GroupId $groupId -DirectoryObjectId $user.Id -ErrorAction SilentlyContinue
  }
}

Point a Conditional Access policy at Phishing-Repeat-Offenders requiring an authentication strength of phishing-resistant MFA. Re-evaluate membership each cycle and remove users who go a few rounds clean.

Treat real reports as the win condition. The metric that actually correlates with resilience is the report rate, not the click rate. Make sure the Report button (the built-in Microsoft report or your reporting mailbox) is deployed, and ensure user-reported messages flow into Submissions and your SOC queue. AST and real reporting share the same button; a healthy program shows click rate falling and report rate rising over successive cycles. If only one moves, you are optimizing the wrong behavior.

Verify

Confirm the program is wired correctly and the data plumbing works end to end:

# 1. License/feature check: AST requires Defender for O365 Plan 2.
#    (Confirm in Defender portal > Settings, or via your licensing report.)

# 2. No accidental allow-listing of AST or sim domains for REAL mail.
Get-TenantAllowBlockListItems -ListType Url |
  Format-Table Value, Action, ExpirationDate
Get-PhishSimOverrideRule | Format-Table Name, Domains, SenderIpRanges  # expect empty for native AST

# 3. Automations exist, are enabled, and have a sane next-run.
Connect-MgGraph -Scopes "AttackSimulation.Read.All"
Get-MgSecurityAttackSimulationAutomation |
  Format-Table DisplayName, Status, LastRunDateTime, NextRunDateTime

# 4. A completed simulation returns per-user report data.
Get-MgSecurityAttackSimulation |
  Format-Table DisplayName, Status, AttackTechnique, LaunchDateTime

# 5. Repeat-offender group is populated and targeted by a CA policy.
Connect-MgGraph -Scopes "GroupMember.Read.All"
$g = Get-MgGroup -Filter "displayName eq 'Phishing-Repeat-Offenders'"
(Get-MgGroupMember -GroupId $g.Id).Count

For a true end-to-end test, target a single pilot mailbox with a Credential Harvest simulation, click the link and submit dummy text on the landing page, and confirm the per-user report shows the click and submission and that the assigned training notification arrives. Then confirm that an actual reported message (use the Report button on a benign test) lands in your SOC submissions queue — the two paths share the same button and both must work.

Enterprise scenario

A platform team at a ~9,000-seat financial-services firm ran a single org-wide Credential Harvest simulation, recorded an 18 percent click rate, assigned training to everyone, and called it done. The CISO asked the only question that mattered: did it work? They had no answer, because they had trained the entire population at once and had nothing to compare against.

The constraint was twofold. First, no control group meant any future improvement would be unattributable. Second, the security operations cohort had clicked the simulation too — and because someone had “fixed” the original delivery by adding the AST phishing domains to a Tenant Allow/Block List allow entry, those domains were now allow-listed for real inbound mail across the tenant. They had widened their attack surface to run an awareness exercise.

The fix was to rebuild it as a program. They removed the TABL allow entries (native AST needs none) and confirmed Defender filtering was intact. They replaced the one-shot blast with a simulation automation on a randomized cadence, rotating payloads — including ones harvested from real user-reported phish — so the lure changed each cycle. They carved out a 10 percent no-training control group and excluded the SOC. Repeat offenders were pushed into a dynamic Entra group fed by exported Graph report data, and that group was targeted by a Conditional Access policy requiring phishing-resistant MFA.

The result over three quarters: the trained population’s submission rate fell from 11 percent to 3 percent while the control group barely moved, proving the training — not awareness of the program — was responsible. Report rate climbed from 6 percent to 31 percent. The repeat-offender group shrank to under 40 users, each now behind stronger CA. The recurring config that made it durable:

# Quarterly: refresh the repeat-offender cohort, expire the clean ones.
$groupId = (Get-MgGroup -Filter "displayName eq 'Phishing-Repeat-Offenders'").Id
$current = Get-MgGroupMember -GroupId $groupId
$offenders = Import-Csv ./repeat-offenders-this-quarter.csv   # 2+ compromises across sims

# Remove members who are no longer repeat offenders this quarter
foreach ($m in $current) {
  $upn = (Get-MgUser -UserId $m.Id).UserPrincipalName
  if ($offenders.UPN -notcontains $upn) {
    Remove-MgGroupMemberByRef -GroupId $groupId -DirectoryObjectId $m.Id
  }
}

The lesson: AST’s value is not the simulation, it is the comparison. Without a control group and a closed loop into CA, you are buying a number, not a reduction.

Checklist

• Defender for Office 365 Plan 2 licensing confirmed for the target population
• No TABL allow entries or transport bypasses for native AST domains (none are needed)
• Advanced delivery phishing-sim override reserved for third-party vendors only
• Tenant payloads authored with your branding and dynamic tags; login pages mirror real sign-in
• Technique matched to behavior; org baselined on Credential Harvest before aggressive techniques
• OAuth Consent Grant simulation run against high-risk cohorts with help-desk forewarned
• Targeting by dynamic Entra group; SOC and active-incident accounts excluded
• Simulation automation with randomized payloads and a randomized send window
• Payload harvesting from real reported mail feeding the automation
• Risk-based training assigned with a due date; reminders and completion tracked
• No-training control group maintained for honest before/after measurement
• Predicted Compromise Rate used to grade payloads, not just users
• Repeat-offender cohort exported via Graph and fed into stricter Conditional Access
• Report rate tracked as the primary success metric (recurring per-cycle review)
• Repeat-offender CA group re-evaluated each cycle; clean users removed

Pitfalls

The failure modes repeat across every tenant. Allow-listing the sim domains to “make delivery work” is the worst — native AST never needs it, and the entry quietly weakens real mail filtering. One-shot org-wide blasts with no control group produce a number nobody can act on. Static payloads train users to recognize one email, not phishing; rotate them or harvest from real reports. Optimizing click rate alone ignores the metric that actually predicts resilience — the report rate. And assigning training with no due date guarantees single-digit completion.

Treat AST as a measurement system with a closed loop: simulate against realistic, rotating lures; assign training conditionally; keep a control group so you can prove impact; and feed your repeat offenders into Conditional Access so a known-susceptible population carries less blast radius. Source-control your automation inventory and your Graph export pipeline, review report rate every cycle, and let the data — not a single annual exercise — drive the program.