Building an AI SOC, Part 1: From Software Factories to Investigations

First in a series about building, testing, and questioning an AI-assisted SOC.

An alert arrives. Someone finds the procedure, collects the evidence, checks the context, writes the findings, and decides whether to close or escalate. Sometimes the work needs a specialist. Sometimes the procedure stops being useful halfway through.

While reading about Fabro, I kept coming back to that sequence. Its approach to software factories puts agents inside explicit workflows, with verification gates, human intervention, and durable execution records. The interesting part, to me, is the machinery around the model.

I have already been prototyping some of that machinery for security investigations. Not a fully autonomous SOC. An investigator foundation: controlled tools, case access, evidence storage, and a way to distinguish an action that succeeded from one whose outcome is still unknown.

This series will follow the project as it develops. I want to show the architecture, useful bits of code, things that fail, and the evidence behind any claim that the system is getting better.

The factory parallel, and where it breaks

The parallel is straightforward: an input enters a workflow, workers perform bounded tasks, quality checks review the result, and exceptions go to someone who can make a decision.

For a SOC, the output should be a defensible investigation. What happened? When? Which accounts and systems were involved? How did it happen? What remains unknown?

There is an important difference, though. Passing software tests does not map neatly to establishing the scope of an incident. Security evidence can be missing, misleading, or deliberately adversarial. A search returning no events might mean no activity, the wrong time window, or no useful telemetry at all.

So the factory cannot simply reward throughput. Closing more cases is not the same thing as understanding more cases.

Start with the foundation, not the agent org chart

The current prototype is written in Rust and exposes typed tools through MCP, the Model Context Protocol. A client can read authorized cases, create investigation leads, upload and retrieve evidence, search an indicator index, and append case notes. Selected enrichment lookups are available too.

The boundary matters more than the interface. Each hosted client has its own identity, permissions, and case grants. Access to the downstream case system uses a separate request-scoped credential. Knowing that a caller can reach the service does not mean they can read every case.

Figure 1. Implemented foundation, not an autonomous investigation loop. SQLite persistence must be configured; S3 evidence storage is optional. The arrows show logical operations, not every request and response.

Leads, evidence, note-write state, and audit events can survive a restart when persistence is enabled. Evidence gets a hash and a case binding. A note can reference a stable evidence ID rather than becoming the only place where the investigation exists.

There are still limits. Audit persistence is currently best-effort, not an immutable execution ledger. Retention metadata does not prove that storage enforces retention. Uploading artifacts is not yet automatic capture of every agent's queries and decisions.

These are implementation gaps to address, not details to hide behind the word "auditable."

An unknown outcome deserves its own type

One small piece of the implementation captures a surprisingly large operational problem:

pub enum AppendNoteResult {
    Confirmed { operation_id: Uuid, note_id: String },
    Unconfirmed { operation_id: Uuid },
}

Actual prototype code, with the derive attributes omitted.

Suppose the service submits a case note and the connection drops before the response arrives. Did the write fail, or did the note get saved?

Calling that a failure and retrying can create a duplicate. Calling it a success can invent a result. The prototype keeps the outcome unconfirmed, records the operation ID, and prevents a retry with the same idempotency key from posting again. With persistence enabled, that protection survives a restart.

The next step is reconciliation: inspect the destination and establish what actually happened.

I want the same discipline throughout the eventual investigation workflow. A tool error is not an empty result. A missing artifact is not a clean artifact. A plausible answer is not a verified answer.

Evidence integrity has a similarly explicit representation:

let metadata = EvidenceMetadata {
    id: Uuid::new_v4(),
    case_id: session.case_id.clone(),
    lead_id: session.lead_id,
    filename: session.filename,
    content_type: session.content_type,
    size_bytes: bytes.len() as u64,
    sha256: digest,
    created_by: actor.id.clone(),
    retention_until: now + self.limits.evidence_retention,
};

Actual prototype code from evidence-upload completion.

That hash helps detect changed bytes. It does not establish that a source is truthful, that collection was complete, or that an interpretation is correct. Integrity is one part of the evidence story, not the whole story.

The architecture I am working toward

The next layer would coordinate constrained, headless workers around a shared investigation record. "Headless" means the worker can execute without an interactive chat session; it does not mean unrestricted access.

Figure 2. Target design. The supervisor, specialist workflow, independent verification, and autonomous closure shown here are not implemented in the current investigator core. Individual worker roles are grouped to keep the control boundaries visible.

A supervisor would track the unanswered questions, choose an approved procedure, and assign bounded work. For a user-related alert, permitted enrichment might include identity, department, Okta or Duo activity, and an Oort risk profile. For a machine-related alert, it might include ownership, MDE evidence, and related activity in the subnet.

The specialists would have different responsibilities. A threat-intelligence worker could read advisories and turn relevant developments into hunting hypotheses. A hunting worker could run scoped searches. A case-intelligence worker could add relevant context or extract candidate indicators. Forensics and reverse-engineering workers would handle deeper artifact analysis, with appropriate evidence handling and isolation.

These are roles, not a requirement to keep six agents talking all day. A short-lived worker with a narrow task, limited tools, and an explicit output contract may be enough.

The proposed loop is intelligence to hypothesis, hunt to evidence, investigation to reviewed intelligence. The word "reviewed" is essential: an uncertain case observation must not become trusted intelligence and then return as supposed independent corroboration.

That is also where I see the useful difference from a conventional automation playbook. SOAR can already branch and call tools. What I want to explore is adaptive investigation inside a controlled workflow: choosing the next useful question without gaining the authority to do anything the model suggests.

A procedure is also an evidence contract

Phishing and cloud-provider alerts are the first workflows I would try. They offer recognizable starting points, but neither is automatically suitable for autonomous closure.

The SOPs will be available as SKILL.md files. Metadata describes when a skill applies and its dependencies; the Markdown body carries the investigation steps, conditional branches, evidence requirements, expected outputs, and safety rules.

The phishing-report skill is a useful example. It separates the report wrapper from the suspicious message, scopes delivery in Splunk when warranted, and prepares an evidence-backed verdict. Cleanup requests remain draft-only. A condensed example shows how those instructions fit together:

---
name: phishing-report
description: Investigate reported emails and prepare an evidence-backed verdict.
compatibility: >
  Requires email, email-message, local-analysis, dns-whois,
  virustotal skills and authorized Splunk access.
---

# Phishing Report Investigation

## Purpose
Classify reported messages as malicious, suspicious, spam, benign,
or inconclusive. Establish delivery scope when warranted and prepare
an analyst-ready report.

## Investigation Workflow

### 1. Collect the report
- Find the report using a message ID or bounded search window.
- Fetch attachments separately using each report's exact message ID;
  a bulk listing does not retrieve them.
- Capture the reporter, subject, received time, and artifact paths.

### 2. Extract the suspicious message
- Parse the attached .eml/.msg locally, not the reporting wrapper.
- Extract headers, body, URLs, domains, and attachment metadata.
- If only inline-forwarded content exists, document missing headers.

### 3. Check headers and authentication
- Compare From, Reply-To, Return-Path, and the claimed sender identity.
- Review SPF, DKIM, DMARC, and Received hops.
- Verify relevant domains; record mismatches and missing evidence.

### 4. Assess indicators and risk
- Extract indicators locally before permitted remote enrichment.
- Consult reputation sources such as VirusTotal; prefer existing
  URLScan results to new submissions.
- Assign a verdict supported by the message and available evidence.

### 5. Scope delivery when warranted
- For suspicious or malicious messages, search approved mail telemetry
  in Splunk using relevant identifiers and a bounded time window.
- Separate directly targeted recipients from forwarded-to recipients.
- Preserve the query and raw results. Record scoping as completed,
  skipped, failed, or unavailable, with the reason and coverage limits.

### 6. Prepare the report and any cleanup draft
- Return the verdict and the evidence supporting it.
- Include: Phish: yes|no|unclear.
- Include: Multiple users targeted: yes|no|unknown.
- Cite artifact paths, tool outputs, and exact queries; defang URLs.
- When Phish is yes and multiple users were targeted, draft an
  Exchange deletion request with recipients and matching criteria.
- Save the verdict, entities, and target list in investigation memory.

## Core Rules
- Keep the report wrapper separate from the suspicious message.
- Preserve original artifacts; save parsed or enriched data separately.
- Link every claim to its source and record unresolved evidence gaps.
- Never click suspicious URLs or submit internal artifacts externally.
- Missing data or a failed lookup does not establish a benign verdict.
- Draft deletion requests only; do not send them.

Condensed and generalized from the phishing-report SKILL.md, with evidence-preservation and coverage-status requirements made explicit for the proposed workflow. Not an executable workflow in the current prototype.

That makes the skill both a procedure and an evidence contract. The agent must retain what each step produced, explain skipped or failed checks, and distinguish insufficient evidence from a benign result. Original artifacts stay pristine; parsing and enrichment produce separate derived outputs.

A skill describes the rules; it does not enforce them or grant access. Tool permissions and action boundaries still belong in the runtime. Autonomous closure would need explicit, reviewed criteria covering evidence completeness, acceptable sources, coverage, and permitted dispositions. The phishing skill above does not authorize closure.

If the case follows an approved flow and meets its closure criteria, I want the system to be able to finish the investigation and close it. If there is no matching procedure, useful enrichment can still happen, but it should produce a handoff rather than an invented closure policy.

The verifier must inspect original artifacts, exact queries, returned data, and the conclusions built from them. It should challenge both oversimplification and overstatement. Saying "the account authenticated" is not necessarily the same as establishing which person performed the activity.

Emergency account suspension, external communications, and consequential escalation to legal or data-loss teams would remain human-approved. An agent can identify the need; that does not grant it authority to act.

Metrics, audit, and evidence

Metrics and audit are operational requirements, not reporting extras. A KPI overview should show backlog, time to disposition, escalation and reopening rates, evidence completeness, and cost per case. Operational support also needs to see stalled workflows and tool failures. The prototype's tool counters are only a starting point.

Audit provides explainability and accountability: which agent did what, under which permissions, and which evidence supported each decision. Findings and summaries must remain linked to original outputs, not replace them.

Raw evidence must stay pristine and access-controlled, with provenance and integrity checks. Parsing, redaction, and enrichment should produce separate derived artifacts, leaving the originals available for review, handoff, and reanalysis.

What comes next

Next, I want to turn a narrow phishing SOP into an end-to-end investigation: explicit evidence requirements, controlled tool use, a reviewable case record, and clear conditions for closure or human escalation.

Later posts will follow specialist handoffs, intelligence-driven hunts, and the dashboards and audit trail needed to support them. Each entry will show what was built, what remains uncertain, and why the architecture changed.

The goal is a complete investigation that another analyst can review and continue, with the original evidence intact and responsibility for decisions visible.