Skip to content

L9.1 — Helios Health: scenario brief

Type: Reading · Duration: ~30 min · Status: Mandatory Module: Module 9 — Capstone Project

CONFIDENTIAL — for AI Security Engineering review

From: Marcus Thompson, VP Engineering, Helios Health To: [you], AI Security Engineer (contractor) Re: Triage Copilot — security review request Date: 2026-05-25

We've engaged you to conduct an AI security review of our upcoming Triage Copilot product before our July 1 GA launch. This brief covers the product, our current architecture, our compliance posture, and the deliverables we need from you. Please review and produce the four artifacts described in the deliverable spec.

— Marcus

The company

Helios Health is a US-headquartered B2B SaaS provider for mid-size and large hospital systems. Founded 2022. ~80 employees. Series B, $45M raised. Customers in the US (HIPAA-covered) and increasingly in the EU (GDPR + EU AI Act applicability).

Our existing products: - Helios Scheduler — staff scheduling for nursing teams. Not AI-bearing. - Helios Notes — clinical-note transcription (speech-to-text only; no LLM analysis). - Triage Copilot — the new product, subject of this engagement.

The product: Triage Copilot

What it does: an LLM-powered assistant for intake nurses. When a patient presents at the ED or urgent-care intake, the nurse enters their chief complaint and observed vital signs. The Copilot:

  1. Suggests likely conditions consistent with the symptoms (differential diagnosis support — not diagnosis itself).
  2. Recommends triage acuity (1-5 Emergency Severity Index).
  3. Surfaces relevant clinical protocols from the hospital's protocol library.
  4. Drafts an intake summary for the EHR.
  5. Flags potential drug interactions if the patient mentions current medications.

The Copilot is advisory only. Final triage and disposition decisions are made by the nurse. The product UX explicitly frames every suggestion as "for your consideration" and requires nurse confirmation before any output is committed to the EHR.

Volume: at GA, we expect ~50 deployed hospitals processing ~30,000 intakes/day in aggregate.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                       Hospital Network                          │
│                                                                 │
│   ┌────────────┐     ┌────────────────┐     ┌──────────────┐    │
│   │   Nurse    │────▶│  Triage UI     │────▶│  Helios API  │    │
│   │ (browser)  │     │  (Next.js)     │     │  (FastAPI)   │    │
│   └────────────┘     └────────────────┘     └──────┬───────┘    │
│                                                    │            │
│                              ┌─────────────────────┴───┐        │
│                              ▼                         ▼        │
│                       ┌────────────┐          ┌──────────────┐  │
│                       │ Hospital   │          │  Helios      │  │
│                       │ FHIR API   │          │  Vector DB   │  │
│                       │ (records)  │          │ (pgvector)   │  │
│                       └────────────┘          │ - Hospital   │  │
│                                               │   protocols  │  │
│                                               │ - Public     │  │
│                                               │   clinical   │  │
│                                               │   guidelines │  │
│                                               └──────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼ (egress via Helios proxy)
                          ┌───────────────────┐
                          │   Anthropic API   │
                          │  (Claude Sonnet)  │
                          └───────────────────┘

Tools the Copilot agent has access to: - search_protocols(query) — semantic search over hospital protocols + public clinical guidelines. - read_patient_record(patient_id) — fetch the patient's record from the hospital FHIR API. - search_drug_interactions(drugs) — query a third-party drug-interaction database. - draft_intake_summary(content) — write a draft summary (not committed until nurse confirms). - flag_critical(reason) — flag a case as needing immediate physician attention.

LLM backend: Anthropic Claude Sonnet 4.6 via API. No fine-tuning. System prompt defines persona, refusal triggers, output format, drug-interaction handling rules. We pay Anthropic ~$0.30 per intake on average.

RAG corpus: - Hospital-specific protocols (uploaded by hospital admin during onboarding; updateable by hospital clinical leadership). - Public clinical guidelines from CDC, NIH, WHO, professional societies (curated by Helios team; updated quarterly).

Vector DB: pgvector hosted alongside our application database. Per-tenant isolation via row-level security in Postgres.

Auth: SAML / OIDC SSO into hospital IDP. Each user is mapped to a hospital tenant. Per-user audit logging.

Compliance posture (as of 2026-05-25)

What we have: - SOC 2 Type II (Helios platform overall, including Triage Copilot infrastructure). - HIPAA BAA in place with Anthropic. - HIPAA BAA with each customer hospital. - GDPR Data Processing Agreement with EU hospital customers. - Privacy Impact Assessment completed for Triage Copilot (privacy team). - Per-tenant audit logging at the API layer.

What we don't have: - No formal AI risk register. We haven't done a structured AI threat model. - No formal AI red-team campaign. Internal eng has done ad-hoc prompt-injection testing but no documented campaign. - No EU AI Act risk-tier classification documented. We assume high-risk but haven't formally classified. - No Article 11 / Annex IV technical documentation package. - No AI-BOM. - No runtime guardrails beyond Anthropic's default Claude content moderation. No input filters, no structured-output enforcement, no dual-LLM pattern, no output sanitization beyond the EHR-commit gate. - No prompt/response logging with the specific shape your Module 7 lessons describe (we log request metadata and tool calls but not prompts/responses with PII redaction; we worry about HIPAA implications and have been conservative). - No AI incident response playbook. Our standard IR covers infrastructure; we haven't extended for AI-specific scenarios.

Stakeholders

  • Marcus Thompson (VP Engineering) — your direct reporting line for this engagement. Final approval on technical recommendations.
  • Dr. Priya Singh (Chief Medical Officer) — clinical accuracy and patient-safety stakeholder. Her sign-off is required before launch.
  • Jordan Reyes (Chief Compliance Officer) — HIPAA, GDPR, EU AI Act compliance. Will need formal documentation she can present to regulators.
  • Aisha Okonkwo (Product Manager, Triage Copilot) — feature owner. Concerned about launch timeline.
  • External: Hospital security teams — each customer hospital's CISO will conduct vendor security review. Past reviews have been thorough.

Constraints

  • Launch date: July 1, 2026. Marcus would prefer not to slip, but is open to "launch with mitigations" framing if findings warrant.
  • EU rollout: 8 EU hospital customers signed; first goes live August 15. The EU AI Act high-risk obligations apply.
  • Budget: budget for Q3 is set; meaningful new tooling or hires would need Q4 budget cycle (which starts in August).
  • External red-team: we have $80k allocated for an external red-team engagement post-launch (around September), so your job is the pre-launch internal gate.

What we want from you

Per your engagement letter, the deliverables (described in detail in L9.2) are:

  1. Threat model of Triage Copilot.
  2. Red-team report with prioritized findings.
  3. Remediation plan with what we should do before launch, at launch, and post-launch.
  4. Pre-launch checklist mapped to NIST AI RMF + EU AI Act.

Marcus, Dr. Singh, and Jordan will read your deliverables together. The recommendations need to be defensible to each of them — engineering credibility (Marcus), clinical-safety framing (Dr. Singh), regulatory-fit (Jordan).

We'll meet after you complete the work for a 60-minute readout. Your deliverables should stand on their own — Jordan will likely circulate to her board's risk committee, so write for that audience too.

Read this before starting the capstone

Questions to think about as you read (notes welcome; you'll need them):

  • Where does untrusted content enter the LLM's context? (Hint: more than one place.)
  • What's the agent's tool surface and what would a misuse of each tool look like?
  • What's the writer population for the RAG corpus? (Hospital admins, Helios curators, both?)
  • Where could Triage Copilot's output reach a downstream system that treats it as authoritative? (Hint: the EHR is one example, but there are others.)
  • Which OWASP LLM Top 10 entries apply most directly? (Spoiler: most of them.)
  • What's the cross-jurisdictional reporting matrix if you discover a serious incident at a EU hospital customer?

When you're ready, go to L9.2 for the deliverable spec and rubric.