---
title: "Detection and reconciliation"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Detection and reconciliation}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```

This vignette walks through how a laboratory result becomes an outbreak or epidemic dossier an epidemiologist can act on: detection, scale routing, reconciliation, and suppression across runs.

## The lattice: streams, not one big feed

Every case belongs to several *streams* at once - a ward-level stream, an institution-level stream, an area, a province, the whole catchment - each watched independently. Streams are enumerated automatically from the data every run, so a newly appearing pathogen creates its own streams without any configuration step:

| Level | Unit                             |
| ----- | -------------------------------- |
| L1    | Ward or specialism               |
| L2    | Hospital or care home            |
| L3    | A contiguous geographic grouping |
| L4    | Province                         |
| L5    | Full catchment                   |

Each level is a strict partition of the one above it - a ward sits inside exactly one hospital, a hospital inside exactly one area, and so on - which is what makes suppression (below) meaningful.

## Four detectors, not one algorithm

- **`farrington`** (`surveillance::farringtonFlexible()`) - the standard
  statistical aberration detector for streams with enough baseline
  history, run at L1/L2 with a patient-day population offset where one
  exists, so a busy month and a quiet one at identical
  transmission-per-patient-day read as the same signal strength.
- **`same_place`** - a rule-based detector needing no baseline at all:
  *n* or more cases of the same organism at the same place within *k*
  days. This is what catches long-term care institutions, which are
  individually too sparse for a statistical baseline, and it also runs
  alongside Farrington at hospitals, where it can fire first on a fast
  cluster before a weekly aggregation has produced a point to test.
- **`rare_trigger`** - a curated list of organisms (default is the shipped
  `r EpiSODIC:::doc_system_file("inst/config/episodic_default_config.yaml")`) where a single occurrence anywhere is
  itself the signal - certain notifiable or exceptionally rare
  pathogens do not need a second case to warrant a look.
- **`mem`** (the `mem` package's Moving Epidemic Method) - for organisms
  whose own data show a seasonal pattern (derived automatically, or
  overridden per pathogen via `mem_mode`), answering "has the epidemic
  started" rather than "are counts elevated", with pre-/post-epidemic
  thresholds derived from historical seasons. The season anchor is
  derived from a 52-week climatology, so it works in either hemisphere
  with no configuration. MEM runs at province (L4) and region (L5) by
  default, configurable via `mem.levels`.

A stream's own history feeds back into its baseline: a period an epidemiologist confirmed as a real epidemic is excluded from what `farringtonFlexible()` sees on subsequent runs for that stream, so last winter's outbreak does not silently raise this winter's threshold.

## Scale routing: outbreaks and epidemics

A configurable boundary (`scale.epidemic_levels`) splits the lattice into two scales. Streams below the boundary produce **outbreaks** (localised, L1 to L3), shown on the Outbreaks screen. Streams at or above it produce **epidemics** (regional, L4 and L5), shown on the Epidemics screen. Each cluster receives its `scale` at creation.

Outbreaks are identified as `O-{id}` and epidemics as `E-{id}` in English. Both draw on one `cluster_id` sequence, so a given number is either an outbreak or an epidemic, never both. The prefix is not stored: it is the first letter of the screen's own name in the reader's language (`U-12` and `E-12` in Dutch, `A-12` and `E-12` in German), and a language whose two names begin with the same letter carries a distinct pair instead.

## Reconciliation: turning detections into outbreaks and epidemics

Detection alone produces a burst of raised signals every run, the same underlying outbreak would otherwise appear as a new row every single time it is still elevated. Reconciliation, run per stream immediately after detection inside the same transaction, is what turns that into a small number of *persistent* outbreaks and epidemics an epidemiologist can actually track:

1. Detections from different detectors whose intervals overlap merge
   into one candidate episode.
2. Each candidate is matched against currently open clusters in the same
   stream by interval overlap (or proximity within `case_free_days`):
   no match creates a new cluster; one match extends the existing one
   and recomputes its score; multiple matches merge, with the oldest
   surviving and the others' assessment history preserved, never
   discarded.
3. An outbreak nobody has seen a new candidate for gets its
   `runs_since_detected` counter incremented; past a threshold, if it has
   never been assessed at all, it closes itself automatically, a plain
   system closure, no verdict attached. An outbreak an epidemiologist has
   actually classified, whether as noise or as worth watching, is never
   auto-closed by any verdict of its own, closing it always takes a
   person's deliberate decision from the assessment form's own closure
   checkbox.

### Epidemic closure

Seasonal epidemics close automatically when the weekly case count drops below the post-epidemic threshold (`ended_reason = 'post_epidemic_threshold'`). As a backstop for noisy series that never cross cleanly, closure also fires when the evaluated week's count is back at or below the pre-epidemic threshold in a week inside the pathogen's derived trough (`ended_reason = 'trough'`). The trough on its own is a calendar rather than a measurement: an epidemic running out of season, with its count still above the pre-epidemic threshold, stays open until the post-epidemic threshold or an epidemiologist's declaration ends it. Non-seasonal epidemics close on the same `case_free_days` and `stale_open_days` rules as outbreaks.

An epidemic's identity is preserved across the season anchor rollover: a rise that has not been closed keeps its cluster ID when the season label changes, and no spurious second epidemic is opened.

### The "during" link

After reconciliation and before suppression, each open outbreak is checked against open epidemics for the same pathogen with overlapping dates and geographically nested geography. An outbreak that qualifies is linked as occurring **during** that epidemic, never **part of** it, because claiming a ward outbreak is part of the regional epidemic is a transmission claim that cannot be supported without typing data. An outbreak can be during more than one epidemic (a province and a region epidemic at once).

Because the anchor is the sample date, a late-arriving case can legitimately change an already-detected signal retrospectively - reconciliation deliberately re-runs over a rolling window, not only "today", sized from the empirical reporting-delay curve.

## Suppression: one lattice, not five parallel views

Without suppression, one real outbreak would appear as five separate signals, one per level, burying the actual signal in restatements of itself. After reconciliation:

- A child cluster suppresses its parent when it accounts for most of the
  parent's excess (the rise is local; the wider view adds nothing).
- A parent suppresses its children when the rise is spread across
  several of them with no single dominant one (the rise is diffuse;
  five separate area-level outbreaks would be noise).
- The suppressed clusters are never discarded, they stay attached as
  context, visible on the surviving cluster's detail screen.

Suppression is continuous across the outbreak/epidemic scale boundary. The `pathogen_area` (outbreak) to `pathogen_province` (epidemic) pair crosses it, so an area-level outbreak can suppress or be suppressed by a province-level epidemic. The geographic levels group on the *patient's* postcode, so a hospital's cases sit in no single area; and letting a diffuse regional signal suppress a ward outbreak built into it would hide the more actionable of the two behind the vaguer one.

That leaves clusters which share cases and both stand: a regional norovirus rise driven by a ward outbreak and a nursing home is one set of cases in three dossiers. Rather than collapse them, the dossier says so, each names the others in its header and lists them under Related clusters. Two views of one rise, and the epidemiologist is told which is which.

## Derived state, not a stored status

A cluster's displayed state (new, assessing, monitoring, closed, needs reassessment) is *computed*, every time, from its classification history, the case-free clock, and whether it changed since it was last assessed, never stored and hand-edited. See `episodic_derive_state()` for the exact rule and `state_derive.R`'s own tests for every transition it covers.

## Declarations: a human act

MEM's threshold crossing is a measurement. Declaring the season started is a decision with an author, a timestamp and policy consequences across multiple hospitals. The system raises the signal; a person declares. An epidemiologist records `season_started`, `season_not_yet` or `season_ended` through the assessment event stream on the Epidemics screen. Nothing in the system writes a declaration automatically.