---
title: "Tuning detection for your catchment"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Tuning detection for your catchment}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```
This vignette is for someone with a running EpiSODIC instance whose detection behaviour does not quite match their catchment: too many false alarms, missing outbreaks, clusters closing too early, the wrong things at the top of the board. It maps each of those complaints to the parameters that control it, the trade-off each adjustment creates, and how to verify the change helped. It does not re-explain how the detectors work ([**Detection and reconciliation**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/detection-reconciliation.html)) or how the configuration overlay system works ([**Deployment**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/deployment.html)).
## Before you change anything
Every detection run records the exact configuration it used: a SHA-1 hash of the resolved settings (`config_hash`) and a full JSON snapshot (`config_snapshot`), both on the run's own row. A change you make today affects only future runs, never the meaning of a past one. You can tune with confidence, since nothing becomes retroactively wrong.
Point the `EPISODIC_CONFIG` environment variable at a YAML file with only the keys you want to change; everything else keeps its shipped default. See [**Deployment**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/deployment.html) for the mechanics.
```yaml
# my_instance.yaml - only what differs from the defaults.
farrington:
alpha: 0.01
```
Three guidelines before you start:
1. Change one thing at a time. Two changes at once, and the effect
could be either or both.
2. State what you expect the change to do before you make it.
3. Decide how you will measure whether it worked: the Performance
screen for a live instance, or `episodic_validate_detection()` for a
controlled experiment. See "Verifying that a change helped" below.
## Too many false alarms
### The Farrington bound is too narrow
If Farrington fires on modest rises that are within normal variation for your catchment, widen the statistical bound or raise the effect-size floor that gates new clusters.
`farrington.alpha` is the significance level for the upper bound. Lowering it from 0.05 to 0.01 widens the bound, so a run needs a larger exceedance before it reports one. This is a sensitivity-specificity trade-off: fewer false alarms, but genuine small outbreaks take longer to appear or are missed entirely.
`effect_size_floor.min_excess_over_upperbound` (default: 3) requires at least this many cases above the bound before a *new* cluster is opened. Raising it filters out borderline exceedances. A match against an already-open cluster is always merged, regardless of this floor.
`effect_size_floor.min_ratio_observed_expected` (default: 1.5) requires the observed/expected ratio to reach this level. Together with the excess floor, it means a signal must be both statistically and practically notable. Either can be set to `~` (null) to disable it.
```yaml
farrington:
alpha: 0.01
effect_size_floor:
min_excess_over_upperbound: 5
min_ratio_observed_expected: 2.0
```
To measure the effect before deploying it:
```{r}
result_before <- episodic_validate_detection(history_years = 4)
result_after <- episodic_validate_detection(
history_years = 4,
config = list(
farrington = list(alpha = 0.01),
effect_size_floor = list(min_excess_over_upperbound = 5)
)
)
result_before$summary[result_before$summary$metric == "sensitivity", ]
result_after$summary[result_after$summary$metric == "sensitivity", ]
```
### same_place fires on endemic patterns
The rule-based detector opens clusters whenever N cases of the same pathogen appear at the same institution within K days. For a common organism at a large hospital, three cases in 14 days may be entirely unremarkable.
The global defaults (`same_place.default_n_cases` and `same_place.default_k_days`) affect every pathogen that does not have its own override. Raising `default_n_cases` globally also raises it for rare organisms where it should stay low, so per-pathogen overrides under `same_place.overrides` are usually the better lever:
```yaml
same_place:
overrides:
Escherichia coli: {n_cases: 5, k_days: 14}
Staphylococcus aureus: {n_cases: 4, k_days: 10}
```
Lowering `default_k_days` requires cases to be more tightly clustered in time, which reduces noise from background endemic rates.
### Suppression lets redundant signals through
If the same real outbreak appears as separate clusters at multiple lattice levels (a ward cluster and its parent institution cluster), suppression is not aggressive enough.
`suppression.child_dominance_threshold` (default: 0.70) is the share of a parent's cases that a single child must account for to suppress the parent. Lowering it makes it easier for a child to absorb its parent. `suppression.parent_diffuse_threshold` (default: 0.50) and `suppression.parent_min_flagged_children` (default: 1) control the reverse: a parent absorbs its children when each holds less than that share of its cases and at least that many children overlap it. The default of 1 lets a region absorb a single flagged province whose rise is mostly outside it; raise it to require the rise to be seen in several children first. A cluster someone has already assessed is never suppressed, regardless of these thresholds.
```yaml
suppression:
child_dominance_threshold: 0.60
parent_min_flagged_children: 3
```
Too aggressive suppression hides a genuinely independent signal behind a coincidentally overlapping one. If you find suppressed clusters that should not have been, raise the thresholds.
## Missing outbreaks
### Farrington never fits
If Farrington reports nothing and the run log says a stream does not have enough baseline history, the model cannot fit because the stream's history is shorter than the baseline requirement.
`farrington.b` (default: 2) is the number of reference years. The model needs `(b + 1) * 52` weeks before it fits at all, so at `b: 2` a stream needs three years of history. Lowering it to 1 reduces the requirement to two years, at the cost of a less stable model. Raise it as history accrues: 5 is the conventional choice for a mature instance.
The three eligibility gates (`eligibility.min_baseline_weeks`, `eligibility.min_median_weekly_count`, `eligibility.min_nonzero_week_share`) determine whether a stream reaches the statistical detectors at all. Streams that fail still run through `same_place` and `rare_trigger`. Loosening these lets the model fit on sparser streams, where it may not be meaningful.
```yaml
farrington:
b: 1
eligibility:
min_baseline_weeks: 26
```
### same_place thresholds are too high
A ward-level cluster of 2 cases of a serious organism may go unreported because the global `default_n_cases` is 3. A per-pathogen override is the targeted fix:
```yaml
same_place:
overrides:
Legionella: {n_cases: 2, k_days: 14}
```
### The eligibility gate excludes a stream
A stream you expect to be watched may fail the eligibility check. `eligibility.min_nonzero_week_share` (default: 0.2) requires at least 20% of baseline weeks to have at least one case. `eligibility.min_median_weekly_count` (default: 1) requires a median weekly count of at least 1. For genuinely sparse organisms, lowering these lets the stream through, though the statistical model is not meaningful on a stream with very few cases.
```yaml
eligibility:
min_nonzero_week_share: 0.10
min_median_weekly_count: 0
```
## Clusters close too early or too late
### Clusters close before the outbreak is really over
If a cluster auto-closes and a new one opens for the same ongoing outbreak, the case-free interval or the run-count limit is too short.
`reconciliation.case_free_days_default` (default: 14) is the fallback interval: if no case arrives within this many days, the cluster's episode ends. Per-pathogen `case_free_days` in the pathogen configuration CSV overrides this for pathogens with longer incubation periods. `reconciliation.close_after_runs` (default: 14) closes an unconfirmed cluster after this many consecutive runs with no new case.
```yaml
reconciliation:
case_free_days_default: 21
close_after_runs: 21
```
A cluster that has been confirmed by an epidemiologist is never auto-closed by `close_after_runs`, only by case-free days.
### Clusters stay open long after the last case
If the active queue has stale clusters nobody has looked at, `reconciliation.stale_open_days` (default: 60) force-closes any still-unassessed cluster this many days after its last case. `reconciliation.autoclose_unassessed` must be `true` (the default) for either `close_after_runs` or `stale_open_days` to act on an unassessed cluster.
```yaml
reconciliation:
stale_open_days: 30
close_after_runs: 7
```
Aggressive auto-closure can close a genuine slow-burn outbreak before anyone has assessed it.
### Cooldown absorption swallows a real recurrence
After a cluster closes, a cooldown window (per-pathogen `cooldown_days` in the pathogen configuration CSV) absorbs nearby cases into the recently-closed cluster rather than opening a new one. `reconciliation.cooldown_reopen_ratio` (default: 1.5) is an escape hatch: if the absorbed cases are at least 1.5 times the cluster's own size, the absorption is flagged rather than silent. Set to `~` to disable this escape hatch entirely, or shorten `cooldown_days` per pathogen to narrow the absorption window.
```yaml
reconciliation:
cooldown_reopen_ratio: ~
```
## Priority ranking does not match your triage
The priority score is a 0-100 weighted mean of seven components. A component that cannot be computed for a given cluster (for example, `density_component` requires a patient-day denominator) drops out and the remaining weights renormalise.
`priority_score.weights` controls how much each component contributes:
```yaml
priority_score:
weights:
excess_component: 0.25
ratio_component: 0.15
severity_component: 0.20
growth_component: 0.15
agreement_component: 0.10
density_component: 0.10
spatial_component: 0.05
```
Common adjustments:
- **Low-severity organisms rank too high**: raise `severity_component`.
It reads from `severity_weight` in the pathogen configuration CSV.
- **Large-count clusters of common organisms outrank small-count
clusters of serious ones**: raise `severity_component`, lower
`excess_component`.
- **Spatially concentrated clusters should rank higher**: raise
`spatial_component`.
- **Clusters where only one detector fired rank too high**: raise
`agreement_component`, so signals corroborated by multiple detectors
are ranked above those seen by one alone.
All seven weights must sum to 1.0.
## Seasonal detection is wrong
### MEM never runs
If no epidemics appear even for a pathogen you expect to be seasonal, MEM may lack enough history to fit.
`mem.min_seasons` (default: 2) requires two fully-observed prior seasons before fitting. `mem.min_climatology_years` (default: 3) requires three complete ISO years of case history to derive a season anchor. Both are "need real history first" dials with the same consequence as `farrington.b`: set beyond what an instance carries, MEM says nothing. Lower them for a newer instance, and raise them as history accrues.
Per-pathogen `mem_mode` in the pathogen configuration CSV overrides the automatic seasonality test. Set to `yes` to force MEM for a pathogen that the automatic test (`auto`) does not flag as seasonal, or to `no` to suppress it for one that it does.
```yaml
mem:
min_seasons: 1
min_climatology_years: 2
```
### MEM fires too early or too late
If the pre-epidemic threshold crossing does not align with clinical reality, adjust the climatology or the seasonality test.
`mem.climatology_smooth_weeks` (default: 5) is the smoothing window for the 52-week climatology used to find the season trough. `mem.trough_percentile` (default: 0.25) is the quantile at or below which a week counts as part of the trough. `mem.seasonality_peak_weeks` (default: 8) and `mem.seasonality_min_peak_share` (default: 0.40) control the automatic seasonality test: a pathogen is seasonal under `mem_mode = 'auto'` if at least 40% of its detrended annual cases fall within an 8-week peak window.
```yaml
mem:
climatology_smooth_weeks: 7
trough_percentile: 0.20
```
Declaring the season started or ended is always a human act, never automatic. MEM raises the signal; a person declares. See [**Detection and reconciliation**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/detection-reconciliation.html).
## Per-pathogen parameters
Per-pathogen biology lives in a CSV, separate from the YAML configuration. The default file is `r EpiSODIC:::doc_system_file("inst/config/episodic_default_pathogen_config.csv")`.
To override it, point `EPISODIC_PATHOGEN_CONFIG` at your own CSV containing only the pathogens and columns you want to change. For each pathogen you list, non-empty values override the default; empty cells keep it. Pathogens not in your file keep their row entirely. New pathogens are added.
```r
Sys.setenv(EPISODIC_PATHOGEN_CONFIG = "/path/to/my_pathogens.csv")
```
A minimal overlay that changes only the severity weight for MRSA and adds a local pathogen:
```
pathogen,severity_weight,case_free_days
MRSA,0.95,
My Local Pathogen,0.60,21
```
The columns and what they control:
| Column | Effect |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `pathogen` | Match key, raw lab-provided string |
| `episode_days` | Deduplication window for repeated isolates from the same patient (via `AMR::get_episode()`) |
| `incub_min_days`, `incub_max_days` | Incubation period bounds, shown on the dossier timeline |
| `case_free_days` | Days without a new case before a cluster's episode boundary, overrides reconciliation:
case_free_days_default |
| `cooldown_days` | Cool-down window after closure during which new cases are absorbed rather than opening a new cluster |
| `rt_applicable` | Whether Rt estimation applies (0 or 1) |
| `si_mean_days`, `si_sd_days` | Serial interval parameters for Rt estimation |
| `si_dist` | Serial interval distribution: `gamma`, `lognormal`, or `weibull` |
| `end_r`, `end_k` | Mean and dispersion of the negative binomial number of cases each outbreak case goes on to cause, for the probability that an outbreak is over; both or neither, `end_r` at least 0 and `end_k` above 0. Shipped empty for every pathogen |
| `mem_mode` | MEM seasonal detection: `auto` (derive from data), `yes` (force on), `no` (force off) |
| `severity_weight` | Clinical severity weight (0 to 1) feeding the priority score's severity component |
A common confusion point: the per-pathogen *detection* thresholds for `same_place` are in the YAML (under `same_place.overrides`), not in this CSV. The CSV holds per-pathogen *biology*, the YAML holds per-pathogen *detection rules*.
The validation framework accepts per-pathogen overrides through the `pathogen_config` argument:
```{r}
result <- episodic_validate_detection(
history_years = 4,
pathogen_config = data.frame(
pathogen = "MRSA",
severity_weight = 0.95,
stringsAsFactors = FALSE
)
)
```
## Verifying that a change helped
### Against synthetic ground truth
`episodic_validate_detection()` replays generated history through the full detection pipeline week by week and matches what it found against seeded outbreaks on case sets. Pass a `config` list (for YAML keys) or a `pathogen_config` data frame (for CSV columns) with the settings you want to test:
```{r}
baseline <- episodic_validate_detection(
seeds = 1:10,
history_years = 4,
evaluation_weeks = 52
)
adjusted <- episodic_validate_detection(
seeds = 1:10,
history_years = 4,
evaluation_weeks = 52,
config = list(farrington = list(alpha = 0.01))
)
# Compare
baseline$summary[baseline$summary$metric %in% c("sensitivity", "ppv"), ]
adjusted$summary[adjusted$summary$metric %in% c("sensitivity", "ppv"), ]
```
This measures the machinery against known truth: does the change improve sensitivity, reduce false alarms, or affect time-to-detection? The defaults are deliberately small (one seed, four weeks) so it returns in seconds; for a proper comparison use more seeds and a longer window. See [**Measuring detection against known truth**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/detection-validation.html) for the full framework.
### Against your own board's verdicts
The Performance screen measures the instance against its own epidemiologists' verdicts: what share of clusters the board classified as noise (positive predictive value), and how quickly real signals were detected (time-to-detection). This is not circular for the question "did this change reduce false alarms", which is exactly what PPV answers. It cannot measure sensitivity, because an outbreak the detectors miss never becomes a cluster for anyone to judge.
`config_hash` lets you compare runs before and after a change: filter the Performance screen or the `episodic_detection_run` table by hash to see the metrics under the old settings beside the new ones.
## Reference
Every tunable parameter across both the YAML configuration and the per-pathogen CSV, with its default and one-line effect.
### Reconciliation
| Key | Default | Effect |
| --------------------------------------- | ------: | ----------------------------------------------------------------------------- |
| reconciliation:
case_free_days_default | 14 | Fallback days without a case before a cluster episode boundary |
| reconciliation:
close_after_runs | 14 | Consecutive runs with no new case before an unconfirmed cluster auto-closes |
| reconciliation:
cooldown_reopen_ratio | 1.5 | Absorption escape-hatch ratio; `~` disables |
| reconciliation:
stale_open_days | 60 | Force-closes unassessed clusters this many days after last case; `~` disables |
| reconciliation:
autoclose_unassessed | `TRUE` | Whether auto-closure rules apply to clusters nobody assessed |
### Eligibility
| Key | Default | Effect |
| ------------------------------------- | ------: | --------------------------------------------------------------------------------- |
| eligibility:
min_baseline_weeks | 52 | Weeks of history required before statistical detection runs |
| eligibility:
min_median_weekly_count | 1 | Median weekly count over baseline; streams below are excluded from Farrington/MEM |
| eligibility:
min_nonzero_week_share | 0.2 | Share of baseline weeks with at least one case |
### Effect-size floor
| Key | Default | Effect |
| ----------------------------------------------- | ------: | --------------------------------------------------------------------------------- |
| effect_size_floor:
min_excess_over_upperbound | 3 | Cases above Farrington's upper bound required to open a new cluster; `~` disables |
| effect_size_floor:
min_ratio_observed_expected | 1.5 | Observed/expected ratio required to open a new cluster; `~` disables |
### same_place detector
| Key | Default | Effect |
| --------------------------------- | -------------: | --------------------------------------------------------------- |
| same_place:
enabled | `TRUE` | Master switch |
| same_place:
default_n_cases | 3 | Cases at one institution within K days to fire |
| same_place:
default_k_days | 14 | Time window in days |
| same_place:
lookback_days | 90 | How far back a run reports hits; `~` unbounds (not recommended) |
| same_place:
overrides:
<pathogen> | (per pathogen) | `{n_cases, k_days}` per raw pathogen string |
### Farrington detector
| Key | Default | Effect |
| --------------------------------- | ------: | ----------------------------------------------------------------------------------- |
| farrington:
enabled | `TRUE` | Master switch; switching off stops the detector, not the Pathogen screen's baseline |
| farrington:
aggregation | week | Aggregation unit |
| farrington:
b | 2 | Reference years; model needs `(b+1)*52` weeks |
| farrington:
w | 3 | Half-window (weeks) around the same calendar week in each reference year |
| farrington:
reweight | `TRUE` | Down-weight past outbreak weeks by Anscombe residual |
| farrington:
weightsThreshold | 2.58 | Residual cutoff for reweighting |
| farrington:
trend | `TRUE` | Allow a linear trend term |
| farrington:
pastWeeksNotIncluded | 26 | Trailing weeks excluded from the noise estimate |
| farrington:
limit54 | [5, 4] | Low-count safeguard |
| farrington:
alpha | 0.05 | Significance level; lower = wider bound = fewer alarms |
| farrington:
max_weeks_tested | 8 | Cap on trailing weeks tested per run |
### MEM detector
| Key | Default | Effect |
| -------------------------------- | -----------------------------------: | --------------------------------------------------------------------------- |
| mem:
enabled | `TRUE` | Master switch |
| mem:
min_seasons | 2 | Fully-observed prior seasons required before fitting |
| mem:
levels | [pathogen_province, pathogen_region] | Lattice levels MEM runs at |
| mem:
min_climatology_years | 3 | Complete ISO years required to derive a season anchor |
| mem:
climatology_smooth_weeks | 5 | Smoothing window for the 52-week climatology |
| mem:
trough_percentile | 0.25 | Quantile at or below which a week is part of the trough |
| mem:
seasonality_peak_weeks | 8 | Peak-concentration window width for the seasonality test |
| mem:
seasonality_min_peak_share | 0.40 | Minimum share of detrended cases in the peak window for `mem_mode = 'auto'` |
| mem:
typical_curve | `median` | Statistic the typical past season is summarised by: `median`, `geometric_mean` or `arithmetic_mean`; see "Typical past season" below |
### rare_trigger detector
| Key | Default | Effect |
| ---------------------------- | ----------: | --------------------------------------------------------------- |
| rare_trigger:
enabled | `TRUE` | Master switch |
| rare_trigger:
pathogens | (4 shipped) | Curated list of raw pathogen strings; a single case fires |
| rare_trigger:
min_cases | 1 | Cases in the window to fire |
| rare_trigger:
lookback_days | 90 | How far back a run reports hits; `~` unbounds (not recommended) |
### Nowcast
Every run nowcasts each stream that carries an open cluster: the expected final count of its most recent days and weeks, once reporting is complete, from the stream's own reporting delays (`surveillance::nowcast()`, method `bayes.trunc`). A case's reporting date is the date of the run that first loaded it, so the delay's resolution is the cron schedule. Only cases sampled on or after the first run on the database carry an observed delay; a backfilled archive does not. Each run's nowcast is stored, drawn on the dossier curves, and scored on the Performance screen once its targets are past `max_delay_days`.
| Key | Default | Effect |
| --------------------------------- | ------: | ----------------------------------------------------------------------------------------------- |
| forecast:
nowcast:
enabled | `TRUE` | Master switch |
| forecast:
nowcast:
max_delay_days | 21 | Reporting horizon in days: the longest delay allowed for, and the days nowcast |
| forecast:
nowcast:
min_completeness | 0.25 | Expected share already reported below which a day is not nowcast |
| forecast:
nowcast:
min_reported_cases | 20 | Cases with a fully observed delay a stream needs before it is nowcast |
| forecast:
nowcast:
prior_window_days | 28 | Fully reported days whose mean daily count centres the prior |
| forecast:
nowcast:
delay_window_days | `~` | Days of reports the delay is estimated from; `~` uses the whole history, a value must be at least `max_delay_days` |
| forecast:
nowcast:
n_samples | 500 | Delay-distribution samples per nowcast (at least 100) |
### Outbreak-end probability
Every run gives each open outbreak of a transmissible pathogen (`rt_applicable` 1, with a complete serial interval) the probability that it is over: that none of its cases goes on to cause a case not yet reported (Nishiura et al. 2016, with the negative binomial offspring distribution of Linton et al. 2022). Its inputs are the pathogen's serial interval and its offspring mean and dispersion (`end_r`, `end_k` in the pathogen CSV), and the stream's reporting completeness, so quiet days whose cases may still be on their way count for less. It is not computed for an outbreak whose cases all fall within one maximum incubation period, which points to a common source rather than transmission. The dossier shows it beside the case-free days, with the date it reaches `probability_threshold` if no further case is reported; it never closes an outbreak. Each run's probability is stored and scored on the Performance screen once the case-free window after the last case has passed and been reported.
`end_r` and `end_k` ship empty for every pathogen: set them in your pathogen CSV from a source you have checked for your setting. Until they are set, the dossier says so instead of showing a probability.
| Key | Default | Effect |
| ------------------------------------ | ------: | --------------------------------------------------------------- |
| forecast:
outbreak_end:
enabled | `TRUE` | Master switch |
| forecast:
outbreak_end:
probability_threshold | 0.95 | Probability whose date the dossier reports (above 0, below 1) |
### Epidemic direction
Every run estimates, for each open epidemic, its stream's weekly growth rate over the last `window_weeks` weeks: a log-linear Poisson regression on the weekly counts, its standard error scaled by the Pearson dispersion (never below Poisson). The weeks still inside the nowcast's reporting horizon (`forecast.nowcast.max_delay_days`) are taken from the nowcast, their uncertainty carried through by sampling; a week neither fully reported nor nowcast ends the window before it rather than being read as final, so without a nowcast the window ends some weeks back, and the dossier names the weeks it covers. Weeks that begin before the earliest sample date in the case data are left out, never counted as weeks without cases. The dossier shows the change in weekly cases the trend implies, the doubling or halving time when the 90% interval is on one side of zero, the chance the epidemic is growing, and the chance it is past its peak so far: declining, with its last week below its highest since its first week. That last says nothing about whether it will rise again. Each run's estimate is stored and scored on the Performance screen once its weeks have passed the reporting horizon, against the same trend fitted to their counts as then reported.
| Key | Default | Effect |
| ------------------------------------ | ------: | --------------------------------------------------------------- |
| forecast:
epidemic_direction:
enabled | `TRUE` | Master switch |
| forecast:
epidemic_direction:
window_weeks | 4 | Weeks the growth rate is estimated over (at least 3) |
| forecast:
epidemic_direction:
min_cases | 20 | Reported cases those weeks need before a rate is estimated |
| forecast:
epidemic_direction:
n_samples | 1000 | Draws the interval and probabilities are computed from (at least 100) |
### Epidemic outlook
Every run forecasts, for each open epidemic, its stream's weekly count in each of the `horizon_weeks` weeks after the last complete week. The forecast is an equally weighted ensemble of two models, combined by averaging their quantiles as the European and US COVID-19 forecast hubs combine theirs: a model of the stream's whole history, which knows its seasons and usual levels, and the trend of its last weeks extended ahead, which follows a rise or fall the history has not seen. Averaging quantiles keeps the ensemble about as sharp as its members, where pooling their draws would widen it to cover both.
The history model is `surveillance::hhh4()` with a negative binomial count: each week's expected count is an endemic level, seasonal (one sine-cosine pair of period 52 weeks) for a seasonal epidemic and constant otherwise, plus an autoregressive share of the week before. It is fitted on the stream's fully reported weeks, of which it needs `min_history_weeks`. When the autoregressive share is not identified (its log-scale standard error above 1, as when the counts show no week-to-week dependence), the endemic level alone is used rather than a share whose uncertainty would produce runaway paths.
Paths are simulated from the last fully reported week with parameters drawn from the fit's approximate sampling distribution, so the intervals carry the uncertainty of the estimates as well as the counts' noise. Weeks still being reported are taken from the nowcast, or, without one, drawn from the model but never below what has already been reported; the week in progress likewise.
The trend is the log-linear trend the direction tile reports, fitted along each path over the last `forecast.epidemic_direction.window_weeks` weeks and extended to the target weeks with its parameter uncertainty and the counts' dispersion. It takes part only when every week of that window, up to the last complete week, is fully reported or nowcast: a trend is a short-range forecast, and one fitted to the history model's own draws, or extended from a window that ended weeks back, carries a growth rate far beyond where it was measured. Without it, which is the case until the nowcast has a reporting history, the outlook is the history model's alone, and the dossier says so.
For a seasonal epidemic, each week ahead also gets the chance of reaching each MEM band it was opened against, read from the averaged quantiles, and each band the chance of being reached in any of the weeks, read from the averaged distribution of each path's highest week.
The Epidemics dossier shows the outlook, as a table and as marks after the weekly curve's bars, only once that pathogen's outlooks have been scored on `min_scored` weeks and, over those, beat expecting no change (the last complete week's count carried forward): their interval score relative to it is below 1. It states that record, with the share inside the 90% interval. Until then it says how far the record has come, or that it does not beat expecting no change: a model that fits past seasons can still miss an unusual one, which is what the record is there to show. Each run's outlook is stored and scored on the Performance screen as its weeks are reported.
| Key | Default | Effect |
| ------------------------------------ | ------: | --------------------------------------------------------------- |
| forecast:
epidemic_outlook:
enabled | `TRUE` | Master switch |
| forecast:
epidemic_outlook:
horizon_weeks | 4 | Weeks ahead forecast (1 to 8) |
| forecast:
epidemic_outlook:
min_history_weeks | 104 | Fully reported weeks the model is fitted on at least (at least 26) |
| forecast:
epidemic_outlook:
n_samples | 1000 | Simulated paths per outlook (at least 100) |
| forecast:
epidemic_outlook:
min_scored | 20 | Scored weeks a pathogen's outlooks need, and must beat expecting no change over, before the dossier shows them |
### Typical past season
When MEM opens a seasonal epidemic, the fit that set its thresholds also gives the typical course of the past seasons' epidemics (`mem::memmodel()`'s typical curve, with its 95% interval), the week they typically started, and their typical length. These are stored with the epidemic and fixed, like its thresholds. The Epidemics dossier draws the typical curve on the weekly curve with its start at this epidemic's onset, and gives the typical length and how many weeks earlier or later this epidemic started. It describes past seasons and is labelled as such; it is not a forecast.
`mem.typical_curve` sets the statistic the curve summarises the aligned past seasons by, week by week. It changes only the curve: the thresholds, the intensity bands, and the typical start and length are the same whichever it is. The start and length are always medians.
| Value | Curve | Interval |
| ----------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| `median` | The middle past season in each week: one extreme season does not move it. Equal to the arithmetic mean with two seasons | Hettmansperger-Sheather/Nyblom, normal approximation with few seasons |
| `geometric_mean` | `exp(mean(log(x + 1))) - 1`: less pulled up by one large season than the arithmetic mean, pulled down by one quiet one, and always below it | Normal on the log scale |
| `arithmetic_mean` | The average: a single severe season raises the whole curve | Normal, symmetric, so its lower limit can fall below zero in quiet weeks |
`mem`'s median with a bootstrap interval (`i.type.curve = 4`) is deliberately not offered: it samples at random, so the same seasons give a different curve on each fit and the fit could not be reused from run to run; it takes some thirty seconds per fit; and it fails on two seasons. The dossier names the statistic under the curve, and a limit below zero is drawn at zero. Changing the value refits every stream's MEM model on the next run, but an epidemic already open keeps the curve it opened with.
### Priority score
| Key | Default | Effect |
| -------------------------------------------- | ------: | ----------------------------------------------------------- |
| priority_score:
weights:
excess_component | 0.25 | Weight for cases above the model's upper bound |
| priority_score:
weights:
ratio_component | 0.15 | Weight for observed/expected ratio |
| priority_score:
weights:
severity_component | 0.20 | Weight for the pathogen's severity_weight |
| priority_score:
weights:
growth_component | 0.15 | Weight for cluster acceleration |
| priority_score:
weights:
agreement_component | 0.10 | Weight for share of detectors that fired |
| priority_score:
weights:
density_component | 0.10 | Weight for incidence per 1,000 patient-days versus baseline |
| priority_score:
weights:
spatial_component | 0.05 | Weight for case concentration in one location |
### Scale
| Key | Default | Effect |
| ----------------------- | ------------------------------------ | ----------------------------------------------------------- |
| scale:
epidemic_levels | [pathogen_province, pathogen_region] | Lattice levels that produce epidemics rather than outbreaks |
### Geography
| Key | Default | Effect |
| ------------------------------ | :------ | ----------------------------------------------------- |
| geography:
region_code | REGION | L5 whole-catchment code; name your service area |
| geography:
area_code_prefix | AREA- | L3 prefix before the leading postcode characters |
| geography:
area_pc_characters | 2 | Number of leading postcode characters for L3 grouping |
### Suppression
| Key | Default | Effect |
| ----------------------------------------- | ------: | ---------------------------------------------------------- |
| suppression:
child_dominance_threshold | 0.70 | Case-overlap share for a child to suppress its parent |
| suppression:
parent_diffuse_threshold | 0.50 | Below this per child, the rise counts as diffuse |
| suppression:
parent_min_flagged_children | 1 | Minimum overlapping children before the diffuse rule fires |
### Notifications
`notifications` ships empty (`~`): nothing is sent until it is configured. Its keys are about delivery, not detection, and are left out of `config_hash`.
| Key | Default | Effect |
| ---------------------- | ------- | --------------------------------------------------------- |
| notifications:
enabled | (unset) | Master switch for every channel |
| notifications:
dashboard_url | (unset) | Address of the dashboard, linked from each message |
| notifications:
triggers:
new_clusters | (unset) | Send when a run opens at least one new cluster; must be set to `true` explicitly |
| notifications:
triggers:
run_failure | (unset) | Send when a run fails, including one that stops before it begins; must be set to `true` explicitly |
| notifications:
channels:
<channel> | (unset) | One block per channel (`ntfy`, `smtp`, `sendmail`, `microsoft365`, `teams`, `slack`), each with its own `enabled` and settings |
Every channel's settings, and how to test them, are in [**Notifications**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/notifications.html).
### Report
| Key | Default | Effect |
| ------------------------------ | ------: | --------------------------------------------------------------------------------------------- |
| report:
small_count_threshold | 5 | Counts below this in geography/institution tables are suppressed to prevent re-identification |
| report:
output_dir | ~ | Base directory for rendered reports; mandatory when `EPISODIC_DB` is a DSN |
### Access
| Key | Default | Effect |
| ---------------------- | ------- | --------------------------------------------------------- |
| access:
require_login | `FALSE` | Whether the dashboard requires a sign-in to read anything |
### Database
| Key | Default | Effect |
| ---------------------- | ------- | --------------------------------------------------------- |
| database:
auto_migrate | `TRUE` | Whether a scheduled run brings the database up to the installed package's schema itself; see [**Deployment**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/deployment.html)'s "Upgrading EpiSODIC" |
### Per-pathogen CSV columns
| Column | Typical default | Effect |
| ---------------------------------- | --------------: | ------------------------------------------------- |
| `episode_days` | 30 | Deduplication window for repeated isolates |
| `incub_min_days`, `incub_max_days` | (varies) | Incubation period bounds for the dossier timeline |
| `case_free_days` | 14 | Overrides reconciliation:
case_free_days_default |
| `cooldown_days` | (varies) | Cool-down absorption window after closure |
| `rt_applicable` | 0 | Whether Rt estimation applies (0 or 1) |
| `si_mean_days`, `si_sd_days` | (varies) | Serial interval for Rt |
| `si_dist` | (varies) | Serial interval distribution |
| `end_r`, `end_k` | (empty) | Offspring mean and dispersion for the outbreak-end probability |
| `mem_mode` | auto | MEM: `auto`, `yes`, or `no` |
| `severity_weight` | 1.00 | Clinical severity for priority scoring (0 to 1) |
## See also
- [**Detection and reconciliation**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/detection-reconciliation.html) - how the detectors, reconciliation, suppression, and MEM work
- [**Deployment**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/deployment.html) - the configuration overlay system and instance setup
- [**Measuring detection against known truth**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/detection-validation.html) - measuring detection against known truth
- [**Environment variables**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/environment-variables.html) - all `EPISODIC_*` variables including `EPISODIC_PATHOGEN_CONFIG`