--- title: "Frequently asked questions" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Frequently asked questions} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE) ``` This page collects the questions that come up most often when getting started with EpiSODIC. Skim the questions first, then click the one you need to open its answer - nothing on this page needs to be read start to finish. If your question is not here, [**Getting your data in**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/data-format.html), [**Deployment**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/deployment.html), [**Environment variables**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/environment-variables.html) and [**Detection and reconciliation**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/detection-reconciliation.html) go deeper on most of these topics; and if it still is not answered anywhere, that is worth telling us, since it means this page is missing something it should cover. ## Getting your data in
How do I prepare my data? Do not worry about getting this perfect by hand - that is exactly what `episodic_check_cases()` is for. Hand it your extract, however rough: ```r EpiSODIC::episodic_check_cases(my_cases) ``` No database, no configuration, nothing changed - it just reads your data and tells you, in one pass, everything it finds: which column is wrong, how many rows are affected, which rows those are, what the offending values look like, and what to do about each. It also separates two kinds of finding, so you are never left guessing which ones actually block you: **problems** (a run refuses to start until these are fixed - a date that does not parse, a value outside the allowed set) and **advice** (a run proceeds regardless, but it is worth a look - one pathogen spelled two ways, no `ward` on any hospital row). Run it, fix what it flags as a problem, run it again - most people get to a clean report in two or three passes. [**Getting your data in**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/data-format.html) has the full column requirements this is checking against, including the exact allowed values for `care_line`, `institution_type`, and `sex`. If you are also supplying the optional positivity or hospital-activity feeds, `episodic_check_denominators()` and `episodic_check_institution_activity()` are the same idea, over those: ```r EpiSODIC::episodic_check_denominators(my_denominators) EpiSODIC::episodic_check_institution_activity(my_institution_activity) ```
Do I need to send all of my positives every time, or only new/recent ones? Only a recent window, with a couple of weeks of overlap - that is the recommended norm, and it is fully supported. You do *not* need to re-extract a patient's whole history on every run. Here is why that is safe: every run checks each incoming positive against the most recent episode EpiSODIC already has on file for that patient/pathogen, not just against the other rows in the same batch. So if a patient's first positive was loaded last week and a second positive for the same episode arrives today in an otherwise-unrelated extract, EpiSODIC still recognises it as the same episode and does not create a second case for it. A window with, say, two weeks of overlap with your previous extract is a sensible default - it gives you a safety margin if a run is ever skipped or delayed, without requiring you to resend everything. Two related things worth knowing: - **Re-sending a row is always safe**, however wide your window is. Every row needs a `source_key` unique to your own source system; resending a `source_key` EpiSODIC has already loaded is simply a no-op. There is no bookkeeping you need to do to avoid duplicates - if in doubt, a wider window (or even a full resend) never causes harm, it is just more rows for `episodic_check_cases()`/the load step to look at. - **Do not send negative results.** This feed is deliberately positives-only, regardless of how wide a window you choose.
What do I do when a lab result was rectified or corrected, but EpiSODIC already has the now-false positive? First: do not worry, this happens, and it is a normal part of running any detection system on live laboratory data - it is not something you did wrong. EpiSODIC does not offer a way to delete or edit an individual case once it has been loaded, and that is deliberate: the case data is the audit trail every dossier and every report is built from, so quietly removing a row would make past results harder to explain rather than easier. What this means in practice depends on where the correction lands: - **If the correction arrives before a cluster was raised from it** (or before the affected run), you do not need to do anything - it simply will not be part of any signal, since it is just data sitting in the database like any other case. - **If a cluster has already been raised (or affected) because of it**, handle it the same way you would handle any signal that turns out not to reflect reality: assess and close it in the dashboard with the classification that fits (e.g. not an outbreak), noting the correction in the assessment. That decision - including the reasoning - becomes part of the cluster's own audit trail, which is exactly the record you want to be able to point to later if anyone asks why a signal did not lead anywhere. - **If a stream is regularly noisy for a reason like this** (a lab or feed that corrects unusually often), muting it for a defined period is the right tool, with the reason recorded - see the mute action on a cluster's dossier. If this is happening often enough that it is a recurring nuisance rather than an occasional correction, that is worth raising with whoever owns your extract/transform step - the earlier a correction happens before data reaches EpiSODIC, the less cleanup it costs anyone downstream.
What happens to detection if I am missing optional columns - postcode, ward, institution activity - for some or all of my cases? You lose only the parts of the picture that specific column feeds, and nothing else - EpiSODIC is built so every optional piece degrades gracefully rather than blocking anything: - No `pc` (postcode) on a row: that case just does not contribute to the geography panel/choropleth or the concentration measure for its own cluster; every other detector and panel works exactly as normal. - No `ward` on hospital rows: the ward-level (`L1`) `same_place` detector has nothing to group those cases on, so ward-level detection is effectively skipped for them - institution-level (`L2`) and above are unaffected. - No institution-activity feed at all: `L1`/`L2` Farrington detection simply uses raw case counts instead of counts normalised by patient-days. Perfectly usable, just not adjusted for a busy month looking busier than a quiet one. - No denominators (positivity) feed: the positivity panel on the dossier stays blank for your streams; nothing about detection itself depends on it, since it is an interpretive aid, never a detection input. In short: send what you reliably have. `episodic_check_cases()` will flag missing optional columns as *advice*, not a *problem*, so you will always know what you are missing without anything blocking your run.
## Running it day to day
How do I know whether a scheduled run actually succeeded, partially loaded, or failed? You do not have to go near a log file for this - it is recorded on every run and surfaced in the dashboard itself. Every `episodic_run_cron()` call writes a `status` of `success`, `partial`, or `failed`, along with exactly what each feed delivered (`n_cases_supplied`, `n_cases_inserted`, and the equivalent for denominators/institution activity). The dashboard's Activity screen shows this under each run, so "did last night's extract arrive in full?" is answerable at a glance. A `failed` run means nothing was written at all (the whole run is one transaction, so it is always safe to just retry); a `partial` run means everything supplied was validated and detections are valid, but some optional-feed rows were skipped (e.g. institution activity for an institution EpiSODIC does not know) - both `success` and `partial` are usable runs.
I loaded years of history. What happens to the old clusters in it? They are found, opened, and mostly closed again, in that one run. The **first** run against a database is a backfill: it reports every hit in the case history it was given, rather than only what falls inside the detectors' lookback windows. Any cluster whose last case is further back than `reconciliation.stale_open_days` (60 by default) is closed by the same run that opened it and appears in the Archive, complete, with its real case period. What is still recent stays open for assessment. The run says how the two divided. Every run after the first is bounded again, so none of that is reported twice. That bound is why it matters: without it a settled historical hit is re-emitted on every run, matches its own long-closed cluster, and resets `runs_since_detected`, which stops `reconciliation.close_after_runs` from ever firing again. Note what a backfilled cluster's `opened_at` means. It is the day the archive was imported, not the day anything was noticed, so the Performance screen leaves these clusters out of time-to-detection. They still count towards positive predictive value if a person assesses one, because that is a judgement about the detectors like any other. To override the decision, pass `backfill` to `episodic_run_cron()` explicitly. `FALSE` bounds even the first run, which is what a prospective replay needs.
A run found nothing at all. Is detection broken? The run's own log says which bound you met. - `same_place` and `rare_trigger` report only hits whose most recent case falls inside their `lookback_days` (90 by default). A run says how many further hits that kept out, so "found 0 detection(s)" with nothing after it means a quiet catchment, and "did not report 137 further hit(s)" means history that an earlier run already reported. - Farrington needs `(farrington.b + 1) * 52` weeks of history on a stream before it fits at all - 156 weeks at the shipped `b: 2`. An instance carrying two years of history therefore gets nothing from it anywhere, and the run says on how many of the eligible streams it fitted. Lower `farrington.b` (at `b: 1` it needs 104 weeks) or wait for the history to accrue. - MEM runs at province and region streams (configurable via `mem.levels`), for pathogens whose own data show a seasonal pattern (or overridden per pathogen via `mem_mode`). If your extract does not run up to today, a run dated today sees a catchment that stopped weeks ago. It says so outright when the newest case on file is older than every rule-based lookback. Give `episodic_run_cron()` a `run_date` inside the extract's own window, and run it one date at a time if the dates a cluster was opened on need to mean anything - which is how `episodic_validate_detection()` replays a generated history week by week.
Can I try EpiSODIC out before pointing it at anything real? Yes, and this is the easiest way to get a feel for it. One call: ```r EpiSODIC::episodic_demo() ``` No data, no credentials, no configuration of your own required - it creates a temporary database, runs one detection cycle over bundled synthetic data, provisions a demo epidemiologist account (printed to the console), and opens the dashboard. Pass `launch = FALSE` if you just want a populated database path back for scripting, without opening the app. Nothing about this touches a real database or real data, so feel free to explore freely.
How do I change the dashboard and report language? Set the `EPISODIC_LANGUAGE` environment variable to one of `en`, `ar`, `nl`, `fr`, `de`, `hi`, `zh`, or `es` before starting the app (it defaults to `en` if unset): ```r Sys.setenv(EPISODIC_LANGUAGE = "es") episodic_run_app() ``` Two of those have a regional variant, which differ in the conventions rather than the words - what a decimal mark is, which way round a date runs: ```r Sys.setenv(EPISODIC_LANGUAGE = "es-419") # Latin America: 1,234.5 Sys.setenv(EPISODIC_LANGUAGE = "en-US") # January 7, 2025 ``` `en` is British English and `es` is Spain's Spanish, so `en-GB` and `es-ES` name those and are accepted as such. A region that is not shipped falls back to its language rather than to English - `nl-BE` renders in Dutch - and says so once. This covers the dashboard interface and the outbreak reports it generates alike. It is fixed for the whole running instance - there is no in-app language switcher - so if different people need different languages, that is a case for running more than one instance pointed at the same database.
How do I tune detection thresholds or priority score weights for my own organisation? Point `EPISODIC_CONFIG` at a YAML file with only the keys you want to change; it is merged key-by-key over the shipped defaults. Per-pathogen parameters (severity, episode length, MEM mode) are overlaid via `EPISODIC_PATHOGEN_CONFIG` the same way. See [**Tuning detection for your catchment**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/tuning-detection.html) for which parameters to adjust for common operational problems, their trade-offs, and the complete reference table.
I have more than one laboratory system or data source - can I combine them into one feed? Yes - EpiSODIC only ever sees the single data frame (or tibble) your own transform step hands to `episodic_run_cron()`, so combining several source systems is exactly that: your transform step's job, done before EpiSODIC is ever called. Bring each source into the shared column requirements, `rbind()` (or `dplyr::bind_rows()`) them together, and check the combined result with `episodic_check_cases()` as normal. The one thing to get right across sources is `source_key`: it needs to be unique *per row of the combined data set*, so if two systems could otherwise produce the same key (e.g. both use a simple incrementing lab accession number), prefix each system's keys with something that tells them apart (e.g. `"LIS1-482910"` vs. `"LIS2-482910"`) before combining, so a later re-run cannot collide two genuinely different results into one. `lab_number` itself needs no such treatment - unlike `source_key` it is never required to be unique, so your lab's own accession number can go into it verbatim regardless of how many systems you combine. See [**Getting your data in**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/data-format.html)'s "`source_key`: one identifier per *result*, not per specimen" section for the same idea applied within one source system, where a bare accession number repeats for other reasons (e.g. one culture yielding more than one reported result).
## Accounts and hosting
How do I create a new account? There is no self-service sign-up screen, and that is by design - only whoever administers your instance's database provisions accounts, which keeps sign-in tied to real board membership rather than being open-ended. If that is you, it is one function call: ```r Sys.setenv(EPISODIC_DB = "/path/to/episodic.sqlite") # or your MariaDB/MySQL DSN EpiSODIC::episodic_add_user( username = "jdoe", full_name = "Dr Jane Doe", role = "epidemiologist", # or "viewer" email = "j.doe@example.org", password = "a-temporary-password" ) ``` Pick any temporary password you like - the account is created with a forced password change, so the new holder is required to set their own private password the very first time they sign in, and the temporary one you typed never lingers as their real credential. Pass `role = "viewer"` instead of the default `"epidemiologist"` for someone who should be able to see everything, including patient-level detail, but should never record an assessment themselves. If someone forgets their password or needs their access removed, that is also a job for whoever administers the database (there is no self-service reset either) - reach out to them rather than trying to work around it yourself.
What is the difference between an "epidemiologist" and a "viewer" account, and how do I decide who needs which? `epidemiologist` is, by definition, the person who assesses clusters and classifies them - the core, shared job of being on the board. An epidemiologist can (and should) classify, close and mute clusters, and re-render reports on demand. `viewer` is read-only: it sees exactly what a signed-in epidemiologist sees, including patient-level line lists, but cannot record an assessment. In practice: `epidemiologist` is the right default for board members whose job is reviewing and classifying signals; reach for `viewer` for anyone who needs full visibility into cluster detail without a say in the classification itself (a data-analyst, a manager kept in the loop, an auditor, a colleague covering for the board without taking assessment responsibility). Neither role needs signing in just to see aggregate data - that is anonymous throughout; signing in as either role is what unlocks patient-level detail.
Must I run this on my own machine, or must I host it somewhere? Either is genuinely fine - EpiSODIC does not assume one or the other, and plenty of setups do each: - **On your own machine**: `episodic_run_app()` opens the dashboard locally, and you (or a scheduled task on that same machine) run `episodic_run_cron()` on whatever cadence suits you. This is the simplest way to get started, and exactly what `episodic_demo()` does under the hood. - **Hosted somewhere shared**: a server running `episodic_run_app()` behind your organisation's usual access setup, with `episodic_run_cron()` scheduled separately (cron, a Windows scheduled task, a CI pipeline - there is nothing EpiSODIC-specific about the scheduling mechanism itself). This is the natural choice once more than one person needs to reach the dashboard, since aggregate data is anonymous to anyone who can reach the app - signing in (as either role) only unlocks patient-level detail, and only an `epidemiologist` account can classify. Both point at the same kind of database (a local SQLite file, or a shared MariaDB/MySQL server via `EPISODIC_DB`), so you are not locked into your first choice - moving from "trying it out on my laptop" to "a shared instance the whole board uses" is a deployment change, not a data or code change. See [**Deployment**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/deployment.html) for the full walk-through.
Can more than one board member review the same cluster, and what happens if they disagree? Yes - any signed-in epidemiologist can open and classify any cluster; there is no assignment or locking that reserves one to a single person. Every classification is recorded as its own event on the cluster's audit trail, attributed to the account that made it, rather than overwriting a single "current" verdict - so if two people classify the same cluster differently, both decisions stay visible with who made them and when, rather than one silently replacing the other. In practice, most boards settle disagreements the same way they would for any shared clinical decision: talk it through and let whoever has the final say record the version that stands - EpiSODIC's job is to keep that history honest, not to referee it.
Is patient-identifying information ever shown in the dashboard or in outbreak reports? No, not as long as you have pseudonymised `patient_key` before it reaches EpiSODIC - which is your own transform step's job, not EpiSODIC's; a real name, national identification number, or hospital record number must never be passed through as this column. The pseudonym itself *is* shown, to a signed-in user, on a cluster's line list (alongside `lab_number`, your lab's specimen/culture number - also not itself a patient identifier), together with epidemiological context: sex, age, care line, institution/ward, specialism, postcode-level geography, and dates. That is enough to assess a cluster and to look a case up in your own source system if needed, but - given a properly pseudonymised `patient_key` - not enough to identify who is in it from the dashboard alone. See [**Getting your data in**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/data-format.html) for the exact column requirements, including what "pseudonymised" needs to mean for this guarantee to hold.
My organisation is outside the Netherlands - does the geography panel (the choropleth map) still work for us? Yes. EpiSODIC is not built for the Netherlands specifically - the bundled Dutch postcode data is one worked example, shipped so the demo and a first trial run show a real map out of the box, not a statement about who this package is for. Nothing about detection, reconciliation, or the rest of the dashboard assumes any particular country, coding system, or administrative structure; geography is entirely something you plug in. Two independent things are configurable, and both are optional: - **The choropleth itself** (`EPISODIC_GEO_DATA`): point it at your own `.rds` file holding an [`sf`](https://r-spatial.github.io/sf/) object with a `pc` column (matching whatever your own `pc` values are - postcodes, zip codes, municipality codes, anything) and a `geometry` column, and it is used instead of the shipped default. An optional second layer, `EPISODIC_GEO_DATA_OVERLAY`, can draw region outlines (provinces, counties, states, whatever is useful for orientation) on top. If you have no geographic reference data to hand at all, nothing breaks - the geography panel simply falls back to a plain bar breakdown by `pc` value, exactly as if this feature did not exist. - **L4 (province-level) stream detection**: separately from the map, the lattice's province level needs to know which of your postcodes belong to which province/region, via `EPISODIC_PC_PROVINCE_MAP` - a CSV with `pc` and `province_code` columns. The same mapping is what names the province beside each postcode in the dashboard's geography panel. Left unset, this also falls back to the same Dutch demo default, which only ever matches Dutch postcodes; outside that region L4 simply never has anything to detect on until you supply your own mapping (L1-L3 and L5 are unaffected either way). Set to a file that cannot be used, it stops the run and names the problem instead of falling back - a mapping you supplied and a mapping you did not are two different situations, and quietly treating the first as the second is how a lattice ends up built on somebody else's provinces. See [**Getting your data in**](https://certe-medical-epidemiology.github.io/EpiSODIC/articles/data-format.html)'s "Geographic reference data" section for the exact requirements these files need to satisfy.