Importing and exporting data between instances

Setting up: identity and keys

Every EpiSODIC database has an instance identifier: ten hexadecimal characters, drawn from a cryptographic random source when the database is created and never changed afterwards. It is shown on the dashboard’s Info screen, and read headlessly with:

episodic_instance_id()
#> [1] "3fa9c01b2e"

Each instance then generates its keys, once:

keys <- episodic_import_export_keygen("/etc/episodic/keys")

This writes two key pairs:

The secret key files are written readable by their owner only. Windows keeps no such permission bits, so there, keep them in a directory only the account that runs EpiSODIC can read. Keys never belong in the configuration file: a configuration is exported and stored, and whoever reads a secret key there can read, or forge, every file.

Give each partner your two .pub files and your instance identifier. Before relying on a partner’s public keys, compare their fingerprints with the partner over another channel (a telephone call will do):

episodic_import_export_key_fingerprint("partner_signing.pub")

If both sides read the same fingerprint, the file was not swapped on its way to you.

Generating keys again replaces them. Files encrypted to the old public key can then no longer be opened, and partners must pin the new signing key, which is why episodic_import_export_keygen() refuses to overwrite unless told to.

Exporting

episodic_cases_export(
  from = "2026-01-01",
  to = "2026-03-31",
  file = "/srv/episodic/to-partner.episodic",
  recipients = c("keys/partner_a_encryption.pub", "keys/partner_b_encryption.pub")
)

This writes the cases with a sample date in the range, encrypted to every recipient listed and signed with the key in EPISODIC_EXPORT_SIGNING_KEY. One file can serve several recipients; each opens it with their own secret key, and none of them can forge a file in your name.

There are two ways to encrypt, and the first is strongly preferred:

• Public keys (recipients). Only the recipients’ secret keys open the file. Replacing one partner’s keys involves nobody else.
• A passphrase (passphrase). The file key is derived from the passphrase with scrypt and a random salt per file. Everyone who knows the passphrase can read every file encrypted with it, and changing it means coordinating every party at once. A passphrase must have at least 16 characters, since a file in a public place can be guessed at offline for as long as anyone likes.

Both modes sign the file, and an importer refuses it unless your public signing key is among the senders it has pinned.

export:
  fields: [receipt_date, care_line, municipality, ward, specialism, pc, sex, age]
  pc_characters: ~

Importing

partner <- episodic_cases_import(
  "/srv/episodic/from-partner.episodic",
  trusted_senders = "keys/partner_signing.pub"
)

episodic_run_cron(cases = my_cases, imports = partner)

episodic_cases_import() opens the file with the key in EPISODIC_IMPORT_KEY (or the passphrase), verifies it in full, and returns its cases as a validated data frame in the case data format. It does not write to the database. The case tables belong to the detection run, so imported cases enter the way all cases do, through episodic_run_cron(), with the same checks, deduplication and audit trail, and each import is recorded in episodic_import_event against the run that consumed it.

What the import does to the cases:

• source_key, patient_key and lab_number are prefixed with the sender’s instance identifier and a colon (3fa9c01b2e:12345). Your 12345 and a partner’s 12345 are different results, different patients and different cultures; without the prefix, the partner’s result would be skipped as already present, and two different patients would be merged into one.
• institution_key is used as the stored hash the file declares it to be, never hashed again.
• The cases are stamped with their origin (episodic_case.origin_instance_id), which is what keeps them out of your own exports.

Nothing in a file is returned unless all of it verifies. The import stops with an error, of class episodic_import_export_error with a reason code, on any of these:

A run also refuses an import of its own instance’s export (those cases are already there), an import whose keys no longer carry their sender’s prefix, and one of your own source_key values that collides with an imported one. Write your own keys without a <ten hexadecimal characters>: prefix, or they cannot be told apart from a partner’s.

Scheduling: the Import and export screen

An admin account reaches the Import and export screen from its card on the Instance screen. It shows the instance identifier, offers a download of an export (built with the dashboard process’s own EPISODIC_EXPORT_SIGNING_KEY), and lists the export schedules, the import sources and the audit trail of every export and import. Non-admin accounts see neither the card nor the screen.

Export schedules are carried out by episodic_run_cron() after its detection transaction has committed, the way scheduled reports are. Each covers a rolling window, the last N days up to the run date, every M days; the window must be at least as long as the interval, and is best somewhat longer, so consecutive exports overlap. The overlap is harmless: imports are append-only on prefixed keys, so a case carried by two files is taken in once. An export that fails is recorded and logged and never fails the run.

Import sources are fetched by every run before detection, and their cases enter in the same transaction as your own. A source that fails (unreachable, a wrong key, a forged file) is recorded with its reason and marks the run partial, with the source and the reason on the run row; the run goes on with everything else. It is never skipped in silence.

The dashboard never imports cases itself. A source is a record of where the next run fetches from; the run takes the cases in.

Nothing secret is stored. A schedule or source names the environment variable that holds each key, passphrase, password, token or secret URL, and the variable is read by the process that uses it, at the moment it uses it. Set those variables in the environment the scheduled run starts in; the dashboard’s own environment matters only for its download button and its connection test.

The connection test checks a schedule or source the way the next run will use it, and writes nothing. For an import source it reads the key variable, fetches the file and imports it in full, so a passing test means the next run takes it in. For an export schedule it reads the signing key and checks the destination without writing to it, since that would replace the file partners fetch: a directory must exist and be writable, and an HTTP(S) destination is asked whether it answers and whether it refuses the credentials outright. Whether an upload is accepted is known only from the first real export, and the test says so. The same test runs headless:

episodic_import_export_test_connection("Partner lab", kind = "source")

Transport

A file path always works. Beyond it, HTTP(S) through the httr2 package: PUT to upload, GET to fetch. That covers WebDAV (Nextcloud, ownCloud), a pre-signed Amazon S3 URL, an Azure Blob SAS URL and any web server that accepts a PUT. SharePoint, OneDrive and SFTP are not supported.

Authentication is HTTP basic (a user name variable and a password variable, as Nextcloud app passwords use) or a bearer token variable. A URL stored in a schedule or source may carry no credentials: no user:password@ and no query string, where a pre-signed URL keeps its signature. A URL that is itself a secret is named by its variable instead (env:PARTNER_URL in the audit trail), and errors never repeat it.

Name the published file neutrally. The file’s content reveals nothing, but its name is the first thing anyone sees.

What importing and exporting do not solve

These are limits of importing and exporting, stated so they are not discovered later.

Patients cannot be linked across laboratories. Each laboratory pseudonymises its patients its own way, so the same patient tested at both becomes two patients after an import, and episode deduplication cannot merge them. Cross-instance duplicates are therefore possible and inflate counts. Only a jointly agreed pseudonymisation, such as a keyed hash both laboratories apply with a shared secret before their data reaches EpiSODIC, would solve this, and that is an organisational decision rather than something EpiSODIC can do.

Institutions are matched by key. The stored hash of an institution’s key is the same at every instance, so two laboratories that use the same key for the same hospital, such as a national registry code, see one hospital. Two that happen to use the same internal code for different institutions would see one as well. Agree on a shared institution coding, or keep the codes apart. An import may add an institution the receiver has never seen, but never renames or retypes one it already knows.

Pathogen names are used verbatim. pathogen is free text, so "Norovirus" at one laboratory and "norovirus GII" at another become two separate streams at the receiver. Agree on names with your partners, or map the partner’s names in your own extract step before the run. For bacteria and fungi, the AMR package’s AMR::as.mo() and AMR::mo_name() normalise names to one taxonomy; AMR does not cover viruses, so viral pathogen names need an agreed list.

Imports are append-only. A case is never updated once it is on file. If the sender later corrects a case, the receiver keeps the version it first imported; if the sender retracts one, it remains at the receiver. Re-exporting a range does not replace what an earlier file delivered.

History imported after the first run is baseline only. The first detection run on a database reports the whole case history it is given (a backfill). Every later run reports only what falls inside the detectors’ lookback windows. A partner’s history imported after your first run therefore serves as baseline for Farrington and MEM, and its past clusters are not reported. Import a partner’s history in your first run if you want its past clusters in your archive.

Data protection

Pseudonymised case-level health data remains personal data under the GDPR, encrypted or not, and storing an encrypted file in a public place is still a processing operation. Encryption makes the storage location safe; it does not replace a data-sharing agreement or a legal basis for sharing the data, which the parties must have in place before the first file is sent.

Each export supports data minimisation: leave out the columns the partner does not need, and cut postcodes to the precision the shared purpose requires. The file records what was left out, so the receiver knows what it does not have. The audit trail at both ends, the export record at the sender and the import record at the receiver, documents who shared what with whom, when, and with which keys.

Keep the secret key files and every variable named by a schedule or source with the same care as the case data itself, and revoke a compromised key by generating new keys and sending partners the new public keys.

File format

An export file is a versioned envelope around a plain, documented payload. The payload is deliberately not an R serialisation: unserialize() on bytes from another party is a code-execution risk whatever signature checked them, and an RDS file is unreadable outside R.

Envelope, format version 1. All integers are unsigned and big-endian.

Nothing else is in plaintext. The sender, the range, the row counts and the schema version are all inside the encrypted payload.

Payload, version 1. A zlib stream (memCompress(type = "gzip")) of one UTF-8 JSON document:

sha256 is the SHA-256 of data’s exact UTF-8 bytes, checked for every table before any table is read. A newer EpiSODIC reads older files; an older EpiSODIC refuses a newer envelope or payload by name rather than misreading it.

See also

• Getting your data in for the case data format an import returns, and the geographic reference data that decides where a partner’s cases can be placed.
• Deployment for scheduling episodic_run_cron(), which fetches the import sources and carries out the export schedules, and for admin accounts.
• Environment variables for EPISODIC_IMPORT_KEY, EPISODIC_EXPORT_SIGNING_KEY and every other EPISODIC_* variable.
• Tuning detection for your catchment for the export.fields and export.pc_characters configuration keys.
• ?episodic_cases_export and ?episodic_cases_import for the API-level documentation.
• ?episodic_import_export_test_connection for testing a schedule or source headless.