Data Provenance in mtDNA Research

Data provenance answers a basic question: where did this result come from? For mitochondrial analysis, that path may cross instruments, file conversions, quality review, consensus construction, classification tools, tree building, and report generation.

The answer should be specific enough that another qualified person can identify the source bytes, the sample represented, the transformation performed, the settings used, and the output reviewed. “It came from the final folder” is not provenance.

Good provenance is both a scientific and an operational control. It supports investigation, review, reanalysis, and responsible sharing. It also exposes uncertainty: if a parameter, tool version, or manual decision was not recorded, the system should not imply otherwise.

Model a chain, not a pile of files

Names such as sample_final_v2_revised.fasta try to compress identity, state, and chronology into one string. They fail when two people use the same name, a file is renamed, or “final” is regenerated. Folder paths are useful navigation, but they are weak primary identifiers.

A more durable approach represents distinct entities and the relationships between them:

A project defines the working and access context.
A sample has a stable code and links to its source material.
A file record describes the uploaded object, including format, size, checksum, and storage location.
A pipeline definition describes an intended analysis graph and reference setting.
A run represents one execution of that definition.
Step executions record the inputs, outputs, statuses, times, and errors inside the run.
A report identifies the project or sample it summarizes and, where applicable, the run it reports.

This graph does not eliminate filenames. It stops asking filenames to carry the entire evidential burden.

Begin with a failure model

Provenance requirements become clearer when a team names the failures it needs to detect or reconstruct.

Sample swaps and identifier drift. An AB1 filename may disagree with a tube label or LIMS identifier. A perfectly preserved analysis history cannot repair an incorrect initial association. Intake procedures need a documented rule for resolving discrepancies and identifying who approved the mapping.

Silent replacement. A collaborator sends a corrected FASTA under the same name. Without a content digest, receipt time, and a new record or version decision, later analysts may not know which bytes were used.

Unrecorded transformation. Trimming, reverse complementation, consensus editing, or reference conversion happens outside the managed workflow. The derived sequence then appears without its method or operator.

Configuration drift. The same nominal pipeline runs against different tool builds, reference assets, or thresholds. Recording only the pipeline name hides a potentially meaningful difference.

Partial and failed execution. One branch of an analysis completes while a dependent step fails. A report assembled from whatever outputs exist can look complete unless status is part of the evidence.

Detached export. A PDF or table leaves the platform without identifiers that let a recipient reconnect it to the project, sample, run, and generation date.

Decide what must be captured

Identity and custody: project identifier, sample code, source organization or instrument where appropriate, uploader or importer, receipt time, and any approved remapping of identifiers.

File fixity and location: original filename, format, byte size, cryptographic checksum and algorithm, storage object key, upload time, and processing state. A checksum should be verified at defined checkpoints, not merely stored.

Transformation context: input identifiers, output identifiers, operation performed, parameters, reference sequence or phylogenetic build, software and version, execution time, status, and error details. Manual edits should record the before/after object and a reason.

Review and interpretation: QC evidence, acceptance or rejection decision, classifier call and confidence, conflict-resolution notes, reviewer identity, and review date. Automated output and scientific approval should be separate states.

Dissemination: report identifier, report type, generation time, creator, source run when known, and the exact dataset covered. If an export can circulate independently, include enough identifiers in the file itself to reconnect it.

The FAIR Guiding Principles are useful here. FAIR does not mean that every dataset must be public. It asks that data and metadata be findable, accessible under defined conditions, interoperable, and reusable. Rich provenance directly supports reuse, but access may still be restricted for ethical, contractual, or institutional reasons.

Put controls at lifecycle boundaries

Capture is most reliable when it occurs as work happens. A practical SOP defines checkpoints:

Receipt: assign the project and sample identifiers; preserve the received file; calculate and record a checksum; reconcile naming conflicts.
Ingest: confirm format, size, parser result, sample count, and error status; quarantine or investigate unexpected content.
QC: retain quantitative evidence and the acceptance rule; distinguish an automated trim suggestion from an approved sequence edit.
Analysis: create a new run rather than overwriting old output; bind parameters, reference choice, inputs, and software versions to it.
Review: record exceptions, conflict resolutions, and who made the decision.
Reporting: link the report to its covered samples and run where possible; label preliminary output clearly.
Retention or disposal: apply approved retention rules to database records, objects, exports, logs, and backups together.

The Digital Curation Centre’s Curation Lifecycle Model is a helpful framework for assigning these responsibilities across creation, preservation, access, reuse, and eventual disposition.

Questions for a provenance SOP

Which identifier is authoritative when the tube, instrument file, submission sheet, and collaborator email disagree?
Which source formats are preserved exactly as received, and which derived formats are generated?
At what points are checksums recomputed and compared?
How are reference sequences, tree builds, software versions, and analysis parameters recorded?
Which manual changes require a reason and a second review?
Can failed and cancelled runs be distinguished from completed runs?
What metadata travels with an exported report or dataset?
How are object deletion, database deletion, backups, and retention schedules reconciled?
How will the team test reconstruction of one published or handed-off result?

How GeneFlow represents provenance

GeneFlow uses the project as the access and organization boundary. Uploaded AB1, FASTA, FASTQ, and VCF records retain original name, kind, byte size, SHA-256 checksum, storage key, uploader, and processing status. Samples link to a source file and can hold a consensus sequence and structured metadata.

Pipeline definitions belong to projects and record a graph plus an rCRS or RSRS reference setting. A pipeline run links back to its definition and records status, creator, timing, input overrides, and errors. Each step execution records its own input, output, order, timing, status, and error. Reports belong to a project, may belong to a sample, and can optionally link to a pipeline run; generated report files are referenced by an object-storage key.

GeneFlow also keeps change history for selected records: projects, samples, pipeline definitions, and haplogroup consensus records. This is deliberately not described as a complete audit log. File records, memberships, runs, step executions, trees, and reports do not all carry the same history mechanism, and model history does not prove that an object was never altered outside the application.

Implementation checklist

Publish an identifier policy for projects, samples, files, runs, and reports.
Preserve source bytes separately from derived sequences and reports.
Record checksum algorithm as well as value in external manifests.
Require parameters, reference assets, and software versions for reproducible runs.
Make failed, partial, cancelled, and completed states visible.
Record manual decisions separately from automated results.
Include stable identifiers and generation dates in exports.
Back up both object data and relational metadata, then test joined recovery.
Perform a periodic “trace one result backward” exercise and document every missing link.

Provenance is successful when it survives the difficult questions, not when it produces the largest metadata table. Start with the decisions that must be defensible, connect each one to its evidence, and state plainly where the chain ends.