Object Storage for Genomics Workflows

Object storage keeps files as addressable objects with metadata rather than as blocks on one server filesystem. S3-compatible systems are commonly used for research data because they separate durable file storage from application and worker instances.

That separation fits analysis platforms well. Web processes can accept uploads, workers can retrieve the same objects for analysis, and report generators can write outputs without placing large binaries in database rows.

The architectural benefit is real, but “stored in S3” is not a complete data-management plan. S3 is an API model implemented by many providers and products, each with its own guarantees and optional controls. Security, retention, recoverability, and cost depend on configuration and operations.

Understand the object model

An object store usually has three concepts:

A bucket is an administrative container.
An object key identifies an object within that bucket.
The object contains bytes plus system or user metadata.

Keys may look like paths, but they are identifiers in a namespace, not necessarily directories on a filesystem. Applications should not infer scientific identity solely from a key such as project-7/final/sample.ab1. Keep project, sample, checksum, analysis, and retention metadata in managed records with explicit relationships.

The Amazon S3 conceptual overview describes buckets, objects, keys, versions, policies, and consistency. S3-compatible services generally implement the core API, but optional features and exact behavior vary. Verify your chosen provider rather than assuming feature parity from the word “compatible.”

Why genomics systems use object storage

Independent compute and storage. Application and worker instances can be replaced or scaled without moving the source dataset on their local disks.

API-based access. Services can use a common authenticated interface. Short-lived signed URLs can provide controlled downloads.

Large binary handling. AB1, FASTA, FASTQ, VCF, alignment output, and reports remain objects, while the database stores searchable metadata and relationships.

Operational policy options. Depending on the provider, operators may configure versioning, lifecycle transitions, retention features, replication, inventory, access logs, and encryption. These are capabilities to configure, not properties automatically inherited by every bucket.

Stable internal references. A generated, non-semantic key avoids collisions when two collaborators upload files with the same original name. The original name can remain display metadata.

Build a storage failure model

Public exposure. A bucket policy, access-control list, or signed URL is too broad or long-lived. Sequence data become retrievable outside the intended project boundary.

Credential compromise. A leaked application or operator credential permits bulk reads, overwrites, or deletion. Scope credentials to required actions and buckets, rotate them, and monitor their use.

Application and object drift. A database row exists but its object is missing, or an object remains after its row is deleted. This can happen when a multi-system operation partly fails. Reconciliation procedures need to detect both directions without deleting evidence automatically.

Accidental overwrite or deletion. A deterministic key is reused, an operator runs a bulk command against the wrong prefix, or application deletion is authorized by mistake. Unique keys reduce collision risk; versioning, retention controls, backups, and tested recovery address different parts of the problem.

Corrupt or incomplete transfer. An upload is interrupted or bytes differ from the source. Compare a cryptographic checksum from a trusted source with the stored object at defined checkpoints.

Ransomware or destructive automation. A credential with broad delete access can affect many objects quickly. Separate backup credentials and accounts, restrict destructive operations, and consider independently protected recovery copies.

Provider or region disruption. An object store can become unavailable even when data remain intact. Define acceptable recovery time, local workarounds, replication or backup strategy, and responsibility for declaring an incident.

Unexpected cost. Storage, requests, retrieval, egress, inventory, and replication have different cost models. A lifecycle rule can reduce cost but can also delay retrieval or delete data if designed poorly.

Feature mismatch. A deployment assumes AWS-specific behavior that another S3-compatible service does not implement. Test required controls against the actual endpoint and version.

Design the application boundary carefully

A sound pattern separates binary and relational responsibilities.

The object store holds source files and generated artifacts. The database holds the project, original filename, object key, byte size, checksum, media kind, uploader, processing status, sample association, run relationship, and report metadata. Workers fetch by object key rather than by an application-server path.

This design creates a distributed consistency problem: a database transaction cannot normally commit atomically with a remote object write or delete. Plan explicit states such as uploading, processing, ready, and error. Reconcile missing objects, unreferenced objects, stale uploads, and failed cleanup. Review destructive repairs when provenance is at stake.

Do not expose raw object keys as authorization. A user who guesses a key should not gain access. The application should verify project membership before issuing a short-lived download URL, and the bucket itself should remain non-public unless a documented use case requires otherwise.

Governance decisions belong outside the API

Object-storage lifecycle rules can implement policy, but should not invent it. Data stewards should decide:

Which objects are source data, working derivatives, released outputs, or disposable intermediates?
Which contractual, ethical, funder, or institutional requirements apply?
How long is each class retained, and who approves disposal?
Must deleted or replaced data remain recoverable for a defined period?
Which region and provider are acceptable for the dataset?
Who can read objects directly, bypassing the research application?
How will a project be exported in a usable form if the service changes?

The Digital Curation Centre’s Curation Lifecycle Model helps connect storage operations to appraisal, preservation, access, reuse, and disposal rather than treating all objects as one undifferentiated archive.

How GeneFlow uses storage

GeneFlow uses a local filesystem backend for development. Staging and production deployments can select an S3-compatible backend, including Wasabi or another service configured through an endpoint, region, bucket, and signed-request credentials. The application does not assign a public object ACL and uses authenticated URLs for downloads, but deployment-level bucket and account policies still determine actual exposure.

On file upload, GeneFlow calculates SHA-256 from the incoming stream and stores the object under a key containing the project ID, a generated UUID, and the original upload name. The database file record retains that storage key, original name, kind, byte size, checksum, uploader, and processing status. Parsers and analysis workers open the object through the configured storage backend.

Generated report files are stored under generated project-scoped keys, while report records retain project, optional sample, report kind, creator, structured payload, and optional pipeline-run relationship. Downloads use short-lived signed URLs where the backend supports them.

These relationships improve traceability, but GeneFlow does not claim that the configured object store is immutable, versioned, replicated, backed up, or periodically rehashed. Those controls are deployment decisions. File deletion also spans the database and storage service, so operators should monitor for orphaned records or objects and test recovery instead of assuming perfect cross-system cleanup.

Implementation checklist

Classify source, derived, report, temporary, and backup objects.
Use generated keys; retain original filenames as metadata.
Keep buckets private and test every application download path.
Give application and worker credentials only the actions and buckets they require.
Enable and test encryption settings appropriate to the deployment.
Decide explicitly whether versioning, retention, replication, and lifecycle features are required.
Confirm feature behavior with the actual S3-compatible provider.
Record byte size and a modern cryptographic checksum at ingest.
Verify stored and restored objects at documented checkpoints.
Back up relational metadata and objects as one recoverable system.
Test restoration of a project, including relationships and permissions.
Review access logs, credential age, public-access settings, storage growth, and lifecycle actions.
Document an exit procedure that exports objects, metadata, checksums, and relationships.

Object storage is a strong component for genomics infrastructure because it gives applications a shared, API-addressable home for binary data. It becomes responsible research storage only when technical configuration is joined to provenance, access review, retention decisions, fixity checks, backups, and tested recovery.