Research data management for software projects

Most research software exists to turn input data into output data, yet data is routinely the least managed part of the project: undocu- mented CSVs, 2GB files in git, results nobody can regenerate. Treat data as a first-class research output with the same care as code - versioned, documented, licensed and citable. FAIR applies to data even more directly than to software (rseng-fair-software covers the software side; the principles at go-fair.org are the shared root).

Project layout and the data/code boundary

• Separate raw, intermediate and final data explicitly - e.g. data/raw/, data/processed/, results/ - and treat raw data as READ-ONLY: nothing ever edits a raw file in place; all cleaning is scripted so the pipeline can regenerate everything downstream (rseng-workflows).
• Keep the mapping from data to code explicit: which script produces which file, recorded in the workflow or a Makefile, not in memory.
• Configuration, not hard-coded paths: data locations belong in a config file or environment variable so the project runs outside its author's laptop (rseng-reproducible-environments).

Keeping data out of git - but versioned

Git is for text; repositories bloat permanently with every committed binary revision. Before a large or binary data file lands in git, intervene:

• Small, stable reference data (kilobytes, test fixtures) may live in the repository - that is fine and convenient.
• For everything else use a data versioning layer: DVC or git-annex track content by hash in git while the bytes live in ordinary storage; DataLad builds full dataset management on git-annex and is widespread in neuroscience and beyond. All three keep code and data revisions linked, which is the actual requirement: "which data did commit X use?" must have an answer.
• Published, frozen inputs are often best NOT copied at all: record the DOI or URL plus a checksum, and fetch in a scripted step.
• Git LFS exists but suits media assets better than evolving research data; hosting quotas bite and history stays coupled to one forge.

Documenting a dataset

A dataset without documentation is a puzzle, not a resource. Minimum per dataset:

• A README (or datasheet) stating: what the data is, how it was collected or generated, its units, coordinate systems and conventions, known limitations, license, and how to cite it.
• A data dictionary for tabular data: every column's name, type, units, allowed values and meaning. Generate it from the data where possible so it cannot drift silently.
• Formats: prefer open, well-specified formats (CSV with a stated dialect, Parquet, HDF5, NetCDF, domain standards) over proprietary ones; for domain metadata standards and format registries, point users to FAIRsharing and the ELIXIR RDMkit rather than inventing a schema ad hoc.

Depositing, DOIs and citation

Data that supports a publication belongs in a data repository, not in supplementary ZIPs or the code repo:

• Zenodo is the general-purpose default (free, DOI per version, versioned records); domain repositories are better when one exists - RDMkit and FAIRsharing list them per field.
• Give the dataset its own DOI and its own citation entry; link data DOI and software DOI both ways so each cites the other (rseng-citation-metadata, rseng-publishing-releasing).
• License the data explicitly - and note that code licenses do not fit data well: CC0 or CC-BY are the common choices for open data, while ODbL and domain-specific terms exist for databases (rseng-licensing). "No license" means "all rights reserved", for data exactly as for code.

Sensitive and personal data

When data involves people, patients, protected locations or commercial restrictions:

• Never commit sensitive data, even briefly - git history is forever and forges cache aggressively. Add ignore rules and pre-commit guards BEFORE the first sensitive file exists in the project.
• Keep sensitive data in the access-controlled storage the institution provides; the repository carries only synthetic or anonymized samples plus the code that ran on the real thing.
• Anonymization is a research task, not a rename: removing names is not de-identification. Route the user to their data steward or ethics board rather than improvising GDPR compliance.
• A data management plan (DMP) may already govern the project - ask; when one exists it decides storage, retention and sharing, and the software should implement it rather than contradict it. Writing and maintaining DMPs is rseng-data-management-plans.

Records and transparency

When AI assistance produced or transformed datasets, record it in the project's AI declaration - dataset entries are a supported content type in aidecl.yaml (rseng-ai-declaration). Data provenance and AI provenance are the same habit applied to different artifacts.

Working with this skill

This skill is source-independent: its authority is the FAIR data principles and the community resources linked below.

Learn more (verified):

• https://www.gofair.foundation/fair-principles - the FAIR principles
• https://rdmkit.elixir-europe.org - ELIXIR RDMkit, per-domain and per-task RDM guidance
• https://fairsharing.org - registry of metadata standards and data repositories
• https://zenodo.org - general-purpose data repository with DOIs
• https://dvc.org - Data Version Control
• https://www.datalad.org - DataLad dataset management

Related skills

Check whether any of these applies before moving on:

• rseng-archiving - long-term data deposit
• rseng-citation-metadata - data DOIs and two-way citation
• rseng-data-management-plans - funder plan over the practice
• rseng-regulatory-compliance - sensitive and personal data obligations
• rseng-scientific-file-formats - choosing and engineering the format
• rseng-workflows - scripted regeneration of derived data