How clean is OpenNeuro's metadata?

Where the problems are

Of the 148 raw datasets, 76 (51%) have no warnings or errors, 68 have at least one warning, and 10 (7%) have an error that breaks the standard. Most findings are about consistency across files (participants against data folders, sidecars against each other), which a check of one file at a time can miss.

Participants are the weak spot

28 datasets (19%) have no participants.tsv at all. The file is only RECOMMENDED by BIDS, but without it age, sex and group are simply not available to anyone reusing the data. Where the file does exist, 10 datasets list people with no data in the release, 5 have data folders for subjects the table never mentions, and 5 write participant_id without the required sub- prefix.

Small formatting slips are enough to break a join. In ds004226 the two sides differ only in zero padding. In ds001838 the table also drops the prefix, so its unpadded IDs match nothing:

Hard errors are uncommon and concentrated

All errors come from 10 datasets. In 4 of them the only error is the missing sub- prefix above. The rest:

• ds004166: 213 EEG recordings with no sidecar at all, so fields BIDS requires (sampling rate, power-line frequency, task name) are undefined.
• ds004348: 9 EEG recordings with no sidecar at all, so fields BIDS requires (sampling rate, power-line frequency, task name) are undefined.
• ds005065: 3 MEG recordings with no sidecar at all, so fields BIDS requires (sampling rate, power-line frequency, task name) are undefined.
• ds005697: its one EEG sidecar is misnamed (sub-16task-..., missing an underscore), so no sidecar applies to the recording.
• ds001541: participant IDs without the prefix, and the same participant listed on several rows.
• ds003012: a participants.tsv row with a trailing tab, one column more than the header.

Each of these is cheap to fix and expensive to discover late, which is the case for checking metadata in a pipeline rather than at analysis time.

Same task, different acquisition

11 datasets mix acquisition parameters within a single task and acquisition label: repetition times for fMRI, or sampling rates for EEG, MEG and iEEG (differences under 0.5% are ignored as rounding). None of this breaks BIDS, and much of it is deliberate. Multi-site and clinical recordings vary by scanner or hospital. It still matters, because a pipeline that reads one TR or one sampling rate per task will be wrong for some runs.

The largest spread is in ds005003, where resting-state runs use 12 different TRs (2.16 to 3.53 s). In the iEEG dataset ds005522, recordings of a single task range from 500 to 2000 Hz.

.bidsignore files that ignore too much

A .bidsignore tells the validator which files to skip. In 5 datasets a pattern also matches core files such as the README or the dataset description. neurolake never ignores core files and reports the pattern instead.

• ds004103 ends its ignore file with *, which matches every file. All of its data sits outside the checks, so it is reported once rather than counted as participants with no data.
• ds006833 uses **/*dat*, which also matches dataset_description.json.

Are newer datasets cleaner?

Datasets whose latest snapshot is from 2024 or later are clean 61% of the time (47 of 77), against 41% for older ones (29 of 71). Fisher's exact test gives p = 0.02, but treat that as suggestive. An earlier version of the rules gave p = 0.07 on the same sample, so the result depends on which rules count, and a snapshot's year is when the dataset was last updated, not first published. The share without participants.tsv falls too (p = 0.06).

Do datasets get cleaner after release?

The year comparison above uses each dataset's latest snapshot, which mixes when a dataset was published with how much it was revised. To separate the two I ran the same rules on the first published snapshot of every dataset in the sample. 88 of the 150 have been updated at least once, a median of 0.4 years apart, and all 88 are raw datasets.

Updates rarely touch the metadata. In 72 of the 88 updated datasets every finding is the same in the latest snapshot as in the first. The share that is fully clean moves from 53% to 58% (47 to 51), and the number with errors is 6 at both ends: 1 fixed, 1 introduced. Datasets that ended with fewer findings outnumber those that ended with more, 9 to 5, which is well within chance (sign test p = 0.42).

Counting individual findings tells a different story: 1,344 resolved against 46 introduced. But 1,275 of those come from one dataset, ds003768: 255 recordings that each lacked 5 required sidecar fields in its first snapshot (1.0.0) and have them all by 1.0.13. Some improvements are not metadata fixes. In 4 of the 16 datasets whose findings changed, the latest snapshot has at least half again as many data files as the first, which points to an unfinished first upload, and in 2 of them the gain is listed participants that simply received their data.

Updates also break things. By 1.0.4, ds004103 has a .bidsignore that covers every data file, so none of its recordings are checked any more. Its first snapshot (1.0.0) had no such pattern. ds001619 has fewer data files by 1.0.1, and 7 people still listed in its participants.tsv no longer have any data.

The practical reading: problems present at first upload tend to stay, so the cheapest place to catch them is before the first snapshot is published.

Cross-check with the official validator

To see where these rules stand against the reference tool, I ran the official bids-validator on the same 150 snapshots. It is the legacy version 1.15.0 from npm, not the current schema-based 2.x, which is distributed through Deno and could not be installed here. Because imaging data is never downloaded, it ran in git-tree mode on the shallow clones (--gitTreeMode) with NIfTI header checks switched off (--ignoreNiftiHeaders), so its header checks are excluded from everything below.

The validator could assess 144 of the 150 datasets. It stopped early on the rest (INCOMPLETE_DATASET in 4, QUICK_VALIDATION_FAILED in 1, crashed without JSON output in 1). Agreement is computed over those 144 only: a dataset the validator could not assess never counts as agreement.

11 neurolake rules have a validator check that tests the same thing (audit/validator_map.csv). On those, the two tools flagged 25 dataset-check pairs between them and agreed on 8 (32%). Leaving out the participants.tsv vs subject folders check, where validator 1.15.0 never emits PARTICIPANT_ID_MISMATCH, they agree on 7 of 13 (54%). Each disagreement is classified by rules in audit/validator.py, and 5 of the 17 disagreements were checked by hand.

15 disagreements in 13 datasets are problems neurolake reports and validator 1.15.0 does not. 11 datasets on participants.tsv vs subject folders covering 215 participant IDs (some people appear twice, once per ID spelling). 4 datasets on required sidecar field missing covering 226 recordings. The recordings have no sidecar at all: validator 1.15.0 checks EEG, iEEG and MEG sidecar files against its schema but does not require that such a recording has one.

The participants check is worth a note. A synthetic dataset whose participants.tsv and subject folders disagree passes validator 1.15.0 without PARTICIPANT_ID_MISMATCH (run in normal and git-tree mode), while the same file's missing sub- prefix is reported, so the table was read. The check is unreachable in 1.15.0: fullTest declares const participants = null (dist/commonjs/cli.js:137602, dist/commonjs/index.js:133003) and passes it to the participants check. Its silence on these datasets says nothing about them.

The other way, 1 disagreement is a problem the validator reports and neurolake misses. ds002841: neurolake does not parse task-masstask_run-01_events.tsv.

.bidsignore semantics (1). ds006833: .bidsignore matches dataset_description.json: the validator ignores it, neurolake never ignores core files.

None of the 17 disagreements comes from the two tools following different versions of the specification.

Outside the paired checks the two tools look at different things. 9 neurolake rules with findings here have no validator counterpart: No participants.tsv (28 datasets), Recommended sidecar field missing (26 datasets), Sampling rates differ within a task (6 datasets), .bidsignore matches core files (5 datasets), TRs differ within a task (5 datasets), Key repeated in a JSON file (3 datasets), Two sidecars apply at one level (1 dataset), .bidsignore hides all data (1 dataset), Duplicate participant_id (1 dataset). In the other direction, the validator raised 11 kinds of error with no neurolake rule, most often JSON_SCHEMA_VALIDATION_ERROR other than a missing required field (16 datasets), NOT_INCLUDED (10 datasets), TSV_EMPTY_CELL (6 datasets), MULTIPLE_COMMAS_IN_AUTHOR_FIELD (3 datasets). neurolake has no rule for any of these.

How this was done

• Sample. Accessions ds000001 to ds007500 were shuffled with a fixed seed (2026) and probed in that order against OpenNeuro's DataLad mirrors on GitHub. The sample is the first 150 with a snapshot tag (650 probes). Every probe is logged, transient errors are retried rather than counted as missing, and the same seed reproduces the same list. The median dataset has 28 subjects (middle half 17 to 52, largest 608).
• Data. Each dataset's latest snapshot was shallow-cloned. Metadata files live in git, while imaging data stays as git-annex pointers whose keys carry the file size, so no imaging bytes were downloaded.
• Rules. Sidecar field levels come from the BIDS schema, and the inheritance principle follows the specification text, including the rule that only one sidecar may apply per folder level. That rule is reported as informational for datasets declaring a version before 1.7.0, when the inheritance principle was rewritten. The TR, sampling-rate and .bidsignore rules are heuristics of my own. .bidsignore is honoured, and derivative datasets only get file-integrity checks.
• Checks on the checker. Running on real data exposed false positives in the first version. 9 such patterns are now test fixtures that must never be flagged. An independent pass then reproduced the sample and every headline number from scratch, and checked 14 findings against the raw files. All 14 were real. It corrected one description and found one .bidsignore artifact, both fixed here, and found no missed violations in five clean datasets.

Reproduce it

pip install -e .
neurolake discover-mirrors --sample 150 --max-id 7500 --seed 2026 \
    --log-file audit/probe_log.csv > audit/sample_ids.txt
neurolake ingest-mirror --from-file audit/sample_ids.txt --lake data/audit
neurolake transform --lake data/audit
python audit/analyze.py data/audit
python audit/render.py
make history   # first vs latest snapshots
make validator-run validator   # cross-check with bids-validator

Code, rules and tests: github.com/pearlfranz20/neurolake.