Currently edited online in https://hackmd.io/z6GJrRSdRySMo1LAppabJw
Session Chair: Yaroslav Halchenko (Dartmouth College) and Thomas Braun (byte physics e.K.)
Needs, requirements and expectations for validation and introspection of NWB files
In principle, any stored by PyNWB/MatNWB file should pass this stage of validation
Validate toward “best practices”. nwb-inspector ATM is the implementation for some of such:
Some of those could be addressed by more restricted data types, e.g. ISO datetime duration for “age” or any other “duration”.
Catch obviously (to human curator) eye incorrectly entered/encoded data (e.g. incorrect unit, e.g. V vs mV). ATM not possible to encode range within NWB schema, to e.g. encode the range.
Question of the units: SI with multipliers. Satra mentioned https://people.csail.mit.edu/jaffer/MIXF/MIXF-10 which is character string encoding for numerical values and units, incorporates SI, etc.
Participants should add topics of possible interest for discussion here
(Ben) Ontologies could not be introduced for a field without breaking compatibility (i.e. forcing more than free form string) by adding a supplementary “Ontologies” table with fields
pynwb provides validation of .nwb files to follow specific schema. Could also be invoked from command line via
python -m pynwb.validate [FILES] but at best should be invoked per each file separately since the whole process would fail as soon as the current file fails validation.
validate (and also during
upload) provide a way to validate files (delegates actual validation to pynwb) from command line:
$> dandi validate v2.0.1/test_* ... v2.0.1/test_CurrentClampStimulusSeries.nwb: ok v2.0.1/test_DecompositionSeries.nwb: 1 error(s) DynamicTable/colnames (processing/test_mod/LFPSpectralAnalysis/bands.colnames): incorrect type - expected 'text', got 'ascii' v2.0.1/test_Device.nwb: ok v2.0.1/test_DynamicTable.nwb: 1 error(s) Units/colnames (units.colnames): incorrect type - expected 'text', got 'ascii' v2.0.1/test_ElectricalSeries.nwb: 1 error(s) DynamicTable/colnames (general/extracellular_ephys/electrodes.colnames): incorrect type - expected 'text', got 'ascii' v2.0.1/test_ElectrodeGroup.nwb: ok ... Summary: Validation errors in 14 out of 31 files
ls provides easy way to see basic information about any .nwb file. For pre-v2 nwb files it would at least provide NWB version information. E.g.:
$> dandi ls v2.0.1/test_Cluster* PATH SIZE IDENTIFIER SESSION_DESCRIPTION SESSION_START_TIME NWB ND_TYPES v2.0.1/test_Clustering.nwb 17.8 kB TEST_Clustering a file to test writing and reading a Clustering 1971-01-01/12:00:00 2.0b Clustering v2.0.1/test_ClusterWaveforms.nwb 20.0 kB TEST_ClusterWaveforms a file to test writing and reading a ClusterWaveforms 1971-01-01/12:00:00 2.0b ClusterWaveforms, Clustering (2) Summary: 37.7 kB 1971-01-01/12:00:00> 1971-01-01/12:00:00<