LEMON dataset example#

This example illustrates the use of sovabids on the LEMON dataset using both the python API and the CLI tool.

The main elements of this example are:
  • A source path with the original dataset.

  • A bids path that will be the output path of the conversion.

  • A rules file that configures how the conversion is done.

  • A mapping file that encodes how the conversion is performed to each individual file of the dataset.

graph LR S>"Source path"] B>"Bids path"] R>"Rules file"] AR(("Apply Rules")) M>"Mappings file"] CT(("Convert Them")) O[("Converted dataset")] S --> AR B --> AR R --> AR AR --> M M --> CT CT --> O

Using the python API#

First we will illustrate how to run the software within python.

Imports#

First we import some functions we will need:

import os # For path manipulation
import shutil # File manipulation
from mne_bids import print_dir_tree # To show the input/output directories structures inside this example
from sovabids.rules import apply_rules # Apply rules for conversion
from sovabids.convert import convert_them # Do the conversion
from sovabids.datasets import lemon_prepare # Download the dataset
from sovabids.settings import REPO_PATH

Getting and preparing the dataset#

We have to download and decompress the dataset. We also need to fix a filename inconsistency (without this correction the file won’t be able to be opened in mne). Luckily all of that is encapsulated in the lemon_prepare function since these issues are not properly of sovabids.

By default the files are saved in the ‘_data’ directory of the sovabids project.

Downloading sub-032301.tar.gz at /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon
WARNING: File already existed. Skipping...
Downloading sub-032302.tar.gz at /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon
WARNING: File already existed. Skipping...
Downloading sub-032303.tar.gz at /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon
WARNING: File already existed. Skipping...
Downloading name_match.csv at /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon
WARNING: File already existed. Skipping...
LEMON PREPARE DONE!

Setting up the paths#

Now we will set up four paths. Because this example is intended to run relative to the repository directory we use relative path but for real use-cases it is easier to just input the absolute-path. We will print these paths for more clarity.

source_path = os.path.abspath(os.path.join(REPO_PATH,'_data','lemon')) # For the input data we will convert
bids_path= os.path.abspath(os.path.join(REPO_PATH,'_data','lemon_bids')) # The output directory that will have the converted data
rules_path = os.path.abspath(os.path.join(REPO_PATH,'examples','lemon_example_rules.yml')) # The rules file that setups the rule for conversion
mapping_path = os.path.abspath(os.path.join(bids_path,'code','sovabids','mappings.yml')) # The mapping file that will hold the results of applying the rules to each file

print('source_path:',source_path.replace(REPO_PATH,''))
print('bids_path:', bids_path.replace(REPO_PATH,''))
print('rules_path:',rules_path.replace(REPO_PATH,''))
print('mapping_path:',mapping_path.replace(REPO_PATH,''))
source_path: /_data/lemon
bids_path: /_data/lemon_bids
rules_path: /examples/lemon_example_rules.yml
mapping_path: /_data/lemon_bids/code/sovabids/mappings.yml

Cleaning the output directory#

We will clean the output path as a safety measure from previous conversions.

try:
    shutil.rmtree(bids_path)
except:
    pass

The input directory#

For clarity purposes we will print here the directory we are trying to convert to BIDS.

print_dir_tree(source_path)
|lemon/
|--- name_match.csv
|--- sub-032301.tar.gz
|--- sub-032302.tar.gz
|--- sub-032303.tar.gz
|--- sub-010002/
|------ RSEEG/
|--------- sub-010002.eeg
|--------- sub-010002.vhdr
|--------- sub-010002.vmrk
|--- sub-010003/
|------ RSEEG/
|--------- sub-010003.eeg
|--------- sub-010003.vhdr
|--------- sub-010003.vmrk
|--- sub-010004/
|------ RSEEG/
|--------- sub-010004.eeg
|--------- sub-010004.vhdr
|--------- sub-010004.vmrk

Making the rules#

The most important and complicated part of this is making the rules file, either by hand or by the “DISCOVER_RULES” module (which is not yet implemented).

This part is already done for you, but for clarification here are the rules we are applying. Please read the following output as the yaml has some basic documentation comments.

See the Rules File Schema documentation for help regarding making this rules file.

with open(rules_path,encoding="utf-8") as f:
    rules = f.read()
    print(rules)
entities:                                       # Configuring the file name structure of bids
  task : resting                                # Setting the task of all files to a fixed string

dataset_description:                            # Configuring the dataset_description.json file
  Name : Lemon                                  # Name of the dataset, set up as a fixed string
  Authors:                                      # Here I put the personnel involved in the acquisition of the dataset
    - Anahit Babayan                            # See http://fcon_1000.projects.nitrc.org/indi/retro/MPI_LEMON.html
    - Miray Erbey
    - Deniz Kumral
    - Janis D. Reinelt
    - Andrea M. F. Reiter
    - Josefin Röbbig
    - H. Lina Schaare
    - Marie Uhlig
    - Alfred Anwander
    - Pierre-Louis Bazin
    - Annette Horstmann
    - Leonie Lampe
    - Vadim V. Nikulin
    - Hadas Okon-Singer
    - Sven Preusser
    - André Pampel
    - Christiane S. Rohr
    - Julia Sacher1
    - Angelika Thöne-Otto
    - Sabrina Trapp
    - Till Nierhaus
    - Denise Altmann
    - Katrin Arelin
    - Maria Blöchl
    - Edith Bongartz
    - Patric Breig
    - Elena Cesnaite
    - Sufang Chen
    - Roberto Cozatl
    - Saskia Czerwonatis
    - Gabriele Dambrauskaite
    - Maria Dreyer
    - Jessica Enders
    - Melina Engelhardt
    - Marie Michele Fischer
    - Norman Forschack
    - Johannes Golchert
    - Laura Golz
    - C. Alexandrina Guran
    - Susanna Hedrich
    - Nicole Hentschel
    - Daria I. Hoffmann
    - Julia M. Huntenburg
    - Rebecca Jost
    - Anna Kosatschek
    - Stella Kunzendorf
    - Hannah Lammers
    - Mark E. Lauckner
    - Keyvan Mahjoory
    - Natacha Mendes
    - Ramona Menger
    - Enzo Morino
    - Karina Näthe
    - Jennifer Neubauer
    - Handan Noyan
    - Sabine Oligschläger
    - Patricia Panczyszyn-Trzewik
    - Dorothee Poehlchen
    - Nadine Putzke
    - Sabrina Roski
    - Marie-Catherine Schaller
    - Anja Schieferbein
    - Benito Schlaak
    - Hanna Maria Schmidt
    - Robert Schmidt
    - Anne Schrimpf
    - Sylvia Stasch
    - Maria Voss
    - Anett Wiedemann
    - Daniel S. Margulies
    - Michael Gaebler
    - Arno Villringer

sidecar:                                        # Configuring the sidecar eeg file
  PowerLineFrequency : 50                       # Noted from the visual inspection of the eeg spectrum
  EEGReference : FCz                            # As mentioned in https://www.nature.com/articles/sdata2018308

channels:                                       # Configuring the channels tsv
  type :                                        # This property allow us to overwrite channel types inferred by MNE
    VEOG : VEOG                                 # Here the syntax is <channel name> : <channel type according to bids notation>
    F3 : EEG                                    # Here we set the type of F3, it was already correctly inferred by mne but it is included to illustrate retyping of various channels.
non-bids:                                       # Additional configuration not belonging specifically to any of the previous objects
  eeg_extension : .vhdr                         # Sets which extension to read as an eeg file
  path_analysis:                                # Some bids properties can be inferred from the path of the source files
    pattern : RSEEG/sub-%entities.subject%.vhdr # For example here we extract from the path the "subject" child of the "entities" object

Applying the rules#

We apply the rules to the input dataset by giving the input,ouput,rules, and mapping paths to the apply_rules function.

This will produce by default a ‘mappings.yml’ file at the specified directory of ‘bids_path/code/sovabids’.

This file holds the result of applying the rules to each of the dataset files.

apply_rules(source_path,bids_path,rules_path,mapping_path)
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010002/RSEEG/sub-010002.vhdr...
Setting channel info structure...
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010003/RSEEG/sub-010003.vhdr...
Setting channel info structure...
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010004/RSEEG/sub-010004.vhdr...
Setting channel info structure...

{'General': {'entities': {'task': 'resting'}, 'dataset_description': {'Name': 'Lemon', 'Authors': ['Anahit Babayan', 'Miray Erbey', 'Deniz Kumral', 'Janis D. Reinelt', 'Andrea M. F. Reiter', 'Josefin Röbbig', 'H. Lina Schaare', 'Marie Uhlig', 'Alfred Anwander', 'Pierre-Louis Bazin', 'Annette Horstmann', 'Leonie Lampe', 'Vadim V. Nikulin', 'Hadas Okon-Singer', 'Sven Preusser', 'André Pampel', 'Christiane S. Rohr', 'Julia Sacher1', 'Angelika Thöne-Otto', 'Sabrina Trapp', 'Till Nierhaus', 'Denise Altmann', 'Katrin Arelin', 'Maria Blöchl', 'Edith Bongartz', 'Patric Breig', 'Elena Cesnaite', 'Sufang Chen', 'Roberto Cozatl', 'Saskia Czerwonatis', 'Gabriele Dambrauskaite', 'Maria Dreyer', 'Jessica Enders', 'Melina Engelhardt', 'Marie Michele Fischer', 'Norman Forschack', 'Johannes Golchert', 'Laura Golz', 'C. Alexandrina Guran', 'Susanna Hedrich', 'Nicole Hentschel', 'Daria I. Hoffmann', 'Julia M. Huntenburg', 'Rebecca Jost', 'Anna Kosatschek', 'Stella Kunzendorf', 'Hannah Lammers', 'Mark E. Lauckner', 'Keyvan Mahjoory', 'Natacha Mendes', 'Ramona Menger', 'Enzo Morino', 'Karina Näthe', 'Jennifer Neubauer', 'Handan Noyan', 'Sabine Oligschläger', 'Patricia Panczyszyn-Trzewik', 'Dorothee Poehlchen', 'Nadine Putzke', 'Sabrina Roski', 'Marie-Catherine Schaller', 'Anja Schieferbein', 'Benito Schlaak', 'Hanna Maria Schmidt', 'Robert Schmidt', 'Anne Schrimpf', 'Sylvia Stasch', 'Maria Voss', 'Anett Wiedemann', 'Daniel S. Margulies', 'Michael Gaebler', 'Arno Villringer']}, 'sidecar': {'PowerLineFrequency': 50, 'EEGReference': 'FCz'}, 'channels': {'type': {'VEOG': 'VEOG', 'F3': 'EEG'}}, 'non-bids': {'eeg_extension': '.vhdr', 'path_analysis': {'pattern': 'RSEEG/sub-%entities.subject%.vhdr'}}, 'IO': {'source': '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon', 'target': '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids'}}, 'Individual': [{'non-bids': {'eeg_extension': '.vhdr', 'path_analysis': {'pattern': 'RSEEG/sub-%entities.subject%.vhdr'}}, 'sidecar': {'PowerLineFrequency': 50, 'EEGReference': 'FCz'}, 'channels': {'type': {'VEOG': 'VEOG', 'F3': 'EEG'}}, 'entities': {'subject': '010002', 'task': 'resting'}, 'IO': {'target': '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010002/eeg/sub-010002_task-resting_eeg.vhdr', 'source': '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010002/RSEEG/sub-010002.vhdr'}}, {'non-bids': {'eeg_extension': '.vhdr', 'path_analysis': {'pattern': 'RSEEG/sub-%entities.subject%.vhdr'}}, 'sidecar': {'PowerLineFrequency': 50, 'EEGReference': 'FCz'}, 'channels': {'type': {'VEOG': 'VEOG', 'F3': 'EEG'}}, 'entities': {'subject': '010003', 'task': 'resting'}, 'IO': {'target': '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010003/eeg/sub-010003_task-resting_eeg.vhdr', 'source': '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010003/RSEEG/sub-010003.vhdr'}}, {'non-bids': {'eeg_extension': '.vhdr', 'path_analysis': {'pattern': 'RSEEG/sub-%entities.subject%.vhdr'}}, 'sidecar': {'PowerLineFrequency': 50, 'EEGReference': 'FCz'}, 'channels': {'type': {'VEOG': 'VEOG', 'F3': 'EEG'}}, 'entities': {'subject': '010004', 'task': 'resting'}, 'IO': {'target': '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010004/eeg/sub-010004_task-resting_eeg.vhdr', 'source': '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010004/RSEEG/sub-010004.vhdr'}}]}

Doing the conversion#

We now do the conversion of the dataset by reading the mapping file (‘mappings.yml’) with the convert them module.

convert_them(mapping_path)
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010002/RSEEG/sub-010002.vhdr...
Setting channel info structure...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/README'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/participants.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/participants.json'...
The provided raw data contains annotations, but you did not pass an "event_id" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass "event_id".
Used Annotations descriptions: ['Comment/no USB Connection to actiCAP', 'New Segment/', 'Stimulus/S  1', 'Stimulus/S200', 'Stimulus/S210']
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010002/eeg/sub-010002_task-resting_events.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010002/eeg/sub-010002_task-resting_events.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/dataset_description.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010002/eeg/sub-010002_task-resting_eeg.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010002/eeg/sub-010002_task-resting_channels.tsv'...
Copying data files to sub-010002_task-resting_eeg.vhdr
/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/sovabids/rules.py:278: RuntimeWarning: Converting data files to BrainVision format
  write_raw_bids(raw, bids_path=bids_path,format=output_format,allow_preload=True,overwrite=True)
/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/sovabids/rules.py:278: RuntimeWarning: Encountered data in "short" format. Converting to float32.
  write_raw_bids(raw, bids_path=bids_path,format=output_format,allow_preload=True,overwrite=True)
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010002/sub-010002_scans.tsv'...
Wrote /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010002/sub-010002_scans.tsv entry with eeg/sub-010002_task-resting_eeg.vhdr.
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010002/eeg/sub-010002_task-resting_eeg.json'...
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010003/RSEEG/sub-010003.vhdr...
Setting channel info structure...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/participants.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/participants.json'...
The provided raw data contains annotations, but you did not pass an "event_id" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass "event_id".
Used Annotations descriptions: ['Comment/no USB Connection to actiCAP', 'New Segment/', 'Stimulus/S  1', 'Stimulus/S200', 'Stimulus/S210']
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010003/eeg/sub-010003_task-resting_events.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010003/eeg/sub-010003_task-resting_events.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/dataset_description.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010003/eeg/sub-010003_task-resting_eeg.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010003/eeg/sub-010003_task-resting_channels.tsv'...
Copying data files to sub-010003_task-resting_eeg.vhdr
/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/sovabids/rules.py:278: RuntimeWarning: Converting data files to BrainVision format
  write_raw_bids(raw, bids_path=bids_path,format=output_format,allow_preload=True,overwrite=True)
/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/sovabids/rules.py:278: RuntimeWarning: Encountered data in "short" format. Converting to float32.
  write_raw_bids(raw, bids_path=bids_path,format=output_format,allow_preload=True,overwrite=True)
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010003/sub-010003_scans.tsv'...
Wrote /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010003/sub-010003_scans.tsv entry with eeg/sub-010003_task-resting_eeg.vhdr.
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010003/eeg/sub-010003_task-resting_eeg.json'...
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010004/RSEEG/sub-010004.vhdr...
Setting channel info structure...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/participants.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/participants.json'...
The provided raw data contains annotations, but you did not pass an "event_id" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass "event_id".
Used Annotations descriptions: ['Comment/actiCAP Data On', 'New Segment/', 'Stimulus/S  1', 'Stimulus/S200', 'Stimulus/S210']
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010004/eeg/sub-010004_task-resting_events.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010004/eeg/sub-010004_task-resting_events.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/dataset_description.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010004/eeg/sub-010004_task-resting_eeg.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010004/eeg/sub-010004_task-resting_channels.tsv'...
Copying data files to sub-010004_task-resting_eeg.vhdr
/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/sovabids/rules.py:278: RuntimeWarning: Converting data files to BrainVision format
  write_raw_bids(raw, bids_path=bids_path,format=output_format,allow_preload=True,overwrite=True)
/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/sovabids/rules.py:278: RuntimeWarning: Encountered data in "short" format. Converting to float32.
  write_raw_bids(raw, bids_path=bids_path,format=output_format,allow_preload=True,overwrite=True)
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010004/sub-010004_scans.tsv'...
Wrote /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010004/sub-010004_scans.tsv entry with eeg/sub-010004_task-resting_eeg.vhdr.
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/sub-010004/eeg/sub-010004_task-resting_eeg.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids/dataset_description.json'...

Checking the conversion#

For clarity purposes we will check the output directory we got from sovabids.

print_dir_tree(bids_path)

print('LEMON CONVERSION FINISHED!')
|lemon_bids/
|--- README
|--- dataset_description.json
|--- participants.json
|--- participants.tsv
|--- code/
|------ sovabids/
|--------- mappings.yml
|--------- sovabids.log
|--------- sovabids.log.errors
|--- sub-010002/
|------ sub-010002_scans.tsv
|------ eeg/
|--------- sub-010002_task-resting_channels.tsv
|--------- sub-010002_task-resting_eeg.eeg
|--------- sub-010002_task-resting_eeg.json
|--------- sub-010002_task-resting_eeg.vhdr
|--------- sub-010002_task-resting_eeg.vmrk
|--------- sub-010002_task-resting_events.json
|--------- sub-010002_task-resting_events.tsv
|--- sub-010003/
|------ sub-010003_scans.tsv
|------ eeg/
|--------- sub-010003_task-resting_channels.tsv
|--------- sub-010003_task-resting_eeg.eeg
|--------- sub-010003_task-resting_eeg.json
|--------- sub-010003_task-resting_eeg.vhdr
|--------- sub-010003_task-resting_eeg.vmrk
|--------- sub-010003_task-resting_events.json
|--------- sub-010003_task-resting_events.tsv
|--- sub-010004/
|------ sub-010004_scans.tsv
|------ eeg/
|--------- sub-010004_task-resting_channels.tsv
|--------- sub-010004_task-resting_eeg.eeg
|--------- sub-010004_task-resting_eeg.json
|--------- sub-010004_task-resting_eeg.vhdr
|--------- sub-010004_task-resting_eeg.vmrk
|--------- sub-010004_task-resting_events.json
|--------- sub-010004_task-resting_events.tsv
LEMON CONVERSION FINISHED!

Using the CLI tool#

sovabids can also be used through the command line. Here we provide an example of how to do so.

The overview of what we are doing is now:

graph LR S>"Source path"] B>"Bids path"] R>"Rules file"] AR(("sovapply")) M>"Mappings file"] CT(("sovaconvert")) O[("Converted dataset")] S --> AR B --> AR R --> AR AR --> M M --> CT CT --> O

Same old blues#

Notice that we will run this inside of python so that the example can be run without needing configuration.

To run this locally you will need to run lemon_prepare() function from the command line. You can do so by running: .. code-block:: bash

python -c “from sovabids.datasets import lemon_prepare; lemon_prepare()”

Since we already have run lemon_prepare() inside this example, we will start from this step.

We set up the paths again, but now we will change the output to a new path (with “_cli” at the end). We will also clean this path as we did before.

source_path = os.path.abspath(os.path.join(REPO_PATH,'_data','lemon')) # For the input data we will convert
bids_path= os.path.abspath(os.path.join(REPO_PATH,'_data','lemon_bids_cli')) # The output directory that will have the converted data
rules_path = os.path.abspath(os.path.join(REPO_PATH,'examples','lemon_example_rules.yml')) # The rules file that setups the rule for conversion
mapping_path = os.path.abspath(os.path.join(bids_path,'code','sovabids','mappings.yml')) # The mapping file that will hold the results of applying the rules to each file

print('source_path:',source_path.replace(REPO_PATH,''))
print('bids_path:', bids_path.replace(REPO_PATH,''))
print('rules_path:',rules_path.replace(REPO_PATH,''))
print('mapping_path:',mapping_path.replace(REPO_PATH,''))

try:
    shutil.rmtree(bids_path)
except:
    pass
source_path: /_data/lemon
bids_path: /_data/lemon_bids_cli
rules_path: /examples/lemon_example_rules.yml
mapping_path: /_data/lemon_bids_cli/code/sovabids/mappings.yml

Some necessary code#

To be able to run commands from this notebook and capture their outputs we need to define the following, nevertheless this is not relevant to actually running this from the command line.

from subprocess import PIPE, run

def out(command):
    result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=True)
    return result.stdout

my_output = out("echo hello world")
print(my_output)
hello world

sovapply#

In this example we have already made the rules. So we will apply them using the sovapply tool.

Use the following command to print the help of the tool:

command = "sovapply --help"
print(command)
sovapply --help

This will give the following output

my_output= out(command)
print(my_output)
usage: sovapply apply_rules [-h] [-m MAPPING] source_path bids_path rules

positional arguments:
  source_path           The path to the input data directory that will be
                        converted to bids
  bids_path             The path to the output bids directory
  rules                 The fullpath of the rules file

optional arguments:
  -h, --help            show this help message and exit
  -m MAPPING, --mapping MAPPING
                        The fullpath of the mapping file to be written. If not
                        set it will be located in
                        bids_path/code/sovabids/mappings.yml

Now we will use the following command to get the mappings file.

command = 'sovapply '+source_path + ' '+ bids_path + ' ' + rules_path + ' -m ' + mapping_path
print(command)
sovapply /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/examples/lemon_example_rules.yml -m /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/code/sovabids/mappings.yml

This will produce the following output:

my_output= out(command)
print(my_output)
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010002/RSEEG/sub-010002.vhdr...
Setting channel info structure...
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010003/RSEEG/sub-010003.vhdr...
Setting channel info structure...
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010004/RSEEG/sub-010004.vhdr...
Setting channel info structure...

sovaconvert#

Now we are ready to perform the conversion given the mapping file just made.

Use the following command to print the help of the tool:

command = "sovaconvert --help"
print(command)
sovaconvert --help

This will give the following output

my_output= out(command)
print(my_output)
usage: sovaconvert convert_them [-h] mappings

positional arguments:
  mappings    The mapping file of the conversion.

optional arguments:
  -h, --help  show this help message and exit

Now we will use the following command to perform the conversion.

command = 'sovaconvert ' + mapping_path
print(command)
sovaconvert /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/code/sovabids/mappings.yml

This will produce the following output:

my_output= out(command)
print(my_output)
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010002/RSEEG/sub-010002.vhdr...
Setting channel info structure...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/README'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/participants.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/participants.json'...
The provided raw data contains annotations, but you did not pass an "event_id" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass "event_id".
Used Annotations descriptions: ['Comment/no USB Connection to actiCAP', 'New Segment/', 'Stimulus/S  1', 'Stimulus/S200', 'Stimulus/S210']
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010002/eeg/sub-010002_task-resting_events.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010002/eeg/sub-010002_task-resting_events.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/dataset_description.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010002/eeg/sub-010002_task-resting_eeg.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010002/eeg/sub-010002_task-resting_channels.tsv'...
Copying data files to sub-010002_task-resting_eeg.vhdr
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010002/sub-010002_scans.tsv'...
Wrote /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010002/sub-010002_scans.tsv entry with eeg/sub-010002_task-resting_eeg.vhdr.
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010002/eeg/sub-010002_task-resting_eeg.json'...
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010003/RSEEG/sub-010003.vhdr...
Setting channel info structure...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/participants.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/participants.json'...
The provided raw data contains annotations, but you did not pass an "event_id" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass "event_id".
Used Annotations descriptions: ['Comment/no USB Connection to actiCAP', 'New Segment/', 'Stimulus/S  1', 'Stimulus/S200', 'Stimulus/S210']
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010003/eeg/sub-010003_task-resting_events.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010003/eeg/sub-010003_task-resting_events.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/dataset_description.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010003/eeg/sub-010003_task-resting_eeg.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010003/eeg/sub-010003_task-resting_channels.tsv'...
Copying data files to sub-010003_task-resting_eeg.vhdr
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010003/sub-010003_scans.tsv'...
Wrote /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010003/sub-010003_scans.tsv entry with eeg/sub-010003_task-resting_eeg.vhdr.
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010003/eeg/sub-010003_task-resting_eeg.json'...
Extracting parameters from /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon/sub-010004/RSEEG/sub-010004.vhdr...
Setting channel info structure...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/participants.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/participants.json'...
The provided raw data contains annotations, but you did not pass an "event_id" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass "event_id".
Used Annotations descriptions: ['Comment/actiCAP Data On', 'New Segment/', 'Stimulus/S  1', 'Stimulus/S200', 'Stimulus/S210']
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010004/eeg/sub-010004_task-resting_events.tsv'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010004/eeg/sub-010004_task-resting_events.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/dataset_description.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010004/eeg/sub-010004_task-resting_eeg.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010004/eeg/sub-010004_task-resting_channels.tsv'...
Copying data files to sub-010004_task-resting_eeg.vhdr
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010004/sub-010004_scans.tsv'...
Wrote /home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010004/sub-010004_scans.tsv entry with eeg/sub-010004_task-resting_eeg.vhdr.
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/sub-010004/eeg/sub-010004_task-resting_eeg.json'...
Writing '/home/docs/checkouts/readthedocs.org/user_builds/sovabids/checkouts/latest/_data/lemon_bids_cli/dataset_description.json'...

Checking the conversion#

For clarity purposes we will check the output directory we got from sovabids.

print_dir_tree(bids_path)

print('LEMON CLI CONVERSION FINISHED!')
|lemon_bids_cli/
|--- README
|--- dataset_description.json
|--- participants.json
|--- participants.tsv
|--- code/
|------ sovabids/
|--------- mappings.yml
|--------- sovabids.log
|--------- sovabids.log.errors
|--- sub-010002/
|------ sub-010002_scans.tsv
|------ eeg/
|--------- sub-010002_task-resting_channels.tsv
|--------- sub-010002_task-resting_eeg.eeg
|--------- sub-010002_task-resting_eeg.json
|--------- sub-010002_task-resting_eeg.vhdr
|--------- sub-010002_task-resting_eeg.vmrk
|--------- sub-010002_task-resting_events.json
|--------- sub-010002_task-resting_events.tsv
|--- sub-010003/
|------ sub-010003_scans.tsv
|------ eeg/
|--------- sub-010003_task-resting_channels.tsv
|--------- sub-010003_task-resting_eeg.eeg
|--------- sub-010003_task-resting_eeg.json
|--------- sub-010003_task-resting_eeg.vhdr
|--------- sub-010003_task-resting_eeg.vmrk
|--------- sub-010003_task-resting_events.json
|--------- sub-010003_task-resting_events.tsv
|--- sub-010004/
|------ sub-010004_scans.tsv
|------ eeg/
|--------- sub-010004_task-resting_channels.tsv
|--------- sub-010004_task-resting_eeg.eeg
|--------- sub-010004_task-resting_eeg.json
|--------- sub-010004_task-resting_eeg.vhdr
|--------- sub-010004_task-resting_eeg.vmrk
|--------- sub-010004_task-resting_events.json
|--------- sub-010004_task-resting_events.tsv
LEMON CLI CONVERSION FINISHED!

Total running time of the script: (1 minutes 11.438 seconds)

Gallery generated by Sphinx-Gallery