We provide a number of derived data products based on this dataset. The technical documentation below describes how they are produced; the same material is also available as a single PDF.

As of 2024-02-22, we recommend using the derived data in the *_sync tables — eeg_sync, fnirs_sync, ekg_sync, and gsr_sync — rather than the CSVs in the older derived-data releases listed at the bottom of this page. The 2024-02-22 release fixes a number of issues with the 2023-08-28 version of the derived data (see the 2024-02-22 update for details).

Derived Data Description

The synchronized EEG, fNIRS, EKG and GSR signals generated by the steps described in the previous section are located in the following tables:

• eeg_sync

• fnirs_sync

• ekg_sync

• gsr_sync

The column frequency indicates the frequency of the shared clock which is 200Hz by default. If we include other frequencies in the feature, the value in this column can be used to select the desired one.

Not all experiments may have data present in the tables due to a couple of primary reasons. Firstly, time constraints can result in certain tasks remaining incomplete in some experiments. Secondly, the recorded signals are compromised during certain tasks can also lead to the absence of signals in the tables.

In cases where only two participants participated in the sections, signals for the third participant are spurious but they were synchronized and saved to the sync tables for a computational choice. However, they can be filtered out by joining with the data_validity table.

the table below, the table below and the table below detail the mapping between the montage in the montage figure and the naming convention used in the EEG and fNIRS sync tables.

Derived data

Synchronization of EEG, EKG, GSR, and fNIRS Signals

In multimodal neuroimaging studies, synchronizing signals from multiple modalities is a crucial step to conducting comprehensive studies on all these modalities together. The discrepancy between EEG and fNIRS signals’ time series is an issue that researchers encounter frequently due to the limitations of recording hardware or the necessity to remove invalid signals. Additionally, these two modalities have distinct recording rates, further complicating their alignment. To facilitate comprehensive evaluation of EEG and fNIRS signals, it is essential to synchronize these two signals.

We plotted histograms of the EEG and fNIRS signals to check they were sampled at the expected hardware frequency (10Hz for fNIRS and 500Hz for EEG). This is important because the filtering step assumes samples are equally spaced. Having confirmed that, the synchronization process is a two-step approach involving the noise artifacts removal from the signals, followed by resampling and synchronization of the interpolated signals at the desired sampling rate.

Removing Noise in EEG Signals with Notch Filter

EEG signals often exhibit susceptibility to artifacts, an interference that can be attributed to several sources. For instance, physiological factors such as eye movements or blinks can induce such artifacts , as can environmental elements like fluorescent lighting or grounding complications .

Upon thorough examination and visualization of the raw EEG data, we identified a consistent 60 Hz electrical disturbance within the signal, along with corresponding harmonics. An anomalous peak was also noted around the 5 Hz mark, potentially attributable to a grounding irregularity or an other environmental factors.

With the aid of MNE-Python , we efficiently mitigated these intrusive noises by deploying a notch filter. The filter was configured with a frequency of 60 Hz, a transition bandwidth of 9 Hz, and notch widths of 2 Hz.

Mitigating Artifacts in fNIRS Signals Utilizing Bandpass Filter

fNIRS signals are often susceptible to motion artifacts (MA) stemming from physiological activities, including cardiac and respiratory disturbances. These artifacts become particularly noticeable in the measurement of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) concentrations within the signal channels.

To address these challenges, we employed a bandpass filter as an effective noise reduction strategy. The filter was calibrated in line with the recommendations provided by . With a low cutoff bandwidth of 0.01 Hz and a high cutoff bandwidth of 0.2 Hz for the 4th order Butterworth method, the filter was tailored to selectively allow signal components within this frequency range while attenuating components outside the range.

Pre-processing EKG and GSR Signals

To remove noise and improve peak-detection accuracy for EKG signals, we employed a finite impulse response (FIR) filter with 0.67 Hz low cutoff frequency, 45 Hz high cutoff frequency, and order of 1.5x the sampling rate (where sampling rate is 500 Hz) implemented by NeuroKit2 .

We removed noise and smoothed the GSR signals using a low-pass filter with a 3 Hz cutoff frequency and a 4th order Butterworth filter, both implemented by Neurokit2.

Synchronization of EEG, EKG, GSR, and fNIRS Signals

After the EEG, EKG, GSR, and fNIRS signals are pre-processed to remove noise, the signals are upsampled to 2000Hz using the FFT-based resampling method mne.filter.resample available in the Python MNE library .

For each experiment, we define a common clock with initial time starting 2 minutes before the beginning of the first task (rest state) and end time set to 2 minutes after the final task (typically Minecraft). We create equally spaced ticks in this clock at the frequency of 200Hz. Then, the signals are downsampled to this clock’s scale via linear interpolation.

Mapping to the Task Data

Mapping between signals and data will depend on the task being performed. For instance, one can choose to map a signal to the closest data observation or a collection of them. Therefore, we opted for providing the timestamp as a column in the synchronized signals tables so that they can be used for alignment with the task observations.

Description of deprecated derived data products (generated 2023-08-28)

The filtering and synchronization process of raw signals was updated to make it easier to use and reduce unnecessary complexities. For reference, below is a description of the derived data products prior to 2024-02-22, and how they were created.

Derived Data Description

The signals generated by this step correspond to the data in the following tables in the new SQLite3 database that was released 2024-02-22:

• eeg_sync

• fnirs_sync

• ekg_sync

• gsr_sync

The column frequency indicates the frequency of the shared clock which is 200Hz by default. If we include other frequencies in the feature, the value in this column can be used to select the desired one.

Derived Data Files

The following folders contain several versions of derived data at various synchronization frequencies:

• fnirs_10hz contains synchronized fNIRS signals and task data at 10 Hz sampling rate.

• fnirs_500hz contains synchronized fNIRS signals and task data at 500 Hz sampling rate.

• eeg_500hz contains synchronized EEG signals and task data at 500 Hz sampling rate.

• ekg_500hz contains synchronized EKG signals and task data at 500 Hz sampling rate.

• gsr_500hz contains synchronized GSR signals and task data at 500 Hz sampling rate.

• fnirs_eeg_ekg_gsr_120hz contains synchronized fNIRS, EEG, EKG, and GSR signals and task data at 120 Hz sampling rate.

• fnirs_eeg_ekg_gsr_1200hz contains synchronized fNIRS, EEG, EKG, and GSR signals and task data at 1200 Hz sampling rate.

Each derived data folder contains synchronized fNIRS, EEG, EKG, and/or GSR signals for each task and for the entire experiment for each experiment:

exp_*/
    all.csv
    rest_state.csv
    finger_tapping.csv
    affective_individual_lion.csv
    affective_individual_tiger.csv
    affective_individual_leopard.csv
    affective_team.csv
    ping_pong_competitive_lion_tiger.csv
    ping_pong_competitive_leopard_cheetah.csv
    ping_pong_cooperative.csv
    minecraft_saturn_a.csv
    minecraft_saturn_b.csv

Each experiment contains the following CSV files, with columns described in the section below:

• all.csv contains the synchronized signal of all participants for the entire recording sessions, including signals and data in all tasks.

• rest_state.csv contains the synchronized signal and rest state task data.

• finger_tapping.csv contains the synchronized signal and finger tapping task data.

• affective_individual_lion.csv contains the synchronized signal and individual affective task data for participant using the Lion computer in the lab.

• affective_individual_tiger.csv contains the synchronized signal and individual affective task data for participant using the Leopard computer in the lab.

• affective_individual_leopard.csv contains the synchronized signal and individual affective task data for participant using the Leopard computer in the lab.

• affective_team.csv contains the synchronized signal and team affective task data.

• ping_pong_competitive_lion_tiger.csv contains the synchronized signal and ping pong competitive task data between participants sitting on Lion and Tiger stations.

• ping_pong_competitive_leopard_cheetah.csv contains the synchronized signal and ping pong competitive task data between a participant sitting on Leopard station and an experimenter sitting on the Cheetah station.

• ping_pong_cooperative.csv contains the synchronized signal and ping pong cooperative task data between three participants against an artificial intelligent agent.

• minecraft_hands_on_training.csv contains the synchronized signal and Minecraft search-and-rescue hands-on training mission data.

• minecraft_saturn_a.csv contains the synchronized signal and Minecraft search-and-rescue Saturn A mission data.

• minecraft_saturn_b.csv contains the synchronized signal and Minecraft search-and-rescue Saturn B mission data.

Not all experiments may include the aforementioned CSV files, due to a couple of primary reasons. Firstly, time constraints can result in certain tasks remaining incomplete in some experiments. Secondly, instances where the experiment involves only two participants or the recorded signals are compromised during certain tasks can also lead to the absence of these CSV files.

Derived Data Columns Descriptions

The columns in each aforementioned CSV file (e.g., rest_state.csv, ping_pong_cooperative.csv) are detailed in the table below, the table below, the table below, the table below, the table below, the table below, the table below, the table below, the table below, the table below, and the table below.

Derived data

Synchronization of EEG, EKG, GSR, and fNIRS Signals

In multimodal neuroimaging studies, synchronizing signals from multiple modalities is a crucial step to conducting comprehensive studies on all these modalities together. The discrepancy between EEG and fNIRS signals’ time series is an issue that researchers encounter frequently due to the limitations of recording hardware or the necessity to remove invalid signals. Additionally, these two modalities have distinct recording rates, further complicating their alignment. To facilitate comprehensive evaluation of EEG and fNIRS signals, it is essential to synchronize these two signals.

The synchronization process is a two-step approach involving the noise artifacts removal from the signals, followed by resampling and synchronization of the interpolated signals at the desired sampling rate.

Removing Noise in EEG Signals with Notch Filter

EEG signals often exhibit susceptibility to artifacts, an interference that can be attributed to several sources. For instance, physiological factors such as eye movements or blinks can induce such artifacts , as can environmental elements like fluorescent lighting or grounding complications .

Upon thorough examination and visualization of the raw EEG data, we identified a consistent 60 Hz electrical disturbance within the signal, along with corresponding harmonics. An anomalous peak was also noted around the 5 Hz mark, potentially attributable to a grounding irregularity or an other environmental factors.

With the aid of MNE-Python , we efficiently mitigated these intrusive noises by deploying a notch filter. The filter was configured with a frequency of 60 Hz, a transition bandwidth of 9 Hz, and notch widths of 2 Hz.

Mitigating Artifacts in fNIRS Signals Utilizing Bandpass Filter

fNIRS signals are often susceptible to motion artifacts (MA) stemming from physiological activities, including cardiac and respiratory disturbances. These artifacts become particularly noticeable in the measurement of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) concentrations within the signal channels.

To address these challenges, we employed a bandpass filter as an effective noise reduction strategy. The filter was calibrated in line with the recommendations provided by . With a low cutoff bandwidth of 0.01 Hz and a high cutoff bandwidth of 0.2 Hz for the 4th order Butterworth method, the filter was tailored to selectively allow signal components within this frequency range while attenuating components outside the range.

Pre-processing EKG and GSR Signals

To remove noise and improve peak-detection accuracy for EKG signals, we employed a finite impulse response (FIR) filter with 0.67 Hz low cutoff frequency, 45 Hz high cutoff frequency, and order of 1.5x the sampling rate (where sampling rate is 500 Hz) implemented by NeuroKit2 .

We removed noise and smoothed the GSR signals using a low-pass filter with a 3 Hz cutoff frequency and a 4th order Butterworth filter, both implemented by Neurokit2.

Synchronization of EEG, EKG, GSR, and fNIRS Signals

After the EEG, EKG, GSR, and fNIRS signals are pre-processed to remove noise, the signals are resampled to a common sampling rate using the FFT-based resampling method mne.filter.resample available in the Python MNE library . To synchronize the signals, we generate a time series matching the common sampling rate of the resampled signals, with the first timestamp rounded to the nearest second. Then, the signals are interpolated to this generated time series via linear interpolation.

Synchronizing Task Data with EEG and fNIRS Resampled Signals

Understanding the relationship between participants’ behaviors, environmental stimuli, and neuroimaging data requires a precise synchronization of task data with the corresponding EEG and fNIRS signals. By aligning these data streams, we can examine the influence of environmental stimuli on the participants’ neuroimaging signals, which in turn, impact their behavior and task performance.

The process of integrating EEG and fNIRS signals with task data starts with grouping of signals by the tasks during which they were recorded, followed by the synchronization of the task data to the corresponding EEG and fNIRS signals.

Grouping EEG, EKG, GSR, and fNIRS Signals by Task

The preliminary step in our approach to synchronizing EEG, EKG, GSR, and fNIRS signals with the task data involves the grouping of the signals by the tasks during which the signals were recorded. The task data can be categorized into two distinct types: status-based and event-based data.

Status-based task data

This type of task data represent the current state of the task, such as task score. For each task, the grouping process of these data begins by including the signals recorded immediately before the task initiation and immediately following task completion. This ensures no data is overlooked at the boundaries of the task. Subsequently, all signals recorded between these two points are included, forming a complete set of signals associated with the task.

Event-based task data

This type of task data, on the other hand, correspond to specific events that occur during the task, such as affective task arousal or the submission of a valence score. For each task, we determine the EEG, EKG, GSR, and fNIRS entry associated with the first event and the last event. These signal entries, as well as all entries recorded between these points, are included into the data set related to the task.

Synchronizing Task Data with EEG, EKG, GSR, and fNIRS Signals

Having grouped the EEG, EKG, GSR, and fNIRS signals according to task type, we then proceed to synchronize these signal entries with their respective task data.

Status-based task data

The synchronization is accomplished by assigning the status data recorded closest in time to each EEG, EKG, GSR, and fNIRS signal entry. This method ensures that each EEG, EKG, GSR, and fNIRS entry is paired with the most representative status data.

Event-based task data

We assign each event data to the EEG, EKG, GSR, and fNIRS signal entry recorded at the time closest to the occurrence of the event. Those EEG, EKG, GSR, and fNIRS signal entries without a corresponding event data are left unassigned, signifying that no specific event occurred during these recordings.

Older derived data releases

We always recommend using the latest version of the derived data products, as older versions may have issues that are fixed in newer versions, and the documentation is only accurate for the newest version. However, we will continue to provide the older versions for as long as we feasibly can.

• Filtered synchronized data (generated 2023-08-28) (836 GB)
• Filtered synchronized data (1.1 TB). Uploaded 2023-07-19. See the 2023-07-19 update for details.
• Filtered synchronized data (168 GB). Uploaded 2023-06-14.