Overview¶

This notebook shows how to access and interact with 2-photon calcium imaging data collected as part of the Allen Institute's Visual Behavior 2P project.

You can learn more about this dataset, behavioral task, and find other useful tools here: Overview page and Allen Brain Atlas

Specifically, this notebook will show how to load neural data for all imaging planes in one 2-photon imaging session into a single 'tidy' dataframe, make simple event-triggered plots, and do some basic analysis using scikit-learn.

This is designed to demonstrate a simple method for interacting with the Visual Behavior 2P data. Many aspects of the dataset are not explored here.

Set up environment and import packages¶

We have built a package called brain_observatory_utilities which contains some useful convenience functions. The allenSDK is a dependency of this package and will be automatically installed when you install brain_observatory_utilities per the instructions below.

We will first install brain_observatory_utilities into our colab environment by running the commands below. When this cell is complete, click on the RESTART RUNTIME button that appears at the end of the output. Note that running this cell will produce a long list of outputs and some error messages. Clicking RESTART RUNTIME at the end will resolve these issues.

You can minimize the cell after you are done to hide the output.

Next we will import packages we need later in the notebook¶

Load the session and experiment summary tables¶

The AllenSDK provides functionality for downloading tables that describe all sessions and experiments (individual imaging planes) in the Visual Behavior 2P dataset. We first download the data cache:

• Ophys_session_table contains metadata describing imaging sessions. If more than one plane was imaged during a session, one ophys session id will be associated multiple ophys experiment ids. Each ophys session id will also have a unique behavior session id.
• Behavior_session_table contains metadata describing behavioral sessions, which may or may not be during imaging. Behavior session ids that do not have ophys session ids were training sessions.
• Ophys_experiment_table contains metadata describing imaging experiments (aka imaging planes). When mesoscope is used, one ophys session may contain up to 8 unique experiments (two visual areas by four imaging depths). Some imaging planes may not be released due to quality control issues, thus each ophys session id is associated with anywhere from one to eight unique experiment ids. Ophys experiment ids are unique and do not repeat across sessions. To find the same imaging plane that was matched across multiple sessions, use the ophys_container_id column that can be found in both ophys_session_table and ophys_experiment_table.

Then we can access the session and experiment tables directly.

Note that a 'session' is a single behavioral session. Sessions that are performed on the mesoscope will have multiple (up to 8) 'experiments' associated with them, where an experiment is a distinct imaging plane.

We can then view the contents of the session table. Note that this contains a lot of useful metadata about each session. One of the columns, ophys_experiment_id provides a list of the experiments (aka imaging planes) that are associated with each session.

The experiment table has one row per experiment. Note that the ophys_session_id column links each experiment to its associated session in the session_table.

Load one example session¶

We are going to select one session from this table, session 854060305. This is a session with Sst-IRES-Cre mouse, which expressed GCaMP6f in Sst+ inhibitory interneurons. There were 6 simultaneously acquired imaging planes for this session. We can view metadata for this session as follows:

Download all associated experiments¶

Each session consists of one or more 'experiments', in which each experiment is a single imaging plane

Each mesoscope session has up to 8 experiments associated with the session. We will load all sessions into a dictionary with the experiment IDs as the keys

The first time that this cell is run, the associated NWB files will be downloaded to your local data_storage_directory. Subsequent runs of this cell will be faster since the data will already be cached locally.

View the max projection and one cell ROI for one of the experiments¶

We can view the cell_specimen_table for one experiment, which contains information about each identified cell in that experiment

We can then visualize the max projection and one of the identified ROIs

Load neural data into memory¶

The cell below will load the neural data into memory in the pandas 'tidy' format by iterating over each of the 6 experiments and using some helpful tools from the brain_observatory_utilities package that was imported above as ophys.

It will also include a subset of metadata from ophys_experiment_table to facilitate splitting by depth, structure (aka cortical area), cre line (aka cell class), etc.

Note that 'tidy' data means that each row represents only one observation. Observations are stacked vertically. Thus, the timestamps columns will repeat for every cell in the dataset.

We can then look at some attributes of the neural_data dataframe we have created.

It is ~2.5 million rows long:

It is so long because has one row for each timestamp for each cell.

Below are the first 5 entries. Again, note that the tidy format means that each row has only one observation, which represents a single GCaMP6 fluorescnce value for a single neuron.

• The cell_roi_id column contains unique roi ids for all cells in a given experiment, which do not repeat across ophys sessions.
• The cell_specimen_id column contains unique ids for cells that were matched across ophys sessions. Thus, a cell that was imaged in more than one session has multiple roi ids but one cell specimen id.

Examine Cell IDs¶

We can get the unique Cell IDs in our dataset as follows:

If we wanted to get the timeseries for one cell, we could query the neural_data dataframe. For example, to get the full timeseries for the cell with cell_specimen_id = 1086557208:

Each cell has three types of traces:

• dff column is the Calcium fluorescence signal, normalized to background fluorescence.
• events column is deconvolved events from dff trace, which approximates neural firing rate and removes the slow decay of the Calcium signal (for more details, you can read EVENT DETECTION section in Visual Behavior whitepaper).
• filtered_events column is events smoothed with a half-gaussian kernel.

We can then plot DeltaF/F for this cell for the full experiment as follows:

Load stimulus data into memory¶

The stimulus table is shared across all experiments (imaging planes) in a session. We can therefore use the stimulus table for just one experiment.

We are going to drop the image_set column because it is not informative for our purposes. We can then view the first 10 rows of the stimulus table.

This table provides helpful information like image name, start, duration and stop of image presentation, and whether the image was omitted. stimulus_block and stimulus_block_name indicate the type of stimulus mice were presented at a given point in a session. To select active change detection behavior, first we need to filter the table for change_detection_behavior or 1 block. Note that sessions may have different number of stimulus blocks, thus change_detection_behavior may be associated with either 0 or 1 in stimulus_block column.

View the stimulus_templates attribute¶

Note that the unwarped column contains the image before the application of a spherical warp. All of the pixels labeled 'NaN' will be off-screen (not visible to the mouse) after the warp is applied.

All experiments in a given session will share the same stimulus_templates

View the unwarped images¶

View the warped images¶

This represents what was actually on the screen during the session

Describe stimulus omissions¶

An important feature of the task is that stimuli are shown at a very regular cadence (250 ms on, 500 ms off), but stimuli are randomly omitted with a probability of ~5%. These unexpected and random stimulus omissions could be perceived as an expectation violation by the mouse.

Omitted stimuli are denoted in the stimulus_table by the omitted column. True means that the stimulus that would have been shown at that time was actually omitted (and was replaced by an extended gray screen between stimuli).

We can look at the first 10 examples of omitted stimuli as follows. Note that each 'omitted' stimulus still has a 'start_time' and a 'stop_time' associated with it. This actually represents the time that a stimulus would have been shown, had it not been omitted.

Stimulus omissions are also indicated in the image_name column by the string omitted

Create an event triggered response dataframe relative to omissions¶

If we want to see how a given cell responds when regularly flashed stimuli are omitted, we can calculate the response around each of the stimulus omissions. The brain_observatory_utilities package has a convenience function to do this, in the module we imported as utilities. We give the function:

• a dataframe of interest (containing activity from one cell)
• the t and y values of interest
• the event times
• how much time before and after each event we are interested in
• the desired sampling rate of the output - this is the rate onto which the response will be interpolated

The function will return a new dataframe with the response for the given cell, aligned to each of the events.

We can see that the output has columns for

• time - this is our new timebase relative to the events. In this case, it ranges from -3 to 3
• dff - this is the deltaF/F value surrounding each event, interpolated onto the new timebase. If, when calling the event_triggered_response function we had passed y = 'events', this column would be events instead of dff.
• event_number - this is an integer representing the count of each event. In this example, there were 185 omissions, so they are numbered from 0 to 184
• event_time - this is the time of each event

The docstring for the event_triggered_response function can be viewed as follows:

Plot an event triggered response¶

The output format of the event_triggered_response function is designed to plug directly into Seaborn's lineplot plotting function. We can then view the mean response to omitted stimuli with 95% confidence intervals very easily:

Note that the regular, image-driven responses with a 750 ms inter-stimulus interval are visible everywhere except at t=0, which is when the unexpectedly omitted stimulus occurred.

Make a function to plot an event triggered average in one line¶

If we make a wrapper function that combines the process of calculating and plotting the event triggered response, it can be called in a single line below. By having event_query input variable, we can use this function to plot responses to any event of interest (omisisons, changes, hits/misses, specific images, etc)

Now plot the omission triggered response for the same cell using filtered events (these events extracted from the deltaF/F timeseries using an event extraction algorithm, then smoothed with a half-gaussian kernel) instead of dff.

Plot the responses for 10 sample cells¶

We can then iterate over 10 randomly chosen cells and plot their activity during omissions.

Interestingly, not all SST cells in this session do the same thing!

Calculate the mean response for each of the individual imaging planes in this experiment¶

By iterating over experiment IDs, we can also calculate the mean response for each of the 6 imaging planes. Do Sst cells in different visual areas respond to omissions in a distinct way?

We will first use a Pandas groupby and mean operations to get the mean timeseries for each cell in that imaging plane:

We can then iterate over our 6 experiment IDs and use our make_event_triggered_plot wrapper function to calculate and plot the omission triggered response for that imaging plane:

There are clearly some large differences in the way that Sst cells respond to these unexpected stimulus omissions by area.

This example could be extended to include cells from the other two cre-lines in the dataset: The VIP-Cre line which labels VIP+ inhibitory interneurons and the Slc17a7 line, which is a pan-excitatory line.

In addition, responses to different stimuli could be explored, along with responses relative to other behavioral measures, such as licking.

For a full description of the dataset and all available data streams, see the Visual Behavior Project Description at: https://portal.brain-map.org/explore/circuits/visual-behavior-2p

Set up data for scikit learn¶

What if we wanted to use scikit-learn for a decoding or clustering analysis? We'd need to get the data into a standard format for scikit learn, which is often a feature matrix (X) and a vector of labels (y).

Instead of just omissions, let's now look at the responses to each of the stimuli in this session, which consists of 8 unique images, plus the omitted stimuli (which we characterize as a unique stimulus type). First, we will calculate an event triggered response to each stimulus start time in the stimulus table.

One way to construct a feature matrix might be to build it such that dimensions are trials x cells. Thus:

• Each row would be one trial, where a trial is defined as a unique image presentation
• Each column would represent the average response of a given cell on that image presentation.

To do so, let's construct another intermediate dataframe called average_responses that contains the average response of each cell (in the 750 ms window we've selected above) to each image presentation. We'll do this using a Pandas groupby to group by cell_specimen_id and stimulus_presentations_id (aka trial).

We're also going to merge in our stimulus metadata.

Now we can construct a dataframe called features_and_labels that will contain one row per trial, one column per cell, plus columns with the image_index and image_name

The X matrix can be extracted by getting the columns associated with the cell_specimen_ids

And y is just the image_name column (it could also be the image_index column if you want a numeric value instead of a string to represent the image identity)

Dimensionality reduction¶

Now we can use t-SNE, which will project our 53-dimensional feature space (53 neurons in the session) into two dimensions.

And visualize the results, with colors representing each unique stimulus.

This demonstrates that the time-averaged population responses to at least some of the stimuli seem to fall into distinct clusters in our 53-dimensional space, while others appear more overlapped. This implies that a decoding analysis might be more successful at decoding some stimuli than others.

Train a simple decoder¶

We can use an SVM decoder from scikit learn to ask how well we can decode image identity from the feature matrix we have constructed.

Split our data into train and test sets, instantiate the model, then fit.

Use the model to make predictions on the held-out test set

Evaluate the accuracy

Evaluate the confusion matrix

This tells us that the model can decode some stimuli well (im035, im075 and im106, for example), while it struggles more with others (im000 and omissions, for example). Do the stimuli that the decoder succeeds in classifying align with those that cluster cleanly in t-SNE space?

Follow up exercise¶

Can you create event triggered averages and perform decoding using other events of interest, such as licks or rewards?

To see the full list of all attributes available for each experiment via the AllenSDK, uncomment the cell below and run it