Skip to main content

VAME Pipeline

Open In Colab

If you haven't yet, please install VAME:

pip install vame-py

The most convenient way to use VAME is through the VAME pipeline. The pipeline class automates the processes of:

  • loading the data
  • preprocessing the data
  • creating and training the VAME model
  • validating the VAME model
  • segmenting the behavior into motifs
  • clustering the motifs into communities
  • visualizing the results

Let's start by importing the necessary libraries:

import pprint
from vame.pipeline import VAMEPipeline
from vame.util.sample_data import download_sample_data

Input data​

To quickly try VAME, you can download sample data and use it as input. If you want to work with your own data, all you need to do is to provide the paths to the pose estimation files as lists of strings. You can also optionally provide the paths to the corresponding video files.

Note on NWB input: the sample-data downloader only ships DeepLabCut and SLEAP files. To run VAME on NWB files (ndx-pose), skip download_sample_data and pass your own .nwb paths to poses_estimations with source_software="NWB".

# You can run VAME with data from different sources:
# "DeepLabCut", "SLEAP", "LightningPose" or "NWB"
source_software = "DeepLabCut"

# Download sample data
ps = download_sample_data(source_software)
videos = [ps["video"]]
poses_estimations = [ps["poses"]]
fps = ps["fps"]

pprint.pp(videos)
pprint.pp(poses_estimations)

Instantiate the VAME pipeline​

Now it's time to instantiate the VAME pipeline. Select your working directory, name of your project and extra configuration arguments. The extra configuration arguments are optional and can be used to customize the VAME pipeline. Here we use fewer epochs and a smaller network so the tutorial runs quickly, especially without a GPU. For real projects you will likely want to keep the defaults; see config.yaml for all available parameters.

Note: if the project folder already exists, the pipeline loads the existing project and ignores config_kwargs. To change the configuration, use a new project_name or update it with vame.update_config.

# Set up your working directory and project name
working_directory = '.'
project_name = 'pipeline_example'

# Customize the configuration for the project
config_kwargs = {
    "n_clusters": 15,
    "pose_confidence": 0.9,
    "max_epochs": 30,
    "hidden_size_layer_1": 128,
    "hidden_size_layer_2": 128,
    "hidden_size_rec": 128,
    "hidden_size_pred": 128,
}

# Instantiate the pipeline
# this will create a VAME project and prepare the data
pipeline = VAMEPipeline(
    working_directory=working_directory,
    project_name=project_name,
    videos=videos,
    poses_estimations=poses_estimations,
    source_software=source_software,
    fps=fps,
    config_kwargs=config_kwargs,
)

Before running the pipeline, you can check the input datasets:

ds = pipeline.get_raw_datasets()
ds

Run the pipeline​

Now you can run the pipeline. At this point, you should pass the names of the pose estimation keypoints to be used for egocentric alignment. You can also pass other preprocessing and training set arguments, such as keypoints to leave out of model training.

Note: The pipeline will take some time to run, depending on the size of the dataset, number of epochs, and if you are using a GPU or not.

preprocessing_kwargs = {
    "centered_reference_keypoint": "snout",
    "orientation_reference_keypoint": "tailbase",
}
trainingset_kwargs = {
    "test_fraction": 0.1,
    "keypoints_to_exclude": ["tail_end"],
}

pipeline.run_pipeline(
    preprocessing_kwargs=preprocessing_kwargs,
    trainingset_kwargs=trainingset_kwargs,
)

Visualize the results​

After running the pipeline, you can visualize the results:

pipeline.visualize_preprocessing(
    show_figure=True,
    save_to_file=False,
)
pipeline.visualize_model_losses(
    show_figure=True,
    save_to_file=False,
)
pipeline.visualize_hierarchical_tree(segmentation_algorithm="hmm")
pipeline.visualize_umap(show_figure="plotly")

Produce the pipeline report​

pipeline.report()

Resuming the pipeline​

If for some reason you need to stop the pipeline, you can resume it later from any step Example: resuming from community clustering step

pipeline.run_pipeline(from_step=5)