Zero-shot prediction of functional protein sequence variants¶

In Moderna Therapeutics's study titled "Therapeutic Enzyme Engineering Using a Generative Neural Network", scientists turbocharge ornithine transcarbamylase (OTC), a critical player in the urea cycle combating a rare metabolic ailment. Their secret sauce? Marrying insights from nature's evolution dance with a dash of artificial evolution guided by savvy deep learning models. Meanwhile, in the paper "Efficient evolution of human antibodies from general protein language models", authors play protein origami, deftly evolving human antibodies sans the cheat sheet on target antigen specifics binding specificity, or protein structure. No experimental data for model supervision; both approaches propose beneficial mutations in a zero-shot manner.

Crafting functional protein sequences in the protein engineering arena is a puzzle akin to navigating a vast cosmic expanse. Picture a protein with 200 amino acids; a mere switcheroo of one amino acid spawns a array of 4000 potential variants (200*20). My previous notebooks used machine learning as the Sherpa guiding directed evolution when armed with experimental compass. But what if the experimental compass is missing? Can we still map out a path to promising protein sequences by divining the protein's mutational horoscope?

Nature leaves us clues, imprinting functional wisdom in protein sequences. Enter the unsupervised models, revealing higher functional variant tales from the sequence data troves, as showcased in the article "Language models enable zero-shot prediction of the effects of mutations on protein function". These unsupervised maestros for predicting protein variant effects fall into three troupes: hybrid protein language models (PLMs), PLMs, Alignment based models, and Inverse folding models.

Our playbook? This notebook orchestrates and contrasts these model from each class, aiming to spotlight protein sequence gems that can lighten the load of experimental quests in directed evolution. The playbook uses competitive models from the benchmarks set by ProteinGym](https://proteingym.org/), providing a comprehensive evaluation of these models' capabilities in predicting the mutational landscape of proteins.

Goals

• Efficiently identify promising protein sequence variants to alleviate the experimental screening burden in directed evolution.
• Provide a canvas of mutational landscape, specifically focusing on the encapsidation protein 22K crucial for recombinant Adeno-Associated Virus (rAAV) production. Maybe there are protein sequence gems that hold the key to improving rAAV production?

Protein fitness landscape prediction using Hybrid PLMs¶

• Large Protein Language Models (PLMs) are trained on large protein databases encompassing all known sequences or on sets of homologous sequences spanning many protein families.
• Hybrid PLMs are PLMs that leverage multiple sequence alignment (MSAs) such as Tranception, and MSA Transformer.
• Below is the take on hybrid PLMs from Pascal Notin author of ProteinGym paper.

From the hybrid PLMs class, we will be using MSA Transformer to predict mutational landscape for encapsidation protein 22K.

Obtain FASTA file of protein sequence¶

Generate sequence alignment in a3m format¶

The sequence alignment for encapsidation protein 22K was generated using MMSeq2 from ColabFold v1.5.3 server.

Generate mutant triplets to make predictions¶

Predict fitness landscape¶

Visualize fitness landscape¶

Heatmap which shows us hotspots for mutations that are beneficial or detrimental to the function of the protein.

Protein fitness landscape prediction using PLMs¶

• Protein Language Models (PLMs) are LLMs trained on large protein databases encompassing all known sequences such as VESPA, ESM2, and ESM1v.

• PLMs adeptly learn evolutionary constraints that generalize across protein families.

• PLM both bidirectional and autoregressive generative models particularly excel in scenarios involving small families with sparse homologs. However, they generally underperform family-specific models for larger families.

We will be using ESM2, and ESM1v to predict mutational landscape for encapsidation protein 22K.

Predict fitness landscape¶

Visualize fitness landscape¶

To do:

• Predict mutational landscape from models esm_if1_gvp4_t16_142M_UR50,

esm2_t36_3B_UR50D, and esm2_t48_15B_UR50D models.

• Why the hypothesis are different?
• How the hypothesis from all of these models can be streamlined to reduce experimental testing in protein engineering workflow.

Alignment based models¶

• Family specific models that learn evolutionary constraints specific to the protein family of interest.
• Search large databases for homologs of protein sequence or domain of interest, align the positions of these homolgs in an multiple sequence alignment (MSA), and then fit statistical sequence models to the MSA.
• Performance of the model is contingent on the availability of sufficiently deep and diverse alignments for training.
• In the realm of zero-shot prediction of mutational landscape competitive models include GEMME, DeepSequence, and EVmutation. Explore their performance benchmarks on ProteinGym.

To do:

• Include GEMME and EVmutation prediction.

Inverse folding models¶

...

Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins

"The key conceptual advance is that by learning the rules underlying local evolution, we can construct a global evolutionary “vector field” that we show can (1) predict the root (or potentially multiple roots) of observed evolutionary trajectories, (2) order protein sequences in evolutionary time, and (3) identify the mutational strategies that drive these trajectories."