This repository contains the code for Module 1 and 2 of the RACCROCHE pipeline.
## Overview
Given the phylogenetic relationships of several extant species, the reconstruction of their ancestral genomes at the gene and chromosome level is made difficult by the cycles of whole genome doubling followed by fractionation in plant lineages. Fractionation scrambles the gene adjacencies that enable existing reconstruction methods. We propose an alternative approach that postpones the selection of gene adjacencies for reconstructing small ancestral segments and instead accumulates a very large number of syntenically validated candidate adjacencies to produce long ancestral contigs through maximum weight matching. Likewise, we do not construct chromosomes by successively piecing together contigs into larger segments, but instead count all contig co-occurrences on the input genomes and cluster these, so that chromosomal assemblies of contigs all emerge naturally ordered at each ancestral node of the phylogeny. These strategies result in substantially more complete reconstructions than existing methods. We deploy a number of quality measures: contig lengths, continuity of contig structure on successive ancestors, coverage of the reconstruction on the input genomes, and rearrangement implications of the chromosomal structures obtained. The reconstructed ancestors can be functionally annotated and are visualized by painting the ancestral projections on the descendant genomes, and by highlighting syntenic ancestor-descendant relationships. Our methods can be applied to genomes drawn from a broad range of clades or orders.
### Module 1: construct gene families and list candidate adjacencies
- Step 1: Pre-process gene families
- Step 2: List generalized adjacencies
- Step 3: List candidate adjacencies
### Module 2: construct ancestral contigs through Maximum Weight Matching
- Step 4: Construct contigs through Maximum Weight Matching
This is a full workflow that shows methods regarding the construction of gene families, listing of candidate adjacencies and construction of ancestral contigs using Maximum Weight Matching.
%% Cell type:markdown id: tags:
# Section 1: Importing Libraries, Modules and Configuration File
%% Cell type:code id: tags:
``` python
# setting the Notebook Display Method
%matplotlibinline
```
%% Cell type:code id: tags:
``` python
# install tqdm-joblib
%pipinstalltqdm-joblib
# install Bio
%pipinstallBio
```
%% Output
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: tqdm-joblib in /student/nma904/.local/lib/python3.8/site-packages (0.0.2)
Requirement already satisfied: tqdm in /usr/local/anaconda3/lib/python3.8/site-packages (from tqdm-joblib) (4.50.2)
Requirement already satisfied: tqdm-joblib in c:\users\16132\anaconda3\lib\site-packages (0.0.2)
Requirement already satisfied: tqdm in c:\users\16132\anaconda3\lib\site-packages (from tqdm-joblib) (4.59.0)
Note: you may need to restart the kernel to use updated packages.
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: Bio in /student/nma904/.local/lib/python3.8/site-packages (1.4.0)
Requirement already satisfied: tqdm in /usr/local/anaconda3/lib/python3.8/site-packages (from Bio) (4.50.2)
Requirement already satisfied: requests in /usr/local/anaconda3/lib/python3.8/site-packages (from Bio) (2.24.0)
Requirement already satisfied: mygene in /student/nma904/.local/lib/python3.8/site-packages (from Bio) (3.2.2)
Requirement already satisfied: biopython>=1.79 in /usr/local/anaconda3/lib/python3.8/site-packages (from Bio) (1.79)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/anaconda3/lib/python3.8/site-packages (from requests->Bio) (1.25.11)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/anaconda3/lib/python3.8/site-packages (from requests->Bio) (2022.5.18.1)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/anaconda3/lib/python3.8/site-packages (from requests->Bio) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/anaconda3/lib/python3.8/site-packages (from requests->Bio) (3.0.4)
Requirement already satisfied: biothings-client>=0.2.6 in /student/nma904/.local/lib/python3.8/site-packages (from mygene->Bio) (0.2.6)
Requirement already satisfied: numpy in /usr/local/anaconda3/lib/python3.8/site-packages (from biopython>=1.79->Bio) (1.23.1)
^C
Note: you may need to restart the kernel to use updated packages.
%% Cell type:code id: tags:
``` python
# import libraries
importsys
importyaml
importtime
importos
importio
importpandasaspd
importseabornassns
importmatplotlib
importmatplotlib.pyplotasplt
fromBioimportPhylo
fromscipy.spatial.distanceimportsquareform
fromtqdmimporttqdm
importtqdm_joblib
fromjoblibimportParallel,delayed
# import modules
#? eventually the below list will be in your package
fromGeneFamilyimportGeneFamily
fromGenomeimportGenome
fromMWMInputTreeNodeimportMWMInputTreeNode
frommwmatchingimport*
fromContigimport*
```
%% Cell type:markdown id: tags:
## Parsing the Configuration File
The configuration file consists of a list of input/output directories, input files and parameters to the pipeline.
Specifications for Input:
- leaf_genome_info: the input genomes with their positions on the phylogenetic tree, i.e. *Genomes.txt*<br>
- synteny_file_name: the post-fix of the synmap output of pairwise genome comparisons; <br>
- synteny_file_path: the path to the synmap output data <br>
- phylo_tree_path: the path to the Newick Phylogenetic Tree structure <br>
- jar_path: the path to the UniMoG jar file <br>
__Note__: The input genomes is in the *Genomes.txt* file.
It contains the information about input genomes, including CoGe ID, genomeName, ancestor<br>
The phylogenetic relationship is in the *phyloTree* file.
It includes the unrooted phylogenetic tree in newick format where each tree leaf is denoted as name_CoGeID. <br>
Sample Newick Tree for the monocot project: ( ( (Spirodela_51364, Acorus_54711), Dioscorea_51051), ( (Ananas_25734, Elaeis_33018), Asparagus_33908) )
Specifications for Output:
- gene_list: path where the gene list output should be created <br>
- gene_family: path where the gene family output should be created <br>
- genomes: path where the genome string output should be created <br>
- mwm_input_template: path and template for the output files created from MWM Input step <br>
- mwm_output_template: path and template for the output files created from MWM <br>
- contig_template: path and template for the output files created from constructing contigs <br>
- dcj_output_path: path to the output file when calculating DCJ distance between ancestors <br>
- dcj_summary_path: path to the output file containing a summary of the DCJ calculations <br><br>
Global parameters:
- min_cutoff_weight: minimum similarity cutoff weight for gene family construction, i.e. *min_cutoff_weight=65*
- max_cutoff_weight: maximum similarity cutoff weight for gene family construction, i.e. *max_cutoff_weight=100*
- ws: window size, i.e. *ws=7*
- min_mwm_weight: minimum adjacency weight to be considered for maximum weight matching, i.e. *min_mwm_weight=100*
- gf1: maximum number of genes in a gene family, i.e. *gf1=50*
- gf2: the maximum number of genes from a genome in a gene family, i.e. *gf2=10*
- gf3: the minimum number of genomes in a gene family, i.e. *gf3=1*
#print("The postfix of SynMap output files:", parsed_config['input_file']['synteny_file_name'])
```
%% Output
Project Name and Input Directory: ../inputData/project-buxus
Project Name and Output Directory: ../outputData/project-buxus
Please check required input data under the Input Directory:
1. Genome.txt with info about input genomes and phylogenetic tree
2. SynMap results between every pair of genomes
1. Genome.txt with info about input genomes
2. phyloTree.txt with Newick Tree Structure
3. SynMap results between every pair of genomes
%% Cell type:markdown id: tags:
# Section 2: Constructing Gene Families from syntenically validated orthologs from SynMap
In order to succesfully construct gene families, we require:
- Successful parsing of the configuration file <br>
- Valid parameters in the configuration file
%% Cell type:markdown id: tags:
## Reading in Input Genome Data and Tree Structure Data
Each genome is a leaf of the input phylogenetic tree. The *get_leaves_and_tree_info()* method extracts the following attribute of each genome from the input file specified by *leaf_genome_info* parameter in the configuration file.
print("Visualizing the number of gene families in each chromosome of each genome")
data={'genome':genomes,'chromosome number':chromosomes,'number of genes':gene_lengths}
data={'genome':genomes,'chromosome number':chromosomes,'number of gene families':gene_lengths}
pddata=pd.DataFrame(data)
sns.set(style="darkgrid")
g=sns.catplot(x='chromosome number',y='number of genes',col='genome',data=pddata,kind="bar",aspect=.7,sharex=False,dodge=False)
g=sns.catplot(x='chromosome number',y='number of gene families',col='genome',data=pddata,kind="bar",aspect=.7,sharex=False,dodge=False)
g.set_xticklabels(rotation=30)
```
%% Output
Visualizing the number of gene families in each chromosome of each genome
/Users/lij313/opt/anaconda3/lib/python3.8/site-packages/seaborn/categorical.py:3803: UserWarning: Setting `sharex=False` with `color=None` may cause different levels of the `x` variable to share colors. This will change in a future version.
This is a full workflow that shows methods regarding the construction of gene families, listing of candidate adjacencies and construction of ancestral contigs using Maximum Weight Matching.
%% Cell type:markdown id: tags:
# Section 1: Importing Libraries, Modules and Configuration File
%% Cell type:code id: tags:
``` python
# setting the Notebook Display Method
%matplotlibinline
```
%% Cell type:code id: tags:
``` python
# install tqdm-joblib
%pipinstalltqdm-joblib
# install Bio
%pipinstallBio
```
%% Output
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: tqdm-joblib in /student/nma904/.local/lib/python3.8/site-packages (0.0.2)
Requirement already satisfied: tqdm in /usr/local/anaconda3/lib/python3.8/site-packages (from tqdm-joblib) (4.50.2)
Requirement already satisfied: tqdm-joblib in c:\users\16132\anaconda3\lib\site-packages (0.0.2)
Requirement already satisfied: tqdm in c:\users\16132\anaconda3\lib\site-packages (from tqdm-joblib) (4.59.0)
Note: you may need to restart the kernel to use updated packages.
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: Bio in /student/nma904/.local/lib/python3.8/site-packages (1.4.0)
Requirement already satisfied: tqdm in /usr/local/anaconda3/lib/python3.8/site-packages (from Bio) (4.50.2)
Requirement already satisfied: requests in /usr/local/anaconda3/lib/python3.8/site-packages (from Bio) (2.24.0)
Requirement already satisfied: mygene in /student/nma904/.local/lib/python3.8/site-packages (from Bio) (3.2.2)
Requirement already satisfied: biopython>=1.79 in /usr/local/anaconda3/lib/python3.8/site-packages (from Bio) (1.79)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/anaconda3/lib/python3.8/site-packages (from requests->Bio) (1.25.11)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/anaconda3/lib/python3.8/site-packages (from requests->Bio) (2022.5.18.1)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/anaconda3/lib/python3.8/site-packages (from requests->Bio) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/anaconda3/lib/python3.8/site-packages (from requests->Bio) (3.0.4)
Requirement already satisfied: biothings-client>=0.2.6 in /student/nma904/.local/lib/python3.8/site-packages (from mygene->Bio) (0.2.6)
Requirement already satisfied: numpy in /usr/local/anaconda3/lib/python3.8/site-packages (from biopython>=1.79->Bio) (1.23.1)
^C
Note: you may need to restart the kernel to use updated packages.
%% Cell type:code id: tags:
``` python
# import libraries
importsys
importyaml
importtime
importos
importio
importpandasaspd
importseabornassns
importmatplotlib
importmatplotlib.pyplotasplt
fromBioimportPhylo
fromscipy.spatial.distanceimportsquareform
fromtqdmimporttqdm
importtqdm_joblib
fromjoblibimportParallel,delayed
# import modules
#? eventually the below list will be in your package
fromGeneFamilyimportGeneFamily
fromGenomeimportGenome
fromMWMInputTreeNodeimportMWMInputTreeNode
frommwmatchingimport*
fromContigimport*
```
%% Cell type:markdown id: tags:
## Parsing the Configuration File
The configuration file consists of a list of input/output directories, input files and parameters to the pipeline.
Specifications for Input:
- leaf_genome_info: the input genomes with their positions on the phylogenetic tree, i.e. *Genomes.txt*<br>
- synteny_file_name: the post-fix of the synmap output of pairwise genome comparisons; <br>
- synteny_file_path: the path to the synmap output data <br>
- phylo_tree_path: the path to the Newick Phylogenetic Tree structure <br>
- jar_path: the path to the UniMoG jar file <br>
__Note__: The input genomes is in the *Genomes.txt* file.
It contains the information about input genomes, including CoGe ID, genomeName, ancestor<br>
The phylogenetic relationship is in the *phyloTree* file.
It includes the unrooted phylogenetic tree in newick format where each tree leaf is denoted as name_CoGeID. <br>
Sample Newick Tree for the monocot project: ( ( (Spirodela_51364, Acorus_54711), Dioscorea_51051), ( (Ananas_25734, Elaeis_33018), Asparagus_33908) )
Specifications for Output:
- gene_list: path where the gene list output should be created <br>
- gene_family: path where the gene family output should be created <br>
- genomes: path where the genome string output should be created <br>
- mwm_input_template: path and template for the output files created from MWM Input step <br>
- mwm_output_template: path and template for the output files created from MWM <br>
- contig_template: path and template for the output files created from constructing contigs <br>
- dcj_output_path: path to the output file when calculating DCJ distance between ancestors <br>
- dcj_summary_path: path to the output file containing a summary of the DCJ calculations <br><br>
Global parameters:
- min_cutoff_weight: minimum similarity cutoff weight for gene family construction, i.e. *min_cutoff_weight=65*
- max_cutoff_weight: maximum similarity cutoff weight for gene family construction, i.e. *max_cutoff_weight=100*
- ws: window size, i.e. *ws=7*
- min_mwm_weight: minimum adjacency weight to be considered for maximum weight matching, i.e. *min_mwm_weight=100*
- gf1: maximum number of genes in a gene family, i.e. *gf1=50*
- gf2: the maximum number of genes from a genome in a gene family, i.e. *gf2=10*
- gf3: the minimum number of genomes in a gene family, i.e. *gf3=1*
#print("The postfix of SynMap output files:", parsed_config['input_file']['synteny_file_name'])
```
%% Output
Project Name and Input Directory: ../inputData/project-buxus
Project Name and Output Directory: ../outputData/project-buxus
Please check required input data under the Input Directory:
1. Genome.txt with info about input genomes and phylogenetic tree
2. SynMap results between every pair of genomes
1. Genome.txt with info about input genomes
2. phyloTree.txt with Newick Tree Structure
3. SynMap results between every pair of genomes
%% Cell type:markdown id: tags:
# Section 2: Constructing Gene Families from syntenically validated orthologs from SynMap
In order to succesfully construct gene families, we require:
- Successful parsing of the configuration file <br>
- Valid parameters in the configuration file
%% Cell type:markdown id: tags:
## Reading in Input Genome Data and Tree Structure Data
Each genome is a leaf of the input phylogenetic tree. The *get_leaves_and_tree_info()* method extracts the following attribute of each genome from the input file specified by *leaf_genome_info* parameter in the configuration file.
print("Visualizing the number of gene families in each chromosome of each genome")
data={'genome':genomes,'chromosome number':chromosomes,'number of genes':gene_lengths}
data={'genome':genomes,'chromosome number':chromosomes,'number of gene families':gene_lengths}
pddata=pd.DataFrame(data)
sns.set(style="darkgrid")
g=sns.catplot(x='chromosome number',y='number of genes',col='genome',data=pddata,kind="bar",aspect=.7,sharex=False,dodge=False)
g=sns.catplot(x='chromosome number',y='number of gene families',col='genome',data=pddata,kind="bar",aspect=.7,sharex=False,dodge=False)
g.set_xticklabels(rotation=30)
```
%% Output
Visualizing the number of gene families in each chromosome of each genome
/Users/lij313/opt/anaconda3/lib/python3.8/site-packages/seaborn/categorical.py:3803: UserWarning: Setting `sharex=False` with `color=None` may cause different levels of the `x` variable to share colors. This will change in a future version.