Running in Docker container on EC2 instance¶
Started Docker container with the following command:¶
docker run -p 8888:8888 -v /home/ubuntu/gitrepos/LabDocs/jupyter_nbs/sam:/home/notebooks -v /home/ubuntu/data:/data/ -it kubu4/bioinformatics:v11 /bin/bash
The command allows access to Jupyter Notebook over port 8888 and makes my Jupyter Notebook GitHub repo and my data files accessible to the Docker container
Once the container was started, started Jupyter Notebook with the following command inside the Docker container:
jupyter notebook
This is configured in the Docker container to launch a Jupyter Notebook without a browser on port 8888.
Created a tunnel from my local computer to the Docker container:¶
ssh -i ~/Dropbox/Lab/Sam/bioinformatics.pem -N -L localhost:8888:localhost:8888 ubuntu@ec2.ip.address
This command is run in a separate Terminal window than the one that is used to ssh into the EC2 instance to start Docker and all of that.
This ssh command specifies to use my Amazon EC2 authentication file (bioinformatics.pem), along with the -N and -L options for port forwarding stuff (see man ssh for deets), and binds the port 8888 on my local computer to port 8888 on the EC2 isntance.
The tunnel allows me to start the Jupyter Notebook in my web browser. I enter localhost:8888 in as the URL.
In [1]:
%%bash
date
Fri Jul 15 16:56:46 UTC 2016
In [2]:
%%bash
hostname
570c28713283
Check computer specs¶
In [3]:
%%bash
lscpu
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz Stepping: 2 CPU MHz: 2900.088 BogoMIPS: 5800.17 Hypervisor vendor: Xen Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 25600K NUMA node0 CPU(s): 0-7
In [4]:
cd /data/
/data
In [5]:
ls
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_1.fq.gz 1NF_25A_2.fq.gz* 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_2.fq.gz 1NF_26A_1.fq.gz* 1HL_10A_1.fq.gz* 1NF_26A_2.fq.gz* 1HL_10A_2.fq.gz* 1NF_27A_1.fq.gz* 1HL_11A_1.fq.gz* 1NF_27A_2.fq.gz* 1HL_11A_2.fq.gz* 1NF_28A_1.fq.gz* 1HL_12A_1.fq.gz* 1NF_28A_2.fq.gz* 1HL_12A_2.fq.gz* 1NF_29A_1.fq.gz* 1HL_13A_1.fq.gz* 1NF_29A_2.fq.gz* 1HL_13A_2.fq.gz* 1NF_2A_1.fq.gz* 1HL_14A_1.fq.gz* 1NF_2A_2.fq.gz* 1HL_14A_2.fq.gz* 1NF_30A_1.fq.gz* 1HL_15A_1.fq.gz* 1NF_30A_2.fq.gz* 1HL_15A_2.fq.gz* 1NF_31A_1.fq.gz* 1HL_16A_1.fq.gz* 1NF_31A_2.fq.gz* 1HL_16A_2.fq.gz* 1NF_32A_1.fq.gz* 1HL_17A_1.fq.gz* 1NF_32A_2.fq.gz* 1HL_17A_2.fq.gz* 1NF_33A_1.fq.gz* 1HL_19A_1.fq.gz* 1NF_33A_2.fq.gz* 1HL_19A_2.fq.gz* 1NF_4A_1.fq.gz* 1HL_1A_1.fq.gz* 1NF_4A_2.fq.gz* 1HL_1A_2.fq.gz* 1NF_5A_1.fq.gz* 1HL_20A_1.fq.gz* 1NF_5A_2.fq.gz* 1HL_20A_2.fq.gz* 1NF_6A_1.fq.gz* 1HL_21A_1.fq.gz* 1NF_6A_2.fq.gz* 1HL_21A_2.fq.gz* 1NF_7A_1.fq.gz* 1HL_22A_1.fq.gz* 1NF_7A_2.fq.gz* 1HL_22A_2.fq.gz* 1NF_8A_1.fq.gz* 1HL_23A_1.fq.gz* 1NF_8A_2.fq.gz* 1HL_23A_2.fq.gz* 1NF_9A_1.fq.gz* 1HL_24A_1.fq.gz* 1NF_9A_2.fq.gz* 1HL_24A_2.fq.gz* 1SN_10A_1.fq.gz* 1HL_25A_1.fq.gz* 1SN_10A_2.fq.gz* 1HL_25A_2.fq.gz* 1SN_11A_1.fq.gz* 1HL_26A_1.fq.gz* 1SN_11A_2.fq.gz* 1HL_26A_2.fq.gz* 1SN_12A_1.fq.gz* 1HL_27A_1.fq.gz* 1SN_12A_2.fq.gz* 1HL_27A_2.fq.gz* 1SN_13A_1.fq.gz* 1HL_28A_1.fq.gz* 1SN_13A_2.fq.gz* 1HL_28A_2.fq.gz* 1SN_14A_1.fq.gz* 1HL_29A_1.fq.gz* 1SN_14A_2.fq.gz* 1HL_29A_2.fq.gz* 1SN_15A_1.fq.gz* 1HL_2A_1.fq.gz* 1SN_15A_2.fq.gz* 1HL_2A_2.fq.gz* 1SN_16A_1.fq.gz* 1HL_31A_1.fq.gz* 1SN_16A_2.fq.gz* 1HL_31A_2.fq.gz* 1SN_17A_1.fq.gz* 1HL_33A_1.fq.gz* 1SN_17A_2.fq.gz* 1HL_33A_2.fq.gz* 1SN_18A_1.fq.gz* 1HL_34A_1.fq.gz* 1SN_18A_2.fq.gz* 1HL_34A_2.fq.gz* 1SN_19A_1.fq.gz* 1HL_35A_1.fq.gz* 1SN_19A_2.fq.gz* 1HL_35A_2.fq.gz* 1SN_1A_1.fq.gz* 1HL_3A_1.fq.gz* 1SN_1A_2.fq.gz* 1HL_3A_2.fq.gz* 1SN_20A_1.fq.gz* 1HL_4A_1.fq.gz* 1SN_20A_2.fq.gz* 1HL_4A_2.fq.gz* 1SN_21A_1.fq.gz* 1HL_5A_1.fq.gz* 1SN_21A_2.fq.gz* 1HL_5A_2.fq.gz* 1SN_22A_1.fq.gz* 1HL_6A_1.fq.gz* 1SN_22A_2.fq.gz* 1HL_6A_2.fq.gz* 1SN_23A_1.fq.gz* 1HL_7A_1.fq.gz* 1SN_23A_2.fq.gz* 1HL_7A_2.fq.gz* 1SN_24A_1.fq.gz* 1HL_8A_1.fq.gz* 1SN_24A_2.fq.gz* 1HL_8A_2.fq.gz* 1SN_25A_1.fq.gz* 1HL_9A_1.fq.gz* 1SN_25A_2.fq.gz* 1HL_9A_2.fq.gz* 1SN_26A_1.fq.gz* 1NF_10A_1.fq.gz* 1SN_26A_2.fq.gz* 1NF_10A_2.fq.gz* 1SN_27A_1.fq.gz* 1NF_11A_1.fq.gz* 1SN_27A_2.fq.gz* 1NF_11A_2.fq.gz* 1SN_28A_1.fq.gz* 1NF_12A_1.fq.gz* 1SN_28A_2.fq.gz* 1NF_12A_2.fq.gz* 1SN_29A_1.fq.gz* 1NF_13A_1.fq.gz* 1SN_29A_2.fq.gz* 1NF_13A_2.fq.gz* 1SN_2A_1.fq.gz* 1NF_14A_1.fq.gz* 1SN_2A_2.fq.gz* 1NF_14A_2.fq.gz* 1SN_30A_1.fq.gz* 1NF_15A_1.fq.gz* 1SN_30A_2.fq.gz* 1NF_15A_2.fq.gz* 1SN_31A_1.fq.gz* 1NF_16A_1.fq.gz* 1SN_31A_2.fq.gz* 1NF_16A_2.fq.gz* 1SN_32A_1.fq.gz* 1NF_17A_1.fq.gz* 1SN_32A_2.fq.gz* 1NF_17A_2.fq.gz* 1SN_3A_1.fq.gz* 1NF_18A_1.fq.gz* 1SN_3A_2.fq.gz* 1NF_18A_2.fq.gz* 1SN_4A_1.fq.gz* 1NF_19A_1.fq.gz* 1SN_4A_2.fq.gz* 1NF_19A_2.fq.gz* 1SN_5A_1.fq.gz* 1NF_1A_1.fq.gz* 1SN_5A_2.fq.gz* 1NF_1A_2.fq.gz* 1SN_6A_1.fq.gz* 1NF_20A_1.fq.gz* 1SN_6A_2.fq.gz* 1NF_20A_2.fq.gz* 1SN_7A_1.fq.gz* 1NF_21A_1.fq.gz* 1SN_7A_2.fq.gz* 1NF_21A_2.fq.gz* 1SN_8A_1.fq.gz* 1NF_22A_1.fq.gz* 1SN_8A_2.fq.gz* 1NF_22A_2.fq.gz* 1SN_9A_1.fq.gz* 1NF_23A_1.fq.gz* 1SN_9A_2.fq.gz* 1NF_23A_2.fq.gz* analysis/ 1NF_24A_1.fq.gz* radtags.stderr 1NF_24A_2.fq.gz* radtags.stdout 1NF_25A_1.fq.gz* stacks/
Rename FASTQ files to match R1 and R2 requirements for pyrad demultiplexing¶¶
In [9]:
%%bash
mv 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_1.fq.gz 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz
In [10]:
%%bash
mv 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_2.fq.gz 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz
In [11]:
cd analysis/
/data/analysis
In [12]:
ls
barcodes.txt params.txt
In [13]:
%%bash
cat params.txt
==** parameter inputs for pyRAD version 3.0.66 **======================== affected step ==
./ ## 1. Working directory (all)
/data/oly_gbs/*.gz ## 2. Loc. of non-demultiplexed files (if not line 18) (s1)
/analysis/20160609_pyrad/barcodes.txt ## 3. Loc. of barcode file (if not line 18) (s1)
/usr/local/bioinformatics/vsearch-1.11.1-linux-x86_64/bin/vsearch ## 4. command (or path) to call vsearch (or usearch) (s3,s6)
/usr/local/bioinformatics/muscle3.8.31_i86linux64 ## 5. command (or path) to call muscle (s3,s7)
CWGC ## 6. Restriction overhang (e.g., C|TGCAG -> TGCAG) (s1,s2)
16 ## 7. N processors (parallel) (all)
6 ## 8. Mindepth: min coverage for a cluster (s4,s5)
4 ## 9. NQual: max # sites with qual < 20 (or see line 20)(s2)
.88 ## 10. Wclust: clustering threshold as a decimal (s3,s6)
pairgbs ## 11. Datatype: rad,gbs,pairgbs,pairddrad,(others:see docs)(all)
4 ## 12. MinCov: min samples in a final locus (s7)
3 ## 13. MaxSH: max inds with shared hetero site (s7)
oly_gbs_pyrad ## 14. Prefix name for final output (no spaces) (s7)
==== optional params below this line =================================== affected step ==
## 15.opt.: select subset (prefix* only selector) (s2-s7)
## 16.opt.: add-on (outgroup) taxa (list or prefix*) (s6,s7)
## 17.opt.: exclude taxa (list or prefix*) (s7)
## 18.opt.: loc. of de-multiplexed data (s2)
## 19.opt.: maxM: N mismatches in barcodes (def= 1) (s1)
## 20.opt.: phred Qscore offset (def= 33) (s2)
2 ## 21.opt.: filter: def=0=NQual 1=NQual+adapters. 2=strict (s2)
## 22.opt.: a priori E,H (def= 0.001,0.01, if not estimated) (s5)
4 ## 23.opt.: maxN: max Ns in a cons seq (def=5) (s5)
8 ## 24.opt.: maxH: max heterozyg. sites in cons seq (def=5) (s5)
## 25.opt.: ploidy: max alleles in cons seq (def=2;see docs) (s4,s5)
## 26.opt.: maxSNPs: (def=100). Paired (def=100,100) (s7)
## 27.opt.: maxIndels: within-clust,across-clust (def. 3,99) (s3,s7)
## 28.opt.: random number seed (def. 112233) (s3,s6,s7)
## 29.opt.: trim overhang left,right on final loci, def(0,0) (s7)
* ## 30.opt.: output formats: p,n,a,s,v,u,t,m,k,g,* (see docs) (s7)
## 31.opt.: maj. base call at depth>x<mindepth (def.x=mindepth) (s5)
50 ## 32.opt.: keep trimmed reads (def=0). Enter min length. (s2)
## 33.opt.: max stack size (int), def= max(500,mean+2*SD) (s3)
## 34.opt.: minDerep: exclude dereps with <= N copies, def=1 (s3)
## 35.opt.: use hierarchical clustering (def.=0, 1=yes) (s6)
## 36.opt.: repeat masking (def.=1='dust' method, 0=no) (s3,s6)
16 ## 37.opt.: vsearch max threads per job (def.=6; see docs) (s3,s6)
==== optional: list group/clade assignments below this line (see docs) ==================
1HL 5 1HL*
1NF 5 1NF*
1SN 5 1SN*
In [14]:
%%bash
mv /data/160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz /data/analysis/
In [16]:
%%bash
mv /data/160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz /data/analysis/
In [17]:
ls
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz barcodes.txt 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz params.txt
Need to edit params file to match current locations of data files. Will do outside of notebook. BRB...
In [18]:
%%bash
cat params.txt
==** parameter inputs for pyRAD version 3.0.66 **======================== affected step ==
/data/analysis/ ## 1. Working directory (all)
/data/analysis/ ## 2. Loc. of non-demultiplexed files (if not line 18) (s1)
/data/analysis/barcodes.txt ## 3. Loc. of barcode file (if not line 18) (s1)
/usr/local/bioinformatics/vsearch-1.11.1-linux-x86_64/bin/vsearch ## 4. command (or path) to call vsearch (or usearch) (s3,s6)
/usr/local/bioinformatics/muscle3.8.31_i86linux64 ## 5. command (or path) to call muscle (s3,s7)
CWGC ## 6. Restriction overhang (e.g., C|TGCAG -> TGCAG) (s1,s2)
16 ## 7. N processors (parallel) (all)
6 ## 8. Mindepth: min coverage for a cluster (s4,s5)
4 ## 9. NQual: max # sites with qual < 20 (or see line 20)(s2)
.88 ## 10. Wclust: clustering threshold as a decimal (s3,s6)
pairgbs ## 11. Datatype: rad,gbs,pairgbs,pairddrad,(others:see docs)(all)
4 ## 12. MinCov: min samples in a final locus (s7)
3 ## 13. MaxSH: max inds with shared hetero site (s7)
oly_gbs_pyrad ## 14. Prefix name for final output (no spaces) (s7)
==== optional params below this line =================================== affected step ==
## 15.opt.: select subset (prefix* only selector) (s2-s7)
## 16.opt.: add-on (outgroup) taxa (list or prefix*) (s6,s7)
## 17.opt.: exclude taxa (list or prefix*) (s7)
## 18.opt.: loc. of de-multiplexed data (s2)
## 19.opt.: maxM: N mismatches in barcodes (def= 1) (s1)
## 20.opt.: phred Qscore offset (def= 33) (s2)
2 ## 21.opt.: filter: def=0=NQual 1=NQual+adapters. 2=strict (s2)
## 22.opt.: a priori E,H (def= 0.001,0.01, if not estimated) (s5)
4 ## 23.opt.: maxN: max Ns in a cons seq (def=5) (s5)
8 ## 24.opt.: maxH: max heterozyg. sites in cons seq (def=5) (s5)
## 25.opt.: ploidy: max alleles in cons seq (def=2;see docs) (s4,s5)
## 26.opt.: maxSNPs: (def=100). Paired (def=100,100) (s7)
## 27.opt.: maxIndels: within-clust,across-clust (def. 3,99) (s3,s7)
## 28.opt.: random number seed (def. 112233) (s3,s6,s7)
## 29.opt.: trim overhang left,right on final loci, def(0,0) (s7)
* ## 30.opt.: output formats: p,n,a,s,v,u,t,m,k,g,* (see docs) (s7)
## 31.opt.: maj. base call at depth>x<mindepth (def.x=mindepth) (s5)
50 ## 32.opt.: keep trimmed reads (def=0). Enter min length. (s2)
## 33.opt.: max stack size (int), def= max(500,mean+2*SD) (s3)
## 34.opt.: minDerep: exclude dereps with <= N copies, def=1 (s3)
## 35.opt.: use hierarchical clustering (def.=0, 1=yes) (s6)
## 36.opt.: repeat masking (def.=1='dust' method, 0=no) (s3,s6)
16 ## 37.opt.: vsearch max threads per job (def.=6; see docs) (s3,s6)
==== optional: list group/clade assignments below this line (see docs) ==================
1HL 5 1HL*
1NF 5 1NF*
1SN 5 1SN*
View barcodes.txt file¶
In [19]:
%%bash
cat barcodes.txt
1NF_1A CTCC 1NF_2A TGCA 1NF_4A ACTA 1NF_5A CAGA 1NF_6A AACT 1NF_7A GCGT 1NF_8A CGAT 1NF_9A GTAA 1NF_10A AGGC 1NF_11A GATC 1NF_12A TCAC 1NF_13A TGCGA 1NF_14A CGCTT 1NF_15A TCACC 1NF_16A CTAGC 1NF_17A ACAAA 1NF_18A TTCTC 1NF_19A AGCCC 1NF_20A GTATT 1NF_21A CTGTA 1NF_22A ACCGT 1NF_23A GCTTA 1NF_24A GGTGT 1NF_25A AGGAT 1NF_26A ATTGA 1NF_27A CATCT 1NF_28A CCTAC 1NF_29A GAGGA 1NF_30A GGAAC 1NF_31A GTCAA 1NF_32A TAATA 1NF_33A TACAT 1SN_1A TCGTT 1SN_2A GGTTGT 1SN_3A CCAGCT 1SN_4A TTCAGA 1SN_5A TAGGAA 1SN_6A GCTCTA 1SN_7A CCACAA 1SN_8A CTTCCA 1SN_9A GAGATA 1SN_10A ATGCCT 1SN_11A AGTGGA 1SN_12A ACCTAA 1SN_13A ATATGT 1SN_14A ATCGTA 1SN_15A CATCGT 1SN_16A CGCGGT 1SN_17A CTATTA 1SN_18A GCCAGT 1SN_19A GGAAGA 1SN_20A GTACTT 1SN_21A GTTGAA 1SN_22A TAACGA 1SN_23A TGGCTA 1SN_24A TATTTTT 1SN_25A CTTGCTT 1SN_26A ATGAAAC 1SN_27A AAAAGTT 1SN_28A GAATTCA 1SN_29A GAACTTC 1SN_30A GGACCTA 1SN_31A GTCGATT 1SN_32A AACGCCT 1HL_1A AATATGC 1HL_2A ACGTGTT 1HL_3A ATTAATT 1HL_4A ATTGGAT 1HL_5A CATAAGT 1HL_6A CGCTGAT 1HL_7A CGGTAGA 1HL_8A CTACGGA 1HL_9A GCGGAAT 1HL_10A TAGCGGA 1HL_11A TCGAAGA 1HL_12A TCTGTGA 1HL_13A TGCTGGA 1HL_14A ACGACTAC 1HL_15A TAGCATGC 1HL_16A TAGGCCAT 1HL_17A TGCAAGGA 1HL_19A TGGTACGT 1HL_20A TCTCAGTC 1HL_21A CCGGATAT 1HL_22A CGCCTTAT 1HL_23A AACCGAGA 1HL_24A ACAGGGAA 1HL_25A ACGTGGTA 1HL_26A CCATGGGT 1HL_27A CGCGGAGA 1HL_28A CGTGTGGT 1HL_29A GCTGTGGA 1HL_31A GGATTGGT 1HL_33A GTGAGGGT 1HL_34A TATCGGGA 1HL_35A TTCCTGGA
Step 1: De-multiplex reads¶
In [20]:
%%bash
time pyrad -p params.txt -s 1
warning: CUT site matches within one of the barcodes, I suggest double checking the file to make sure it properly demultiplexes no stats file generated
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
step 1: sorting reads by barcode
Traceback (most recent call last):
File "/usr/local/bin/pyrad", line 9, in <module>
load_entry_point('pyrad==3.0.66', 'console_scripts', 'pyrad')()
File "build/bdist.linux-x86_64/egg/pyrad/pyRAD.py", line 292, in main
File "build/bdist.linux-x86_64/egg/pyrad/sortandcheck2.py", line 428, in main
File "build/bdist.linux-x86_64/egg/pyrad/sortandcheck2.py", line 389, in writefunc
ValueError: max() arg is an empty sequence
real 93m3.141s
user 81m1.775s
sys 0m19.037s
Hmmm, will re-run this since it seems to have failed...
In [21]:
%%bash
time pyrad -p params.txt -s 1
fastq/ directory in working directory contains data, move/remove it before running step 1
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
real 0m2.740s
user 0m0.304s
sys 0m0.115s
In [23]:
cd fastq/
/data/analysis/fastq
In [25]:
ls
In [26]:
cd ..
/data/analysis
In [27]:
ls
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz fastq/ 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz params.txt barcodes.txt stats/
In [28]:
%%bash
rm -rf fastq/
In [29]:
ls
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz params.txt
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz stats/
barcodes.txt
In [ ]:
%%bash
time pyrad -p params.txt -s 1
Jupyter kernel crased while this was running, so no output to screen. I used the top command to verify that pyrad was still running despite Jupyter being down. Let's look at some files to see what happened...
In [1]:
pwd
Out[1]:
u'/home/notebooks'
In [2]:
cd /data/analysis/
/data/analysis
In [3]:
%%bash
head -n 100 stats/s1.sorting.txt
file Nreads cut_found bar_matched sample true_bar obs_bars N_obs
In [4]:
ls
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz fastq/ 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz params.txt barcodes.txt stats/
In [5]:
ls fastq/
Guess it didn't complete...
In [6]:
rmdir fastq/
rmdir: failed to remove 'fastq/': Directory not empty
In [7]:
cd fastq/
/data/analysis/fastq
In [8]:
ls
In [9]:
rm -rf fastq/
In [10]:
ls
In [11]:
pwd
Out[11]:
u'/data/analysis/fastq'
In [12]:
cd ..
/data/analysis
In [13]:
rm -rf fastq/
In [14]:
ls
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz params.txt
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz stats/
barcodes.txt
In [15]:
cd stats/
/data/analysis/stats
In [16]:
ls
s1.sorting.txt
In [17]:
rm s1.sorting.txt
In [18]:
cd ..
/data/analysis
In [19]:
%%bash
time pyrad -p params.txt -s 1
warning: CUT site matches within one of the barcodes, I suggest double checking the file to make sure it properly demultiplexes
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
step 1: sorting reads by barcode
./bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
/bin/cat: write error: No space left on device
real 911m11.266s
user 718m35.394s
sys 4m32.133s
Let's see if the de-multiplexing really failed or not. Based on the run time, it seems like things should be fine.
In [1]:
cd /data/analysis/fastq/
/data/analysis/fastq
In [2]:
ls
1HL_10A_R1.fq.gz 1HL_35A_R1.fq.gz 1NF_25A_R1.fq.gz 1SN_18A_R1.fq.gz 1HL_10A_R2.fq.gz 1HL_35A_R2.fq.gz 1NF_25A_R2.fq.gz 1SN_18A_R2.fq.gz 1HL_11A_R1.fq.gz 1HL_3A_R1.fq.gz 1NF_26A_R1.fq.gz 1SN_19A_R1.fq.gz 1HL_11A_R2.fq.gz 1HL_3A_R2.fq.gz 1NF_26A_R2.fq.gz 1SN_19A_R2.fq.gz 1HL_12A_R1.fq.gz 1HL_4A_R1.fq.gz 1NF_27A_R1.fq.gz 1SN_1A_R1.fq.gz 1HL_12A_R2.fq.gz 1HL_4A_R2.fq.gz 1NF_27A_R2.fq.gz 1SN_1A_R2.fq.gz 1HL_13A_R1.fq.gz 1HL_5A_R1.fq.gz 1NF_28A_R1.fq.gz 1SN_20A_R1.fq.gz 1HL_13A_R2.fq.gz 1HL_5A_R2.fq.gz 1NF_28A_R2.fq.gz 1SN_20A_R2.fq.gz 1HL_14A_R1.fq.gz 1HL_6A_R1.fq.gz 1NF_29A_R1.fq.gz 1SN_21A_R1.fq.gz 1HL_14A_R2.fq.gz 1HL_6A_R2.fq.gz 1NF_29A_R2.fq.gz 1SN_21A_R2.fq.gz 1HL_15A_R1.fq.gz 1HL_7A_R1.fq.gz 1NF_2A_R1.fq.gz 1SN_22A_R1.fq.gz 1HL_15A_R2.fq.gz 1HL_7A_R2.fq.gz 1NF_2A_R2.fq.gz 1SN_22A_R2.fq.gz 1HL_16A_R1.fq.gz 1HL_8A_R1.fq.gz 1NF_30A_R1.fq.gz 1SN_23A_R1.fq.gz 1HL_16A_R2.fq.gz 1HL_8A_R2.fq.gz 1NF_30A_R2.fq.gz 1SN_23A_R2.fq.gz 1HL_17A_R1.fq.gz 1HL_9A_R1.fq.gz 1NF_31A_R1.fq.gz 1SN_24A_R1.fq.gz 1HL_17A_R2.fq.gz 1HL_9A_R2.fq.gz 1NF_31A_R2.fq.gz 1SN_24A_R2.fq.gz 1HL_19A_R1.fq.gz 1NF_10A_R1.fq.gz 1NF_32A_R1.fq.gz 1SN_25A_R1.fq.gz 1HL_19A_R2.fq.gz 1NF_10A_R2.fq.gz 1NF_32A_R2.fq.gz 1SN_25A_R2.fq.gz 1HL_1A_R1.fq.gz 1NF_11A_R1.fq.gz 1NF_33A_R1.fq.gz 1SN_26A_R1.fq.gz 1HL_1A_R2.fq.gz 1NF_11A_R2.fq.gz 1NF_33A_R2.fq.gz 1SN_26A_R2.fq.gz 1HL_20A_R1.fq.gz 1NF_12A_R1.fq.gz 1NF_4A_R1.fq.gz 1SN_27A_R1.fq.gz 1HL_20A_R2.fq.gz 1NF_12A_R2.fq.gz 1NF_4A_R2.fq.gz 1SN_27A_R2.fq.gz 1HL_21A_R1.fq.gz 1NF_13A_R1.fq.gz 1NF_5A_R1.fq.gz 1SN_28A_R1.fq.gz 1HL_21A_R2.fq.gz 1NF_13A_R2.fq.gz 1NF_5A_R2.fq.gz 1SN_28A_R2.fq.gz 1HL_22A_R1.fq.gz 1NF_14A_R1.fq.gz 1NF_6A_R1.fq.gz 1SN_29A_R1.fq.gz 1HL_22A_R2.fq.gz 1NF_14A_R2.fq.gz 1NF_6A_R2.fq.gz 1SN_29A_R2.fq.gz 1HL_23A_R1.fq.gz 1NF_15A_R1.fq.gz 1NF_7A_R1.fq.gz 1SN_2A_R1.fq.gz 1HL_23A_R2.fq.gz 1NF_15A_R2.fq.gz 1NF_7A_R2.fq.gz 1SN_2A_R2.fq.gz 1HL_24A_R1.fq.gz 1NF_16A_R1.fq.gz 1NF_8A_R1.fq.gz 1SN_30A_R1.fq.gz 1HL_24A_R2.fq.gz 1NF_16A_R2.fq.gz 1NF_8A_R2.fq.gz 1SN_30A_R2.fq.gz 1HL_25A_R1.fq.gz 1NF_17A_R1.fq.gz 1NF_9A_R1.fq.gz 1SN_31A_R1.fq.gz 1HL_25A_R2.fq.gz 1NF_17A_R2.fq.gz 1NF_9A_R2.fq.gz 1SN_31A_R2.fq.gz 1HL_26A_R1.fq.gz 1NF_18A_R1.fq.gz 1SN_10A_R1.fq.gz 1SN_32A_R1.fq.gz 1HL_26A_R2.fq.gz 1NF_18A_R2.fq.gz 1SN_10A_R2.fq.gz 1SN_32A_R2.fq.gz 1HL_27A_R1.fq.gz 1NF_19A_R1.fq.gz 1SN_11A_R1.fq.gz 1SN_3A_R1.fq.gz 1HL_27A_R2.fq.gz 1NF_19A_R2.fq.gz 1SN_11A_R2.fq.gz 1SN_3A_R2.fq.gz 1HL_28A_R1.fq.gz 1NF_1A_R1.fq.gz 1SN_12A_R1.fq.gz 1SN_4A_R1.fq.gz 1HL_28A_R2.fq.gz 1NF_1A_R2.fq.gz 1SN_12A_R2.fq.gz 1SN_4A_R2.fq.gz 1HL_29A_R1.fq.gz 1NF_20A_R1.fq.gz 1SN_13A_R1.fq.gz 1SN_5A_R1.fq.gz 1HL_29A_R2.fq.gz 1NF_20A_R2.fq.gz 1SN_13A_R2.fq.gz 1SN_5A_R2.fq.gz 1HL_2A_R1.fq.gz 1NF_21A_R1.fq.gz 1SN_14A_R1.fq.gz 1SN_6A_R1.fq.gz 1HL_2A_R2.fq.gz 1NF_21A_R2.fq.gz 1SN_14A_R2.fq.gz 1SN_6A_R2.fq.gz 1HL_31A_R1.fq.gz 1NF_22A_R1.fq.gz 1SN_15A_R1.fq.gz 1SN_7A_R1.fq.gz 1HL_31A_R2.fq.gz 1NF_22A_R2.fq.gz 1SN_15A_R2.fq.gz 1SN_7A_R2.fq.gz 1HL_33A_R1.fq.gz 1NF_23A_R1.fq.gz 1SN_16A_R1.fq.gz 1SN_8A_R1.fq.gz 1HL_33A_R2.fq.gz 1NF_23A_R2.fq.gz 1SN_16A_R2.fq.gz 1SN_8A_R2.fq.gz 1HL_34A_R1.fq.gz 1NF_24A_R1.fq.gz 1SN_17A_R1.fq.gz 1SN_9A_R1.fq.gz 1HL_34A_R2.fq.gz 1NF_24A_R2.fq.gz 1SN_17A_R2.fq.gz 1SN_9A_R2.fq.gz
In [3]:
ls -1 | wc -l
192
Correct number of files were produced from demultiplexing, so things look good on that front. Let's just glance at the stats file to see what it looks like...
NOTE: Bash trick above. The ls command above is using the number one (1), NOT lowercase L (l). This provides an accurate line count when you pipe the output to the word count command (wc).
In [4]:
%%bash
head ../stats/s1.sorting.txt
file Nreads cut_found bar_matched 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_.fq.gz 364261046 355811046 354725421 sample true_bar obs_bars N_obs 1SN_27A AAAAGTT AAAAGTT 3325108 1SN_27A AAAAGTT NAAAGTT 8330 1SN_27A AAAAGTT AAAAATT 4748 1SN_27A AAAAGTT ANAAGTT 1455 1SN_27A AAAAGTT AAAATTT 782
Things look OK to me. Next step!
Merge paired-reads¶
The first thing we have to do is unzip the reads if they are gzipped. If you demultiplexed your data in pyrad they will be in the fastq/ directory.
In [5]:
%%bash
time gunzip /data/analysis/fastq/*.fq.gz
gzip: /data/analysis/fastq/1HL_19A_R2.fq.gz: unexpected end of file real 52m44.818s user 1m24.977s sys 0m13.352s
In [7]:
%%bash
time for gfile in /data/analysis/fastq/*_R1.fq;
do /usr/local/bioinformatics/pear-0.9.6-bin-64/pear-0.9.6-bin-64 -f $gfile \
-r ${gfile/_R1.fq/_R2.fq} \
-o ${gfile/_R1.fq/} \
-n 33 \
-t 33 \
-q 10 \
-j 20 >> pear.log 2>&1;
done
real 25m34.538s user 71m32.940s sys 0m17.129s
Set the data location to the de-multiplexed & merged data¶
In [8]:
%%bash
## set location of demultiplexed data that are 'pear' filtered
sed -i '/## 18./c\/data/analysis/fastq/*.assembled.fastq ## 18. demulti data loc ' /data/analysis/params.txt
In [9]:
cat /data/analysis/params.txt
==** parameter inputs for pyRAD version 3.0.66 **======================== affected step ==
/data/analysis/ ## 1. Working directory (all)
/data/analysis/ ## 2. Loc. of non-demultiplexed files (if not line 18) (s1)
/data/analysis/barcodes.txt ## 3. Loc. of barcode file (if not line 18) (s1)
/usr/local/bioinformatics/vsearch-1.11.1-linux-x86_64/bin/vsearch ## 4. command (or path) to call vsearch (or usearch) (s3,s6)
/usr/local/bioinformatics/muscle3.8.31_i86linux64 ## 5. command (or path) to call muscle (s3,s7)
CWGC ## 6. Restriction overhang (e.g., C|TGCAG -> TGCAG) (s1,s2)
16 ## 7. N processors (parallel) (all)
6 ## 8. Mindepth: min coverage for a cluster (s4,s5)
4 ## 9. NQual: max # sites with qual < 20 (or see line 20)(s2)
.88 ## 10. Wclust: clustering threshold as a decimal (s3,s6)
pairgbs ## 11. Datatype: rad,gbs,pairgbs,pairddrad,(others:see docs)(all)
4 ## 12. MinCov: min samples in a final locus (s7)
3 ## 13. MaxSH: max inds with shared hetero site (s7)
oly_gbs_pyrad ## 14. Prefix name for final output (no spaces) (s7)
==== optional params below this line =================================== affected step ==
## 15.opt.: select subset (prefix* only selector) (s2-s7)
## 16.opt.: add-on (outgroup) taxa (list or prefix*) (s6,s7)
## 17.opt.: exclude taxa (list or prefix*) (s7)
/data/analysis/fastq/*.assembled.fastq ## 18. demulti data loc
## 19.opt.: maxM: N mismatches in barcodes (def= 1) (s1)
## 20.opt.: phred Qscore offset (def= 33) (s2)
2 ## 21.opt.: filter: def=0=NQual 1=NQual+adapters. 2=strict (s2)
## 22.opt.: a priori E,H (def= 0.001,0.01, if not estimated) (s5)
4 ## 23.opt.: maxN: max Ns in a cons seq (def=5) (s5)
8 ## 24.opt.: maxH: max heterozyg. sites in cons seq (def=5) (s5)
## 25.opt.: ploidy: max alleles in cons seq (def=2;see docs) (s4,s5)
## 26.opt.: maxSNPs: (def=100). Paired (def=100,100) (s7)
## 27.opt.: maxIndels: within-clust,across-clust (def. 3,99) (s3,s7)
## 28.opt.: random number seed (def. 112233) (s3,s6,s7)
## 29.opt.: trim overhang left,right on final loci, def(0,0) (s7)
* ## 30.opt.: output formats: p,n,a,s,v,u,t,m,k,g,* (see docs) (s7)
## 31.opt.: maj. base call at depth>x<mindepth (def.x=mindepth) (s5)
50 ## 32.opt.: keep trimmed reads (def=0). Enter min length. (s2)
## 33.opt.: max stack size (int), def= max(500,mean+2*SD) (s3)
## 34.opt.: minDerep: exclude dereps with <= N copies, def=1 (s3)
## 35.opt.: use hierarchical clustering (def.=0, 1=yes) (s6)
## 36.opt.: repeat masking (def.=1='dust' method, 0=no) (s3,s6)
16 ## 37.opt.: vsearch max threads per job (def.=6; see docs) (s3,s6)
==== optional: list group/clade assignments below this line (see docs) ==================
1HL 5 1HL*
1NF 5 1NF*
1SN 5 1SN*
Set the datatype to "merged"¶
In [10]:
%%bash
sed -i '/## 11./c\merged ## 11. data type ' /data/analysis/params.txt
In [11]:
%%bash
cat /data/analysis/params.txt
==** parameter inputs for pyRAD version 3.0.66 **======================== affected step ==
/data/analysis/ ## 1. Working directory (all)
/data/analysis/ ## 2. Loc. of non-demultiplexed files (if not line 18) (s1)
/data/analysis/barcodes.txt ## 3. Loc. of barcode file (if not line 18) (s1)
/usr/local/bioinformatics/vsearch-1.11.1-linux-x86_64/bin/vsearch ## 4. command (or path) to call vsearch (or usearch) (s3,s6)
/usr/local/bioinformatics/muscle3.8.31_i86linux64 ## 5. command (or path) to call muscle (s3,s7)
CWGC ## 6. Restriction overhang (e.g., C|TGCAG -> TGCAG) (s1,s2)
16 ## 7. N processors (parallel) (all)
6 ## 8. Mindepth: min coverage for a cluster (s4,s5)
4 ## 9. NQual: max # sites with qual < 20 (or see line 20)(s2)
.88 ## 10. Wclust: clustering threshold as a decimal (s3,s6)
merged ## 11. data type
4 ## 12. MinCov: min samples in a final locus (s7)
3 ## 13. MaxSH: max inds with shared hetero site (s7)
oly_gbs_pyrad ## 14. Prefix name for final output (no spaces) (s7)
==== optional params below this line =================================== affected step ==
## 15.opt.: select subset (prefix* only selector) (s2-s7)
## 16.opt.: add-on (outgroup) taxa (list or prefix*) (s6,s7)
## 17.opt.: exclude taxa (list or prefix*) (s7)
/data/analysis/fastq/*.assembled.fastq ## 18. demulti data loc
## 19.opt.: maxM: N mismatches in barcodes (def= 1) (s1)
## 20.opt.: phred Qscore offset (def= 33) (s2)
2 ## 21.opt.: filter: def=0=NQual 1=NQual+adapters. 2=strict (s2)
## 22.opt.: a priori E,H (def= 0.001,0.01, if not estimated) (s5)
4 ## 23.opt.: maxN: max Ns in a cons seq (def=5) (s5)
8 ## 24.opt.: maxH: max heterozyg. sites in cons seq (def=5) (s5)
## 25.opt.: ploidy: max alleles in cons seq (def=2;see docs) (s4,s5)
## 26.opt.: maxSNPs: (def=100). Paired (def=100,100) (s7)
## 27.opt.: maxIndels: within-clust,across-clust (def. 3,99) (s3,s7)
## 28.opt.: random number seed (def. 112233) (s3,s6,s7)
## 29.opt.: trim overhang left,right on final loci, def(0,0) (s7)
* ## 30.opt.: output formats: p,n,a,s,v,u,t,m,k,g,* (see docs) (s7)
## 31.opt.: maj. base call at depth>x<mindepth (def.x=mindepth) (s5)
50 ## 32.opt.: keep trimmed reads (def=0). Enter min length. (s2)
## 33.opt.: max stack size (int), def= max(500,mean+2*SD) (s3)
## 34.opt.: minDerep: exclude dereps with <= N copies, def=1 (s3)
## 35.opt.: use hierarchical clustering (def.=0, 1=yes) (s6)
## 36.opt.: repeat masking (def.=1='dust' method, 0=no) (s3,s6)
16 ## 37.opt.: vsearch max threads per job (def.=6; see docs) (s3,s6)
==== optional: list group/clade assignments below this line (see docs) ==================
1HL 5 1HL*
1NF 5 1NF*
1SN 5 1SN*
Step 2: Quality filtering¶
In [14]:
%%bash
time pyrad -p /data/analysis/params.txt -s 2
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
sorted .fastq from /data/analysis/fastq/*.assembled.fastq being used
step 2: editing raw reads
........
real 14m52.577s
user 39m28.511s
sys 0m7.845s
Let's take a look at the results¶
In [1]:
%%bash
cat /data/analysis/stats/s2.rawedit.txt
sample Nreads passed passed.w.trim passed.total
1HL_10A.assembled 2890999 2127357 214294 2341651
1HL_11A.assembled 3088425 2242089 246561 2488650
1HL_12A.assembled 2323231 1728934 163227 1892161
1HL_13A.assembled 2701305 2075000 175929 2250929
1HL_14A.assembled 1865526 1407146 129089 1536235
1HL_15A.assembled 1837684 1342114 139898 1482012
1HL_16A.assembled 2100838 1568118 147901 1716019
1HL_17A.assembled 3106139 2259592 237499 2497091
Nreads = total number of reads for a sample
passed = retained reads that passed quality filtering at full length
passed.w.trim= retained reads that were trimmed due to detection of adapters
passed.total = total kept reads of sufficient length
note: you can set the option in params file to include trimmed reads of xx length.
Step 3: Clustering¶
In [2]:
%%bash
time pyrad -p /data/analysis/params.txt -s 3
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
de-replicating files for clustering...
step 3: within-sample clustering of 8 samples at
'.88' similarity. Running 8 parallel jobs
with up to 16 threads per job. If needed,
adjust to avoid CPU and MEM limits
Process Worker-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Process Worker-8:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Process Worker-16:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Process Worker-9:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Process Worker-13:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Process Worker-14:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 538, in final
makederepclust(outfolder, handle, w1, datatype)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 40, in makederepclust
line[5] = re.sub("\..*\n","\n", line[5])
IndexError: list index out of range
Process Worker-11:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Process Worker-4:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/usr/local/bin/pyrad", line 9, in <module>
load_entry_point('pyrad==3.0.66', 'console_scripts', 'pyrad')()
File "build/bdist.linux-x86_64/egg/pyrad/pyRAD.py", line 340, in main
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 721, in main
File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
real 38m28.984s
user 138m32.476s
sys 8m13.640s
Possibly overloading the CPUs. Let's change to reflect the actual number of cores (8).
In [4]:
%%bash
sed -i '/## 7. /c\8 ## 7. N processors ' /data/analysis/params.txt
In [5]:
%%bash
cat /data/analysis/params.txt
==** parameter inputs for pyRAD version 3.0.66 **======================== affected step ==
/data/analysis/ ## 1. Working directory (all)
/data/analysis/ ## 2. Loc. of non-demultiplexed files (if not line 18) (s1)
/data/analysis/barcodes.txt ## 3. Loc. of barcode file (if not line 18) (s1)
/usr/local/bioinformatics/vsearch-1.11.1-linux-x86_64/bin/vsearch ## 4. command (or path) to call vsearch (or usearch) (s3,s6)
/usr/local/bioinformatics/muscle3.8.31_i86linux64 ## 5. command (or path) to call muscle (s3,s7)
CWGC ## 6. Restriction overhang (e.g., C|TGCAG -> TGCAG) (s1,s2)
8 ## 7. N processors
6 ## 8. Mindepth: min coverage for a cluster (s4,s5)
4 ## 9. NQual: max # sites with qual < 20 (or see line 20)(s2)
.88 ## 10. Wclust: clustering threshold as a decimal (s3,s6)
merged ## 11. data type
4 ## 12. MinCov: min samples in a final locus (s7)
3 ## 13. MaxSH: max inds with shared hetero site (s7)
oly_gbs_pyrad ## 14. Prefix name for final output (no spaces) (s7)
==== optional params below this line =================================== affected step ==
## 15.opt.: select subset (prefix* only selector) (s2-s7)
## 16.opt.: add-on (outgroup) taxa (list or prefix*) (s6,s7)
## 17.opt.: exclude taxa (list or prefix*) (s7)
/data/analysis/fastq/*.assembled.fastq ## 18. demulti data loc
## 19.opt.: maxM: N mismatches in barcodes (def= 1) (s1)
## 20.opt.: phred Qscore offset (def= 33) (s2)
2 ## 21.opt.: filter: def=0=NQual 1=NQual+adapters. 2=strict (s2)
## 22.opt.: a priori E,H (def= 0.001,0.01, if not estimated) (s5)
4 ## 23.opt.: maxN: max Ns in a cons seq (def=5) (s5)
8 ## 24.opt.: maxH: max heterozyg. sites in cons seq (def=5) (s5)
## 25.opt.: ploidy: max alleles in cons seq (def=2;see docs) (s4,s5)
## 26.opt.: maxSNPs: (def=100). Paired (def=100,100) (s7)
## 27.opt.: maxIndels: within-clust,across-clust (def. 3,99) (s3,s7)
## 28.opt.: random number seed (def. 112233) (s3,s6,s7)
## 29.opt.: trim overhang left,right on final loci, def(0,0) (s7)
* ## 30.opt.: output formats: p,n,a,s,v,u,t,m,k,g,* (see docs) (s7)
## 31.opt.: maj. base call at depth>x<mindepth (def.x=mindepth) (s5)
50 ## 32.opt.: keep trimmed reads (def=0). Enter min length. (s2)
## 33.opt.: max stack size (int), def= max(500,mean+2*SD) (s3)
## 34.opt.: minDerep: exclude dereps with <= N copies, def=1 (s3)
## 35.opt.: use hierarchical clustering (def.=0, 1=yes) (s6)
## 36.opt.: repeat masking (def.=1='dust' method, 0=no) (s3,s6)
16 ## 37.opt.: vsearch max threads per job (def.=6; see docs) (s3,s6)
==== optional: list group/clade assignments below this line (see docs) ==================
1HL 5 1HL*
1NF 5 1NF*
1SN 5 1SN*
Let's try this again...
In [6]:
%%bash
time pyrad -p /data/analysis/params.txt -s 3
skipping 1HL_17A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_13A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_16A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_14A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_15A.assembled.clustS.gz already exists in /data/analysis/clust.88
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
de-replicating files for clustering...
step 3: within-sample clustering of 8 samples at
'.88' similarity. Running 8 parallel jobs
with up to 16 threads per job. If needed,
adjust to avoid CPU and MEM limits
Process Worker-6:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Process Worker-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Process Worker-4:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "build/bdist.linux-x86_64/egg/pyrad/potpour.py", line 31, in run
res = self.func(*job)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 545, in final
alignwrap(outfolder+"/"+handle.split("/")[-1].replace(".edit",".clust.gz"), mindepth, muscle, w1)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 415, in alignwrap
stringnames = alignfast(names[0:200],seqs[0:200],muscle)
File "build/bdist.linux-x86_64/egg/pyrad/cluster7dp.py", line 301, in alignfast
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
real 21m28.279s
user 56m45.842s
sys 1m52.941s
Well, I think the problem is that I'm running this at the same time as I'm running ustacks (a different program). It seems that ustacks is using all of the EC2 instance's resources (viewed using the Bash program "htop"). I will wait until that has completed and then run PyRad (or, upgrade the EC2 instance to a a greater number of CPUs or clone this instance and run two separate instances).
In [1]:
%%bash
time pyrad -p /data/analysis/params.txt -s 3
skipping 1HL_17A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_10A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_13A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_12A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_16A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_14A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_15A.assembled.clustS.gz already exists in /data/analysis/clust.88
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
de-replicating files for clustering...
step 3: within-sample clustering of 8 samples at
'.88' similarity. Running 8 parallel jobs
with up to 16 threads per job. If needed,
adjust to avoid CPU and MEM limits
sample 1HL_11A finished, 125407 loci
real 63m55.147s
user 44m17.743s
sys 29m14.039s
Let's take a look at the results¶
In [4]:
%%bash
cat /data/analysis/stats/s3.clusters.txt
taxa total dpt.me dpt.sd d>5.tot d>5.me d>5.sd badpairs
## total = total number of clusters, including singletons
## dpt.me = mean depth of clusters
## dpt.sd = standard deviation of cluster depth
## >N.tot = number of clusters with depth greater than N
## >N.me = mean depth of clusters with depth greater than N
## >N.sd = standard deviation of cluster depth for clusters with depth greater than N
## badpairs = mismatched 1st & 2nd reads (only for paired ddRAD data)
HISTOGRAMS
taxa total dpt.me dpt.sd d>5.tot d>5.me d>5.sd badpairs
## total = total number of clusters, including singletons
## dpt.me = mean depth of clusters
## dpt.sd = standard deviation of cluster depth
## >N.tot = number of clusters with depth greater than N
## >N.me = mean depth of clusters with depth greater than N
## >N.sd = standard deviation of cluster depth for clusters with depth greater than N
## badpairs = mismatched 1st & 2nd reads (only for paired ddRAD data)
HISTOGRAMS
taxa total dpt.me dpt.sd d>5.tot d>5.me d>5.sd badpairs
1HL_11A.assembled 125407 10.143 106.054 39824 27.904 186.958 0
## total = total number of clusters, including singletons
## dpt.me = mean depth of clusters
## dpt.sd = standard deviation of cluster depth
## >N.tot = number of clusters with depth greater than N
## >N.me = mean depth of clusters with depth greater than N
## >N.sd = standard deviation of cluster depth for clusters with depth greater than N
## badpairs = mismatched 1st & 2nd reads (only for paired ddRAD data)
HISTOGRAMS
sample: 1HL_11A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 81060
5 ***** 15999
10 *** 8344
15 ** 5099
20 ** 3282
25 ** 2270
30 * 1646
35 * 1247
40 * 1846
50 ** 3628
100 * 738
250 * 160
500 * 88
Step 4: Parameter estimation¶
In [6]:
%%bash
time pyrad -p /data/analysis/params.txt -s 4
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
step 4: estimating error rate and heterozygosity
........
real 34m34.339s
user 60m29.346s
sys 0m0.623s
In [7]:
%%bash
cat /data/analysis/stats/Pi_E_estimate.txt
taxa H E 1HL_13A.assembled 0.01234178 0.00144187 1HL_10A.assembled 0.01399484 0.00139753 1HL_15A.assembled 0.01242723 0.0011879 1HL_12A.assembled 0.01240698 0.00157456 1HL_16A.assembled 0.01381188 0.00054237 1HL_17A.assembled 0.01332716 0.00127763 1HL_14A.assembled 0.01266762 0.00203182 1HL_11A.assembled 0.01216978 0.00165352
Yeesh, this isn't right. Not all of the individuals are here. Should've noticed this before! Going back to unzipping files before merging paired reads.
In [8]:
%%bash
time gunzip /data/analysis/fastq/*.fq.gz
gzip: /data/analysis/fastq/1HL_19A_R2.fq.gz: unexpected end of file real 0m35.063s user 0m4.390s sys 0m0.746s
In [9]:
ls -lh /data/analysis/fastq/
total 49G -rw-r--r-- 1 root root 762M Jul 18 16:17 1HL_10A.assembled.fastq -rw-r--r-- 1 root root 384M Jul 18 16:17 1HL_10A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 18 16:14 1HL_10A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 18 16:14 1HL_10A.unassembled.reverse.fastq -rw-r--r-- 1 root root 969M Jul 17 06:24 1HL_10A_R1.fq -rw-r--r-- 1 root root 1.0G Jul 17 06:25 1HL_10A_R2.fq -rw-r--r-- 1 root root 822M Jul 18 16:22 1HL_11A.assembled.fastq -rw-r--r-- 1 root root 433M Jul 18 16:22 1HL_11A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 18 16:17 1HL_11A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 18 16:17 1HL_11A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 17 06:25 1HL_11A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 17 06:26 1HL_11A_R2.fq -rw-r--r-- 1 root root 612M Jul 18 16:26 1HL_12A.assembled.fastq -rw-r--r-- 1 root root 299M Jul 18 16:26 1HL_12A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 18 16:22 1HL_12A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 18 16:22 1HL_12A.unassembled.reverse.fastq -rw-r--r-- 1 root root 771M Jul 17 06:26 1HL_12A_R1.fq -rw-r--r-- 1 root root 815M Jul 17 06:26 1HL_12A_R2.fq -rw-r--r-- 1 root root 697M Jul 18 16:31 1HL_13A.assembled.fastq -rw-r--r-- 1 root root 328M Jul 18 16:31 1HL_13A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 18 16:26 1HL_13A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 18 16:26 1HL_13A.unassembled.reverse.fastq -rw-r--r-- 1 root root 882M Jul 17 06:27 1HL_13A_R1.fq -rw-r--r-- 1 root root 932M Jul 17 06:27 1HL_13A_R2.fq -rw-r--r-- 1 root root 487M Jul 18 16:32 1HL_14A.assembled.fastq -rw-r--r-- 1 root root 251M Jul 18 16:32 1HL_14A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 18 16:31 1HL_14A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 18 16:31 1HL_14A.unassembled.reverse.fastq -rw-r--r-- 1 root root 622M Jul 17 06:27 1HL_14A_R1.fq -rw-r--r-- 1 root root 663M Jul 17 06:28 1HL_14A_R2.fq -rw-r--r-- 1 root root 490M Jul 18 16:34 1HL_15A.assembled.fastq -rw-r--r-- 1 root root 277M Jul 18 16:34 1HL_15A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 18 16:33 1HL_15A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 18 16:33 1HL_15A.unassembled.reverse.fastq -rw-r--r-- 1 root root 636M Jul 17 06:28 1HL_15A_R1.fq -rw-r--r-- 1 root root 677M Jul 17 06:28 1HL_15A_R2.fq -rw-r--r-- 1 root root 551M Jul 18 16:37 1HL_16A.assembled.fastq -rw-r--r-- 1 root root 290M Jul 18 16:37 1HL_16A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 18 16:34 1HL_16A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 18 16:34 1HL_16A.unassembled.reverse.fastq -rw-r--r-- 1 root root 707M Jul 17 06:29 1HL_16A_R1.fq -rw-r--r-- 1 root root 753M Jul 17 06:29 1HL_16A_R2.fq -rw-r--r-- 1 root root 826M Jul 18 16:40 1HL_17A.assembled.fastq -rw-r--r-- 1 root root 451M Jul 18 16:40 1HL_17A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 18 16:37 1HL_17A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 18 16:37 1HL_17A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 17 06:29 1HL_17A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 17 06:30 1HL_17A_R2.fq -rw-r--r-- 1 root root 1.1G Jul 17 06:31 1HL_19A_R1.fq -rw-r--r-- 1 root root 168M Jul 17 06:32 1HL_19A_R2.fq.gz -rw-r--r-- 1 root root 133M Jul 17 06:19 1HL_1A_R1.fq.gz -rw-r--r-- 1 root root 164M Jul 17 06:19 1HL_1A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_20A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_20A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_21A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_21A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_22A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_22A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_23A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_23A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_24A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_24A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_25A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_25A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_26A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_26A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_27A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_27A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_28A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_28A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_29A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_29A_R2.fq.gz -rw-r--r-- 1 root root 159M Jul 17 06:19 1HL_2A_R1.fq.gz -rw-r--r-- 1 root root 196M Jul 17 06:19 1HL_2A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_31A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_31A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_33A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_33A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_34A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_34A_R2.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_35A_R1.fq.gz -rw-r--r-- 1 root root 0 Jul 17 06:32 1HL_35A_R2.fq.gz -rw-r--r-- 1 root root 129M Jul 17 06:20 1HL_3A_R1.fq.gz -rw-r--r-- 1 root root 158M Jul 17 06:20 1HL_3A_R2.fq.gz -rw-r--r-- 1 root root 151M Jul 17 06:20 1HL_4A_R1.fq.gz -rw-r--r-- 1 root root 186M Jul 17 06:21 1HL_4A_R2.fq.gz -rw-r--r-- 1 root root 118M Jul 17 06:21 1HL_5A_R1.fq.gz -rw-r--r-- 1 root root 145M Jul 17 06:21 1HL_5A_R2.fq.gz -rw-r--r-- 1 root root 119M Jul 17 06:21 1HL_6A_R1.fq.gz -rw-r--r-- 1 root root 149M Jul 17 06:22 1HL_6A_R2.fq.gz -rw-r--r-- 1 root root 162M Jul 17 06:22 1HL_7A_R1.fq.gz -rw-r--r-- 1 root root 220M Jul 17 06:22 1HL_7A_R2.fq.gz -rw-r--r-- 1 root root 148M Jul 17 06:23 1HL_8A_R1.fq.gz -rw-r--r-- 1 root root 182M Jul 17 06:23 1HL_8A_R2.fq.gz -rw-r--r-- 1 root root 165M Jul 17 06:24 1HL_9A_R1.fq.gz -rw-r--r-- 1 root root 203M Jul 17 06:24 1HL_9A_R2.fq.gz -rw-r--r-- 1 root root 128M Jul 17 05:39 1NF_10A_R1.fq.gz -rw-r--r-- 1 root root 156M Jul 17 05:39 1NF_10A_R2.fq.gz -rw-r--r-- 1 root root 134M Jul 17 05:39 1NF_11A_R1.fq.gz -rw-r--r-- 1 root root 163M Jul 17 05:39 1NF_11A_R2.fq.gz -rw-r--r-- 1 root root 132M Jul 17 05:40 1NF_12A_R1.fq.gz -rw-r--r-- 1 root root 160M Jul 17 05:40 1NF_12A_R2.fq.gz -rw-r--r-- 1 root root 229M Jul 17 05:40 1NF_13A_R1.fq.gz -rw-r--r-- 1 root root 278M Jul 17 05:41 1NF_13A_R2.fq.gz -rw-r--r-- 1 root root 158M Jul 17 05:41 1NF_14A_R1.fq.gz -rw-r--r-- 1 root root 192M Jul 17 05:42 1NF_14A_R2.fq.gz -rw-r--r-- 1 root root 150M Jul 17 05:42 1NF_15A_R1.fq.gz -rw-r--r-- 1 root root 184M Jul 17 05:42 1NF_15A_R2.fq.gz -rw-r--r-- 1 root root 164M Jul 17 05:42 1NF_16A_R1.fq.gz -rw-r--r-- 1 root root 199M Jul 17 05:43 1NF_16A_R2.fq.gz -rw-r--r-- 1 root root 184M Jul 17 05:43 1NF_17A_R1.fq.gz -rw-r--r-- 1 root root 224M Jul 17 05:44 1NF_17A_R2.fq.gz -rw-r--r-- 1 root root 102M Jul 17 05:44 1NF_18A_R1.fq.gz -rw-r--r-- 1 root root 125M Jul 17 05:44 1NF_18A_R2.fq.gz -rw-r--r-- 1 root root 120M Jul 17 05:44 1NF_19A_R1.fq.gz -rw-r--r-- 1 root root 147M Jul 17 05:45 1NF_19A_R2.fq.gz -rw-r--r-- 1 root root 82M Jul 17 05:33 1NF_1A_R1.fq.gz -rw-r--r-- 1 root root 99M Jul 17 05:34 1NF_1A_R2.fq.gz -rw-r--r-- 1 root root 146M Jul 17 05:45 1NF_20A_R1.fq.gz -rw-r--r-- 1 root root 177M Jul 17 05:45 1NF_20A_R2.fq.gz -rw-r--r-- 1 root root 182M Jul 17 05:46 1NF_21A_R1.fq.gz -rw-r--r-- 1 root root 221M Jul 17 05:46 1NF_21A_R2.fq.gz -rw-r--r-- 1 root root 171M Jul 17 05:47 1NF_22A_R1.fq.gz -rw-r--r-- 1 root root 209M Jul 17 05:47 1NF_22A_R2.fq.gz -rw-r--r-- 1 root root 192M Jul 17 05:47 1NF_23A_R1.fq.gz -rw-r--r-- 1 root root 234M Jul 17 05:48 1NF_23A_R2.fq.gz -rw-r--r-- 1 root root 175M Jul 17 05:48 1NF_24A_R1.fq.gz -rw-r--r-- 1 root root 214M Jul 17 05:48 1NF_24A_R2.fq.gz -rw-r--r-- 1 root root 128M Jul 17 05:49 1NF_25A_R1.fq.gz -rw-r--r-- 1 root root 156M Jul 17 05:49 1NF_25A_R2.fq.gz -rw-r--r-- 1 root root 172M Jul 17 05:49 1NF_26A_R1.fq.gz -rw-r--r-- 1 root root 209M Jul 17 05:50 1NF_26A_R2.fq.gz -rw-r--r-- 1 root root 137M Jul 17 05:50 1NF_27A_R1.fq.gz -rw-r--r-- 1 root root 165M Jul 17 05:50 1NF_27A_R2.fq.gz -rw-r--r-- 1 root root 141M Jul 17 05:51 1NF_28A_R1.fq.gz -rw-r--r-- 1 root root 171M Jul 17 05:51 1NF_28A_R2.fq.gz -rw-r--r-- 1 root root 224M Jul 17 05:51 1NF_29A_R1.fq.gz -rw-r--r-- 1 root root 274M Jul 17 05:52 1NF_29A_R2.fq.gz -rw-r--r-- 1 root root 204M Jul 17 05:34 1NF_2A_R1.fq.gz -rw-r--r-- 1 root root 246M Jul 17 05:34 1NF_2A_R2.fq.gz -rw-r--r-- 1 root root 167M Jul 17 05:52 1NF_30A_R1.fq.gz -rw-r--r-- 1 root root 203M Jul 17 05:52 1NF_30A_R2.fq.gz -rw-r--r-- 1 root root 207M Jul 17 05:53 1NF_31A_R1.fq.gz -rw-r--r-- 1 root root 253M Jul 17 05:53 1NF_31A_R2.fq.gz -rw-r--r-- 1 root root 208M Jul 17 05:54 1NF_32A_R1.fq.gz -rw-r--r-- 1 root root 251M Jul 17 05:54 1NF_32A_R2.fq.gz -rw-r--r-- 1 root root 156M Jul 17 05:54 1NF_33A_R1.fq.gz -rw-r--r-- 1 root root 189M Jul 17 05:55 1NF_33A_R2.fq.gz -rw-r--r-- 1 root root 212M Jul 17 05:35 1NF_4A_R1.fq.gz -rw-r--r-- 1 root root 256M Jul 17 05:35 1NF_4A_R2.fq.gz -rw-r--r-- 1 root root 164M Jul 17 05:35 1NF_5A_R1.fq.gz -rw-r--r-- 1 root root 197M Jul 17 05:36 1NF_5A_R2.fq.gz -rw-r--r-- 1 root root 145M Jul 17 05:36 1NF_6A_R1.fq.gz -rw-r--r-- 1 root root 175M Jul 17 05:36 1NF_6A_R2.fq.gz -rw-r--r-- 1 root root 154M Jul 17 05:36 1NF_7A_R1.fq.gz -rw-r--r-- 1 root root 187M Jul 17 05:37 1NF_7A_R2.fq.gz -rw-r--r-- 1 root root 163M Jul 17 05:37 1NF_8A_R1.fq.gz -rw-r--r-- 1 root root 198M Jul 17 05:37 1NF_8A_R2.fq.gz -rw-r--r-- 1 root root 199M Jul 17 05:38 1NF_9A_R1.fq.gz -rw-r--r-- 1 root root 240M Jul 17 05:38 1NF_9A_R2.fq.gz -rw-r--r-- 1 root root 151M Jul 17 06:02 1SN_10A_R1.fq.gz -rw-r--r-- 1 root root 185M Jul 17 06:02 1SN_10A_R2.fq.gz -rw-r--r-- 1 root root 154M Jul 17 06:02 1SN_11A_R1.fq.gz -rw-r--r-- 1 root root 188M Jul 17 06:03 1SN_11A_R2.fq.gz -rw-r--r-- 1 root root 197M Jul 17 06:03 1SN_12A_R1.fq.gz -rw-r--r-- 1 root root 242M Jul 17 06:04 1SN_12A_R2.fq.gz -rw-r--r-- 1 root root 172M Jul 17 06:04 1SN_13A_R1.fq.gz -rw-r--r-- 1 root root 210M Jul 17 06:04 1SN_13A_R2.fq.gz -rw-r--r-- 1 root root 206M Jul 17 06:05 1SN_14A_R1.fq.gz -rw-r--r-- 1 root root 252M Jul 17 06:05 1SN_14A_R2.fq.gz -rw-r--r-- 1 root root 144M Jul 17 06:05 1SN_15A_R1.fq.gz -rw-r--r-- 1 root root 176M Jul 17 06:06 1SN_15A_R2.fq.gz -rw-r--r-- 1 root root 152M Jul 17 06:06 1SN_16A_R1.fq.gz -rw-r--r-- 1 root root 189M Jul 17 06:06 1SN_16A_R2.fq.gz -rw-r--r-- 1 root root 171M Jul 17 06:07 1SN_17A_R1.fq.gz -rw-r--r-- 1 root root 209M Jul 17 06:07 1SN_17A_R2.fq.gz -rw-r--r-- 1 root root 188M Jul 17 06:08 1SN_18A_R1.fq.gz -rw-r--r-- 1 root root 233M Jul 17 06:08 1SN_18A_R2.fq.gz -rw-r--r-- 1 root root 177M Jul 17 06:08 1SN_19A_R1.fq.gz -rw-r--r-- 1 root root 216M Jul 17 06:09 1SN_19A_R2.fq.gz -rw-r--r-- 1 root root 176M Jul 17 05:55 1SN_1A_R1.fq.gz -rw-r--r-- 1 root root 214M Jul 17 05:56 1SN_1A_R2.fq.gz -rw-r--r-- 1 root root 130M Jul 17 06:09 1SN_20A_R1.fq.gz -rw-r--r-- 1 root root 160M Jul 17 06:09 1SN_20A_R2.fq.gz -rw-r--r-- 1 root root 201M Jul 17 06:10 1SN_21A_R1.fq.gz -rw-r--r-- 1 root root 248M Jul 17 06:10 1SN_21A_R2.fq.gz -rw-r--r-- 1 root root 179M Jul 17 06:11 1SN_22A_R1.fq.gz -rw-r--r-- 1 root root 219M Jul 17 06:11 1SN_22A_R2.fq.gz -rw-r--r-- 1 root root 213M Jul 17 06:11 1SN_23A_R1.fq.gz -rw-r--r-- 1 root root 261M Jul 17 06:12 1SN_23A_R2.fq.gz -rw-r--r-- 1 root root 110M Jul 17 06:12 1SN_24A_R1.fq.gz -rw-r--r-- 1 root root 134M Jul 17 06:12 1SN_24A_R2.fq.gz -rw-r--r-- 1 root root 171M Jul 17 06:13 1SN_25A_R1.fq.gz -rw-r--r-- 1 root root 212M Jul 17 06:13 1SN_25A_R2.fq.gz -rw-r--r-- 1 root root 158M Jul 17 06:14 1SN_26A_R1.fq.gz -rw-r--r-- 1 root root 194M Jul 17 06:14 1SN_26A_R2.fq.gz -rw-r--r-- 1 root root 148M Jul 17 06:14 1SN_27A_R1.fq.gz -rw-r--r-- 1 root root 183M Jul 17 06:15 1SN_27A_R2.fq.gz -rw-r--r-- 1 root root 162M Jul 17 06:15 1SN_28A_R1.fq.gz -rw-r--r-- 1 root root 200M Jul 17 06:15 1SN_28A_R2.fq.gz -rw-r--r-- 1 root root 128M Jul 17 06:15 1SN_29A_R1.fq.gz -rw-r--r-- 1 root root 161M Jul 17 06:16 1SN_29A_R2.fq.gz -rw-r--r-- 1 root root 178M Jul 17 05:56 1SN_2A_R1.fq.gz -rw-r--r-- 1 root root 220M Jul 17 05:56 1SN_2A_R2.fq.gz -rw-r--r-- 1 root root 219M Jul 17 06:16 1SN_30A_R1.fq.gz -rw-r--r-- 1 root root 271M Jul 17 06:17 1SN_30A_R2.fq.gz -rw-r--r-- 1 root root 190M Jul 17 06:17 1SN_31A_R1.fq.gz -rw-r--r-- 1 root root 235M Jul 17 06:18 1SN_31A_R2.fq.gz -rw-r--r-- 1 root root 147M Jul 17 06:18 1SN_32A_R1.fq.gz -rw-r--r-- 1 root root 181M Jul 17 06:18 1SN_32A_R2.fq.gz -rw-r--r-- 1 root root 140M Jul 17 05:57 1SN_3A_R1.fq.gz -rw-r--r-- 1 root root 172M Jul 17 05:57 1SN_3A_R2.fq.gz -rw-r--r-- 1 root root 191M Jul 17 05:57 1SN_4A_R1.fq.gz -rw-r--r-- 1 root root 234M Jul 17 05:58 1SN_4A_R2.fq.gz -rw-r--r-- 1 root root 174M Jul 17 05:58 1SN_5A_R1.fq.gz -rw-r--r-- 1 root root 213M Jul 17 05:58 1SN_5A_R2.fq.gz -rw-r--r-- 1 root root 194M Jul 17 05:59 1SN_6A_R1.fq.gz -rw-r--r-- 1 root root 239M Jul 17 05:59 1SN_6A_R2.fq.gz -rw-r--r-- 1 root root 173M Jul 17 05:59 1SN_7A_R1.fq.gz -rw-r--r-- 1 root root 213M Jul 17 06:00 1SN_7A_R2.fq.gz -rw-r--r-- 1 root root 194M Jul 17 06:00 1SN_8A_R1.fq.gz -rw-r--r-- 1 root root 237M Jul 17 06:01 1SN_8A_R2.fq.gz -rw-r--r-- 1 root root 185M Jul 17 06:01 1SN_9A_R1.fq.gz -rw-r--r-- 1 root root 227M Jul 17 06:01 1SN_9A_R2.fq.gz -rw-r--r-- 1 root root 21K Jul 18 16:40 pear.log
Well, it turns out the the original demultiplexing actually failed. Many fq.gz files that are empty (i.e. 0 megabytes in size). Need to re-run demultiplexing.
Need to change data type back to pairgbs.¶
In [10]:
%%bash
sed -i '/## 11./c\pairgbs ## 11. data type ' /data/analysis/params.txt
In [12]:
%%bash
cat /data/analysis/params.txt
==** parameter inputs for pyRAD version 3.0.66 **======================== affected step ==
/data/analysis/ ## 1. Working directory (all)
/data/analysis/ ## 2. Loc. of non-demultiplexed files (if not line 18) (s1)
/data/analysis/barcodes.txt ## 3. Loc. of barcode file (if not line 18) (s1)
/usr/local/bioinformatics/vsearch-1.11.1-linux-x86_64/bin/vsearch ## 4. command (or path) to call vsearch (or usearch) (s3,s6)
/usr/local/bioinformatics/muscle3.8.31_i86linux64 ## 5. command (or path) to call muscle (s3,s7)
CWGC ## 6. Restriction overhang (e.g., C|TGCAG -> TGCAG) (s1,s2)
8 ## 7. N processors
6 ## 8. Mindepth: min coverage for a cluster (s4,s5)
4 ## 9. NQual: max # sites with qual < 20 (or see line 20)(s2)
.88 ## 10. Wclust: clustering threshold as a decimal (s3,s6)
pairgbs ## 11. data type
4 ## 12. MinCov: min samples in a final locus (s7)
3 ## 13. MaxSH: max inds with shared hetero site (s7)
oly_gbs_pyrad ## 14. Prefix name for final output (no spaces) (s7)
==== optional params below this line =================================== affected step ==
## 15.opt.: select subset (prefix* only selector) (s2-s7)
## 16.opt.: add-on (outgroup) taxa (list or prefix*) (s6,s7)
## 17.opt.: exclude taxa (list or prefix*) (s7)
/data/analysis/fastq/*.assembled.fastq ## 18. demulti data loc
## 19.opt.: maxM: N mismatches in barcodes (def= 1) (s1)
## 20.opt.: phred Qscore offset (def= 33) (s2)
2 ## 21.opt.: filter: def=0=NQual 1=NQual+adapters. 2=strict (s2)
## 22.opt.: a priori E,H (def= 0.001,0.01, if not estimated) (s5)
4 ## 23.opt.: maxN: max Ns in a cons seq (def=5) (s5)
8 ## 24.opt.: maxH: max heterozyg. sites in cons seq (def=5) (s5)
## 25.opt.: ploidy: max alleles in cons seq (def=2;see docs) (s4,s5)
## 26.opt.: maxSNPs: (def=100). Paired (def=100,100) (s7)
## 27.opt.: maxIndels: within-clust,across-clust (def. 3,99) (s3,s7)
## 28.opt.: random number seed (def. 112233) (s3,s6,s7)
## 29.opt.: trim overhang left,right on final loci, def(0,0) (s7)
* ## 30.opt.: output formats: p,n,a,s,v,u,t,m,k,g,* (see docs) (s7)
## 31.opt.: maj. base call at depth>x<mindepth (def.x=mindepth) (s5)
50 ## 32.opt.: keep trimmed reads (def=0). Enter min length. (s2)
## 33.opt.: max stack size (int), def= max(500,mean+2*SD) (s3)
## 34.opt.: minDerep: exclude dereps with <= N copies, def=1 (s3)
## 35.opt.: use hierarchical clustering (def.=0, 1=yes) (s6)
## 36.opt.: repeat masking (def.=1='dust' method, 0=no) (s3,s6)
16 ## 37.opt.: vsearch max threads per job (def.=6; see docs) (s3,s6)
==== optional: list group/clade assignments below this line (see docs) ==================
1HL 5 1HL*
1NF 5 1NF*
1SN 5 1SN*
In [13]:
%%bash
ls /data/analysis/
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz barcodes.txt clust.88 edits fastq params.txt stats
In [14]:
%%bash
rm -rf /data/analysis/fastq/
In [15]:
%%bash
ls /data/analysis/
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz barcodes.txt clust.88 edits params.txt stats
In [16]:
cd /data/analysis/
/data/analysis
De-multiplex reads (again)
In [17]:
%%bash
time pyrad -p /data/analysis/params.txt -s 1
skipping step 1: line 18 of input file shows seqs already sorted
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
real 0m0.225s
user 0m0.202s
sys 0m0.023s
In [18]:
%%bash
## set location of demultiplexed data that are 'pear' filtered
sed -i '/## 18./c\ ## 18. demulti data loc ' /data/analysis/params.txt
In [19]:
%%bash
cat /data/analysis/params.txt
==** parameter inputs for pyRAD version 3.0.66 **======================== affected step ==
/data/analysis/ ## 1. Working directory (all)
/data/analysis/ ## 2. Loc. of non-demultiplexed files (if not line 18) (s1)
/data/analysis/barcodes.txt ## 3. Loc. of barcode file (if not line 18) (s1)
/usr/local/bioinformatics/vsearch-1.11.1-linux-x86_64/bin/vsearch ## 4. command (or path) to call vsearch (or usearch) (s3,s6)
/usr/local/bioinformatics/muscle3.8.31_i86linux64 ## 5. command (or path) to call muscle (s3,s7)
CWGC ## 6. Restriction overhang (e.g., C|TGCAG -> TGCAG) (s1,s2)
8 ## 7. N processors
6 ## 8. Mindepth: min coverage for a cluster (s4,s5)
4 ## 9. NQual: max # sites with qual < 20 (or see line 20)(s2)
.88 ## 10. Wclust: clustering threshold as a decimal (s3,s6)
pairgbs ## 11. data type
4 ## 12. MinCov: min samples in a final locus (s7)
3 ## 13. MaxSH: max inds with shared hetero site (s7)
oly_gbs_pyrad ## 14. Prefix name for final output (no spaces) (s7)
==== optional params below this line =================================== affected step ==
## 15.opt.: select subset (prefix* only selector) (s2-s7)
## 16.opt.: add-on (outgroup) taxa (list or prefix*) (s6,s7)
## 17.opt.: exclude taxa (list or prefix*) (s7)
## 18. demulti data loc
## 19.opt.: maxM: N mismatches in barcodes (def= 1) (s1)
## 20.opt.: phred Qscore offset (def= 33) (s2)
2 ## 21.opt.: filter: def=0=NQual 1=NQual+adapters. 2=strict (s2)
## 22.opt.: a priori E,H (def= 0.001,0.01, if not estimated) (s5)
4 ## 23.opt.: maxN: max Ns in a cons seq (def=5) (s5)
8 ## 24.opt.: maxH: max heterozyg. sites in cons seq (def=5) (s5)
## 25.opt.: ploidy: max alleles in cons seq (def=2;see docs) (s4,s5)
## 26.opt.: maxSNPs: (def=100). Paired (def=100,100) (s7)
## 27.opt.: maxIndels: within-clust,across-clust (def. 3,99) (s3,s7)
## 28.opt.: random number seed (def. 112233) (s3,s6,s7)
## 29.opt.: trim overhang left,right on final loci, def(0,0) (s7)
* ## 30.opt.: output formats: p,n,a,s,v,u,t,m,k,g,* (see docs) (s7)
## 31.opt.: maj. base call at depth>x<mindepth (def.x=mindepth) (s5)
50 ## 32.opt.: keep trimmed reads (def=0). Enter min length. (s2)
## 33.opt.: max stack size (int), def= max(500,mean+2*SD) (s3)
## 34.opt.: minDerep: exclude dereps with <= N copies, def=1 (s3)
## 35.opt.: use hierarchical clustering (def.=0, 1=yes) (s6)
## 36.opt.: repeat masking (def.=1='dust' method, 0=no) (s3,s6)
16 ## 37.opt.: vsearch max threads per job (def.=6; see docs) (s3,s6)
==== optional: list group/clade assignments below this line (see docs) ==================
1HL 5 1HL*
1NF 5 1NF*
1SN 5 1SN*
In [20]:
%%bash
time pyrad -p /data/analysis/params.txt -s 1
warning: CUT site matches within one of the barcodes, I suggest double checking the file to make sure it properly demultiplexes
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
step 1: sorting reads by barcode
.
real 771m53.005s
user 687m46.614s
sys 3m12.997s
Let's check the output and make sure the files actually contain data...
In [22]:
ls -lh /data/analysis/fastq/
total 35G -rw-r--r-- 1 root root 183M Jul 20 08:30 1HL_10A_R1.fq.gz -rw-r--r-- 1 root root 226M Jul 20 08:31 1HL_10A_R2.fq.gz -rw-r--r-- 1 root root 200M Jul 20 08:31 1HL_11A_R1.fq.gz -rw-r--r-- 1 root root 247M Jul 20 08:32 1HL_11A_R2.fq.gz -rw-r--r-- 1 root root 146M Jul 20 08:32 1HL_12A_R1.fq.gz -rw-r--r-- 1 root root 181M Jul 20 08:32 1HL_12A_R2.fq.gz -rw-r--r-- 1 root root 159M Jul 20 08:33 1HL_13A_R1.fq.gz -rw-r--r-- 1 root root 197M Jul 20 08:33 1HL_13A_R2.fq.gz -rw-r--r-- 1 root root 117M Jul 20 08:33 1HL_14A_R1.fq.gz -rw-r--r-- 1 root root 146M Jul 20 08:34 1HL_14A_R2.fq.gz -rw-r--r-- 1 root root 123M Jul 20 08:34 1HL_15A_R1.fq.gz -rw-r--r-- 1 root root 154M Jul 20 08:34 1HL_15A_R2.fq.gz -rw-r--r-- 1 root root 133M Jul 20 08:35 1HL_16A_R1.fq.gz -rw-r--r-- 1 root root 166M Jul 20 08:35 1HL_16A_R2.fq.gz -rw-r--r-- 1 root root 201M Jul 20 08:36 1HL_17A_R1.fq.gz -rw-r--r-- 1 root root 252M Jul 20 08:36 1HL_17A_R2.fq.gz -rw-r--r-- 1 root root 202M Jul 20 08:37 1HL_19A_R1.fq.gz -rw-r--r-- 1 root root 252M Jul 20 08:37 1HL_19A_R2.fq.gz -rw-r--r-- 1 root root 133M Jul 20 08:24 1HL_1A_R1.fq.gz -rw-r--r-- 1 root root 164M Jul 20 08:24 1HL_1A_R2.fq.gz -rw-r--r-- 1 root root 162M Jul 20 08:38 1HL_20A_R1.fq.gz -rw-r--r-- 1 root root 203M Jul 20 08:38 1HL_20A_R2.fq.gz -rw-r--r-- 1 root root 145M Jul 20 08:38 1HL_21A_R1.fq.gz -rw-r--r-- 1 root root 181M Jul 20 08:39 1HL_21A_R2.fq.gz -rw-r--r-- 1 root root 119M Jul 20 08:39 1HL_22A_R1.fq.gz -rw-r--r-- 1 root root 148M Jul 20 08:39 1HL_22A_R2.fq.gz -rw-r--r-- 1 root root 201M Jul 20 08:40 1HL_23A_R1.fq.gz -rw-r--r-- 1 root root 251M Jul 20 08:40 1HL_23A_R2.fq.gz -rw-r--r-- 1 root root 151M Jul 20 08:41 1HL_24A_R1.fq.gz -rw-r--r-- 1 root root 189M Jul 20 08:41 1HL_24A_R2.fq.gz -rw-r--r-- 1 root root 167M Jul 20 08:42 1HL_25A_R1.fq.gz -rw-r--r-- 1 root root 208M Jul 20 08:42 1HL_25A_R2.fq.gz -rw-r--r-- 1 root root 179M Jul 20 08:42 1HL_26A_R1.fq.gz -rw-r--r-- 1 root root 224M Jul 20 08:43 1HL_26A_R2.fq.gz -rw-r--r-- 1 root root 182M Jul 20 08:43 1HL_27A_R1.fq.gz -rw-r--r-- 1 root root 227M Jul 20 08:44 1HL_27A_R2.fq.gz -rw-r--r-- 1 root root 209M Jul 20 08:44 1HL_28A_R1.fq.gz -rw-r--r-- 1 root root 282M Jul 20 08:45 1HL_28A_R2.fq.gz -rw-r--r-- 1 root root 177M Jul 20 08:45 1HL_29A_R1.fq.gz -rw-r--r-- 1 root root 222M Jul 20 08:46 1HL_29A_R2.fq.gz -rw-r--r-- 1 root root 159M Jul 20 08:24 1HL_2A_R1.fq.gz -rw-r--r-- 1 root root 196M Jul 20 08:25 1HL_2A_R2.fq.gz -rw-r--r-- 1 root root 139M Jul 20 08:46 1HL_31A_R1.fq.gz -rw-r--r-- 1 root root 174M Jul 20 08:46 1HL_31A_R2.fq.gz -rw-r--r-- 1 root root 151M Jul 20 08:47 1HL_33A_R1.fq.gz -rw-r--r-- 1 root root 190M Jul 20 08:47 1HL_33A_R2.fq.gz -rw-r--r-- 1 root root 161M Jul 20 08:47 1HL_34A_R1.fq.gz -rw-r--r-- 1 root root 201M Jul 20 08:48 1HL_34A_R2.fq.gz -rw-r--r-- 1 root root 146M Jul 20 08:48 1HL_35A_R1.fq.gz -rw-r--r-- 1 root root 183M Jul 20 08:49 1HL_35A_R2.fq.gz -rw-r--r-- 1 root root 129M Jul 20 08:25 1HL_3A_R1.fq.gz -rw-r--r-- 1 root root 158M Jul 20 08:25 1HL_3A_R2.fq.gz -rw-r--r-- 1 root root 151M Jul 20 08:26 1HL_4A_R1.fq.gz -rw-r--r-- 1 root root 186M Jul 20 08:26 1HL_4A_R2.fq.gz -rw-r--r-- 1 root root 118M Jul 20 08:26 1HL_5A_R1.fq.gz -rw-r--r-- 1 root root 145M Jul 20 08:27 1HL_5A_R2.fq.gz -rw-r--r-- 1 root root 119M Jul 20 08:27 1HL_6A_R1.fq.gz -rw-r--r-- 1 root root 149M Jul 20 08:27 1HL_6A_R2.fq.gz -rw-r--r-- 1 root root 162M Jul 20 08:28 1HL_7A_R1.fq.gz -rw-r--r-- 1 root root 220M Jul 20 08:28 1HL_7A_R2.fq.gz -rw-r--r-- 1 root root 148M Jul 20 08:28 1HL_8A_R1.fq.gz -rw-r--r-- 1 root root 182M Jul 20 08:29 1HL_8A_R2.fq.gz -rw-r--r-- 1 root root 165M Jul 20 08:29 1HL_9A_R1.fq.gz -rw-r--r-- 1 root root 203M Jul 20 08:30 1HL_9A_R2.fq.gz -rw-r--r-- 1 root root 128M Jul 20 07:39 1NF_10A_R1.fq.gz -rw-r--r-- 1 root root 156M Jul 20 07:40 1NF_10A_R2.fq.gz -rw-r--r-- 1 root root 134M Jul 20 07:40 1NF_11A_R1.fq.gz -rw-r--r-- 1 root root 163M Jul 20 07:40 1NF_11A_R2.fq.gz -rw-r--r-- 1 root root 132M Jul 20 07:41 1NF_12A_R1.fq.gz -rw-r--r-- 1 root root 160M Jul 20 07:41 1NF_12A_R2.fq.gz -rw-r--r-- 1 root root 229M Jul 20 07:41 1NF_13A_R1.fq.gz -rw-r--r-- 1 root root 278M Jul 20 07:42 1NF_13A_R2.fq.gz -rw-r--r-- 1 root root 158M Jul 20 07:42 1NF_14A_R1.fq.gz -rw-r--r-- 1 root root 192M Jul 20 07:42 1NF_14A_R2.fq.gz -rw-r--r-- 1 root root 150M Jul 20 07:43 1NF_15A_R1.fq.gz -rw-r--r-- 1 root root 184M Jul 20 07:43 1NF_15A_R2.fq.gz -rw-r--r-- 1 root root 164M Jul 20 07:44 1NF_16A_R1.fq.gz -rw-r--r-- 1 root root 199M Jul 20 07:44 1NF_16A_R2.fq.gz -rw-r--r-- 1 root root 184M Jul 20 07:44 1NF_17A_R1.fq.gz -rw-r--r-- 1 root root 224M Jul 20 07:45 1NF_17A_R2.fq.gz -rw-r--r-- 1 root root 102M Jul 20 07:45 1NF_18A_R1.fq.gz -rw-r--r-- 1 root root 125M Jul 20 07:45 1NF_18A_R2.fq.gz -rw-r--r-- 1 root root 120M Jul 20 07:46 1NF_19A_R1.fq.gz -rw-r--r-- 1 root root 147M Jul 20 07:46 1NF_19A_R2.fq.gz -rw-r--r-- 1 root root 82M Jul 20 07:34 1NF_1A_R1.fq.gz -rw-r--r-- 1 root root 99M Jul 20 07:34 1NF_1A_R2.fq.gz -rw-r--r-- 1 root root 146M Jul 20 07:46 1NF_20A_R1.fq.gz -rw-r--r-- 1 root root 177M Jul 20 07:47 1NF_20A_R2.fq.gz -rw-r--r-- 1 root root 182M Jul 20 07:47 1NF_21A_R1.fq.gz -rw-r--r-- 1 root root 221M Jul 20 07:47 1NF_21A_R2.fq.gz -rw-r--r-- 1 root root 171M Jul 20 07:48 1NF_22A_R1.fq.gz -rw-r--r-- 1 root root 209M Jul 20 07:48 1NF_22A_R2.fq.gz -rw-r--r-- 1 root root 192M Jul 20 07:48 1NF_23A_R1.fq.gz -rw-r--r-- 1 root root 234M Jul 20 07:49 1NF_23A_R2.fq.gz -rw-r--r-- 1 root root 175M Jul 20 07:49 1NF_24A_R1.fq.gz -rw-r--r-- 1 root root 214M Jul 20 07:50 1NF_24A_R2.fq.gz -rw-r--r-- 1 root root 128M Jul 20 07:50 1NF_25A_R1.fq.gz -rw-r--r-- 1 root root 156M Jul 20 07:50 1NF_25A_R2.fq.gz -rw-r--r-- 1 root root 172M Jul 20 07:51 1NF_26A_R1.fq.gz -rw-r--r-- 1 root root 209M Jul 20 07:51 1NF_26A_R2.fq.gz -rw-r--r-- 1 root root 137M Jul 20 07:51 1NF_27A_R1.fq.gz -rw-r--r-- 1 root root 165M Jul 20 07:52 1NF_27A_R2.fq.gz -rw-r--r-- 1 root root 141M Jul 20 07:52 1NF_28A_R1.fq.gz -rw-r--r-- 1 root root 171M Jul 20 07:52 1NF_28A_R2.fq.gz -rw-r--r-- 1 root root 224M Jul 20 07:53 1NF_29A_R1.fq.gz -rw-r--r-- 1 root root 274M Jul 20 07:53 1NF_29A_R2.fq.gz -rw-r--r-- 1 root root 204M Jul 20 07:34 1NF_2A_R1.fq.gz -rw-r--r-- 1 root root 246M Jul 20 07:34 1NF_2A_R2.fq.gz -rw-r--r-- 1 root root 167M Jul 20 07:54 1NF_30A_R1.fq.gz -rw-r--r-- 1 root root 203M Jul 20 07:54 1NF_30A_R2.fq.gz -rw-r--r-- 1 root root 207M Jul 20 07:55 1NF_31A_R1.fq.gz -rw-r--r-- 1 root root 253M Jul 20 07:55 1NF_31A_R2.fq.gz -rw-r--r-- 1 root root 208M Jul 20 07:56 1NF_32A_R1.fq.gz -rw-r--r-- 1 root root 251M Jul 20 07:56 1NF_32A_R2.fq.gz -rw-r--r-- 1 root root 156M Jul 20 07:56 1NF_33A_R1.fq.gz -rw-r--r-- 1 root root 189M Jul 20 07:57 1NF_33A_R2.fq.gz -rw-r--r-- 1 root root 212M Jul 20 07:35 1NF_4A_R1.fq.gz -rw-r--r-- 1 root root 256M Jul 20 07:35 1NF_4A_R2.fq.gz -rw-r--r-- 1 root root 164M Jul 20 07:35 1NF_5A_R1.fq.gz -rw-r--r-- 1 root root 197M Jul 20 07:36 1NF_5A_R2.fq.gz -rw-r--r-- 1 root root 145M Jul 20 07:36 1NF_6A_R1.fq.gz -rw-r--r-- 1 root root 175M Jul 20 07:37 1NF_6A_R2.fq.gz -rw-r--r-- 1 root root 154M Jul 20 07:37 1NF_7A_R1.fq.gz -rw-r--r-- 1 root root 187M Jul 20 07:37 1NF_7A_R2.fq.gz -rw-r--r-- 1 root root 163M Jul 20 07:38 1NF_8A_R1.fq.gz -rw-r--r-- 1 root root 198M Jul 20 07:38 1NF_8A_R2.fq.gz -rw-r--r-- 1 root root 199M Jul 20 07:39 1NF_9A_R1.fq.gz -rw-r--r-- 1 root root 240M Jul 20 07:39 1NF_9A_R2.fq.gz -rw-r--r-- 1 root root 151M Jul 20 08:05 1SN_10A_R1.fq.gz -rw-r--r-- 1 root root 185M Jul 20 08:05 1SN_10A_R2.fq.gz -rw-r--r-- 1 root root 154M Jul 20 08:06 1SN_11A_R1.fq.gz -rw-r--r-- 1 root root 188M Jul 20 08:06 1SN_11A_R2.fq.gz -rw-r--r-- 1 root root 197M Jul 20 08:06 1SN_12A_R1.fq.gz -rw-r--r-- 1 root root 242M Jul 20 08:07 1SN_12A_R2.fq.gz -rw-r--r-- 1 root root 172M Jul 20 08:08 1SN_13A_R1.fq.gz -rw-r--r-- 1 root root 210M Jul 20 08:08 1SN_13A_R2.fq.gz -rw-r--r-- 1 root root 206M Jul 20 08:08 1SN_14A_R1.fq.gz -rw-r--r-- 1 root root 252M Jul 20 08:09 1SN_14A_R2.fq.gz -rw-r--r-- 1 root root 144M Jul 20 08:09 1SN_15A_R1.fq.gz -rw-r--r-- 1 root root 176M Jul 20 08:10 1SN_15A_R2.fq.gz -rw-r--r-- 1 root root 152M Jul 20 08:10 1SN_16A_R1.fq.gz -rw-r--r-- 1 root root 189M Jul 20 08:10 1SN_16A_R2.fq.gz -rw-r--r-- 1 root root 171M Jul 20 08:11 1SN_17A_R1.fq.gz -rw-r--r-- 1 root root 209M Jul 20 08:11 1SN_17A_R2.fq.gz -rw-r--r-- 1 root root 188M Jul 20 08:12 1SN_18A_R1.fq.gz -rw-r--r-- 1 root root 233M Jul 20 08:12 1SN_18A_R2.fq.gz -rw-r--r-- 1 root root 177M Jul 20 08:12 1SN_19A_R1.fq.gz -rw-r--r-- 1 root root 216M Jul 20 08:13 1SN_19A_R2.fq.gz -rw-r--r-- 1 root root 176M Jul 20 07:57 1SN_1A_R1.fq.gz -rw-r--r-- 1 root root 214M Jul 20 07:58 1SN_1A_R2.fq.gz -rw-r--r-- 1 root root 130M Jul 20 08:13 1SN_20A_R1.fq.gz -rw-r--r-- 1 root root 160M Jul 20 08:14 1SN_20A_R2.fq.gz -rw-r--r-- 1 root root 201M Jul 20 08:14 1SN_21A_R1.fq.gz -rw-r--r-- 1 root root 248M Jul 20 08:14 1SN_21A_R2.fq.gz -rw-r--r-- 1 root root 179M Jul 20 08:15 1SN_22A_R1.fq.gz -rw-r--r-- 1 root root 219M Jul 20 08:15 1SN_22A_R2.fq.gz -rw-r--r-- 1 root root 213M Jul 20 08:16 1SN_23A_R1.fq.gz -rw-r--r-- 1 root root 261M Jul 20 08:16 1SN_23A_R2.fq.gz -rw-r--r-- 1 root root 110M Jul 20 08:17 1SN_24A_R1.fq.gz -rw-r--r-- 1 root root 134M Jul 20 08:17 1SN_24A_R2.fq.gz -rw-r--r-- 1 root root 171M Jul 20 08:17 1SN_25A_R1.fq.gz -rw-r--r-- 1 root root 212M Jul 20 08:18 1SN_25A_R2.fq.gz -rw-r--r-- 1 root root 158M Jul 20 08:18 1SN_26A_R1.fq.gz -rw-r--r-- 1 root root 194M Jul 20 08:19 1SN_26A_R2.fq.gz -rw-r--r-- 1 root root 148M Jul 20 08:19 1SN_27A_R1.fq.gz -rw-r--r-- 1 root root 183M Jul 20 08:19 1SN_27A_R2.fq.gz -rw-r--r-- 1 root root 162M Jul 20 08:20 1SN_28A_R1.fq.gz -rw-r--r-- 1 root root 200M Jul 20 08:20 1SN_28A_R2.fq.gz -rw-r--r-- 1 root root 128M Jul 20 08:20 1SN_29A_R1.fq.gz -rw-r--r-- 1 root root 161M Jul 20 08:21 1SN_29A_R2.fq.gz -rw-r--r-- 1 root root 178M Jul 20 07:58 1SN_2A_R1.fq.gz -rw-r--r-- 1 root root 220M Jul 20 07:59 1SN_2A_R2.fq.gz -rw-r--r-- 1 root root 219M Jul 20 08:21 1SN_30A_R1.fq.gz -rw-r--r-- 1 root root 271M Jul 20 08:22 1SN_30A_R2.fq.gz -rw-r--r-- 1 root root 190M Jul 20 08:22 1SN_31A_R1.fq.gz -rw-r--r-- 1 root root 235M Jul 20 08:23 1SN_31A_R2.fq.gz -rw-r--r-- 1 root root 147M Jul 20 08:23 1SN_32A_R1.fq.gz -rw-r--r-- 1 root root 181M Jul 20 08:23 1SN_32A_R2.fq.gz -rw-r--r-- 1 root root 140M Jul 20 07:59 1SN_3A_R1.fq.gz -rw-r--r-- 1 root root 172M Jul 20 07:59 1SN_3A_R2.fq.gz -rw-r--r-- 1 root root 191M Jul 20 08:00 1SN_4A_R1.fq.gz -rw-r--r-- 1 root root 234M Jul 20 08:00 1SN_4A_R2.fq.gz -rw-r--r-- 1 root root 174M Jul 20 08:01 1SN_5A_R1.fq.gz -rw-r--r-- 1 root root 213M Jul 20 08:01 1SN_5A_R2.fq.gz -rw-r--r-- 1 root root 194M Jul 20 08:01 1SN_6A_R1.fq.gz -rw-r--r-- 1 root root 239M Jul 20 08:02 1SN_6A_R2.fq.gz -rw-r--r-- 1 root root 173M Jul 20 08:02 1SN_7A_R1.fq.gz -rw-r--r-- 1 root root 213M Jul 20 08:03 1SN_7A_R2.fq.gz -rw-r--r-- 1 root root 194M Jul 20 08:03 1SN_8A_R1.fq.gz -rw-r--r-- 1 root root 237M Jul 20 08:04 1SN_8A_R2.fq.gz -rw-r--r-- 1 root root 185M Jul 20 08:04 1SN_9A_R1.fq.gz -rw-r--r-- 1 root root 227M Jul 20 08:04 1SN_9A_R2.fq.gz
And, let's make sure there's the expected number of individuals (96 individuals, sequenced from each end, so 192 total files)...
In [23]:
ls -1 /data/analysis/fastq/ | wc -l
192
Merge paired reads¶
The first thing we have to do is unzip the reads if they are gzipped. If you demultiplexed your data in pyrad they will be in the fastq/ directory.
In [24]:
%%bash
time gunzip /data/analysis/fastq/*.fq.gz
real 145m40.754s user 14m53.245s sys 2m6.261s
In [25]:
ls /data/analysis/stats/
Pi_E_estimate.txt s1.sorting.txt s2.rawedit.txt s3.clusters.txt
In [26]:
ls /data/analysis/
160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R1_.fq.gz edits/ 160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_R2_.fq.gz fastq/ barcodes.txt params.txt clust.88/ stats/
In [27]:
%%bash
time for gfile in /data/analysis/fastq/*_R1.fq;
do /usr/local/bioinformatics/pear-0.9.6-bin-64/pear-0.9.6-bin-64 -f $gfile \
-r ${gfile/_R1.fq/_R2.fq} \
-o ${gfile/_R1.fq/} \
-n 33 \
-t 33 \
-q 10 \
-j 20 >> pear.log 2>&1;
done
real 173m24.008s user 940m23.125s sys 2m39.123s
In [29]:
ls -lh /data/analysis/fastq/
total 263G -rw-r--r-- 1 root root 762M Jul 20 16:53 1HL_10A.assembled.fastq -rw-r--r-- 1 root root 384M Jul 20 16:53 1HL_10A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 16:52 1HL_10A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 16:52 1HL_10A.unassembled.reverse.fastq -rw-r--r-- 1 root root 969M Jul 20 08:30 1HL_10A_R1.fq -rw-r--r-- 1 root root 1.0G Jul 20 08:31 1HL_10A_R2.fq -rw-r--r-- 1 root root 822M Jul 20 16:55 1HL_11A.assembled.fastq -rw-r--r-- 1 root root 433M Jul 20 16:55 1HL_11A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 16:53 1HL_11A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 16:53 1HL_11A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:31 1HL_11A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:32 1HL_11A_R2.fq -rw-r--r-- 1 root root 612M Jul 20 16:57 1HL_12A.assembled.fastq -rw-r--r-- 1 root root 299M Jul 20 16:57 1HL_12A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 16:55 1HL_12A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 16:55 1HL_12A.unassembled.reverse.fastq -rw-r--r-- 1 root root 771M Jul 20 08:32 1HL_12A_R1.fq -rw-r--r-- 1 root root 815M Jul 20 08:32 1HL_12A_R2.fq -rw-r--r-- 1 root root 697M Jul 20 16:59 1HL_13A.assembled.fastq -rw-r--r-- 1 root root 328M Jul 20 16:59 1HL_13A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 16:57 1HL_13A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 16:57 1HL_13A.unassembled.reverse.fastq -rw-r--r-- 1 root root 882M Jul 20 08:33 1HL_13A_R1.fq -rw-r--r-- 1 root root 932M Jul 20 08:33 1HL_13A_R2.fq -rw-r--r-- 1 root root 487M Jul 20 17:00 1HL_14A.assembled.fastq -rw-r--r-- 1 root root 251M Jul 20 17:00 1HL_14A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 16:59 1HL_14A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 16:59 1HL_14A.unassembled.reverse.fastq -rw-r--r-- 1 root root 622M Jul 20 08:33 1HL_14A_R1.fq -rw-r--r-- 1 root root 663M Jul 20 08:34 1HL_14A_R2.fq -rw-r--r-- 1 root root 490M Jul 20 17:02 1HL_15A.assembled.fastq -rw-r--r-- 1 root root 277M Jul 20 17:02 1HL_15A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:00 1HL_15A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:00 1HL_15A.unassembled.reverse.fastq -rw-r--r-- 1 root root 636M Jul 20 08:34 1HL_15A_R1.fq -rw-r--r-- 1 root root 677M Jul 20 08:34 1HL_15A_R2.fq -rw-r--r-- 1 root root 551M Jul 20 17:03 1HL_16A.assembled.fastq -rw-r--r-- 1 root root 290M Jul 20 17:03 1HL_16A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:02 1HL_16A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:02 1HL_16A.unassembled.reverse.fastq -rw-r--r-- 1 root root 707M Jul 20 08:35 1HL_16A_R1.fq -rw-r--r-- 1 root root 753M Jul 20 08:35 1HL_16A_R2.fq -rw-r--r-- 1 root root 826M Jul 20 17:05 1HL_17A.assembled.fastq -rw-r--r-- 1 root root 451M Jul 20 17:05 1HL_17A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:03 1HL_17A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:03 1HL_17A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:36 1HL_17A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 20 08:36 1HL_17A_R2.fq -rw-r--r-- 1 root root 841M Jul 20 17:07 1HL_19A.assembled.fastq -rw-r--r-- 1 root root 437M Jul 20 17:07 1HL_19A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:05 1HL_19A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:05 1HL_19A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:37 1HL_19A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 20 08:37 1HL_19A_R2.fq -rw-r--r-- 1 root root 549M Jul 20 17:09 1HL_1A.assembled.fastq -rw-r--r-- 1 root root 278M Jul 20 17:09 1HL_1A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:07 1HL_1A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:07 1HL_1A.unassembled.reverse.fastq -rw-r--r-- 1 root root 703M Jul 20 08:24 1HL_1A_R1.fq -rw-r--r-- 1 root root 743M Jul 20 08:24 1HL_1A_R2.fq -rw-r--r-- 1 root root 666M Jul 20 17:11 1HL_20A.assembled.fastq -rw-r--r-- 1 root root 358M Jul 20 17:11 1HL_20A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:09 1HL_20A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:09 1HL_20A.unassembled.reverse.fastq -rw-r--r-- 1 root root 855M Jul 20 08:38 1HL_20A_R1.fq -rw-r--r-- 1 root root 911M Jul 20 08:38 1HL_20A_R2.fq -rw-r--r-- 1 root root 585M Jul 20 17:12 1HL_21A.assembled.fastq -rw-r--r-- 1 root root 328M Jul 20 17:12 1HL_21A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:11 1HL_21A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:11 1HL_21A.unassembled.reverse.fastq -rw-r--r-- 1 root root 759M Jul 20 08:38 1HL_21A_R1.fq -rw-r--r-- 1 root root 809M Jul 20 08:39 1HL_21A_R2.fq -rw-r--r-- 1 root root 478M Jul 20 17:14 1HL_22A.assembled.fastq -rw-r--r-- 1 root root 262M Jul 20 17:14 1HL_22A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:12 1HL_22A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:12 1HL_22A.unassembled.reverse.fastq -rw-r--r-- 1 root root 617M Jul 20 08:39 1HL_22A_R1.fq -rw-r--r-- 1 root root 658M Jul 20 08:39 1HL_22A_R2.fq -rw-r--r-- 1 root root 828M Jul 20 17:16 1HL_23A.assembled.fastq -rw-r--r-- 1 root root 442M Jul 20 17:16 1HL_23A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:14 1HL_23A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:14 1HL_23A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:40 1HL_23A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:40 1HL_23A_R2.fq -rw-r--r-- 1 root root 609M Jul 20 17:17 1HL_24A.assembled.fastq -rw-r--r-- 1 root root 344M Jul 20 17:17 1HL_24A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:16 1HL_24A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:16 1HL_24A.unassembled.reverse.fastq -rw-r--r-- 1 root root 791M Jul 20 08:41 1HL_24A_R1.fq -rw-r--r-- 1 root root 843M Jul 20 08:41 1HL_24A_R2.fq -rw-r--r-- 1 root root 688M Jul 20 17:19 1HL_25A.assembled.fastq -rw-r--r-- 1 root root 362M Jul 20 17:19 1HL_25A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:17 1HL_25A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:17 1HL_25A.unassembled.reverse.fastq -rw-r--r-- 1 root root 878M Jul 20 08:42 1HL_25A_R1.fq -rw-r--r-- 1 root root 936M Jul 20 08:42 1HL_25A_R2.fq -rw-r--r-- 1 root root 762M Jul 20 17:21 1HL_26A.assembled.fastq -rw-r--r-- 1 root root 381M Jul 20 17:21 1HL_26A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:19 1HL_26A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:19 1HL_26A.unassembled.reverse.fastq -rw-r--r-- 1 root root 964M Jul 20 08:42 1HL_26A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:43 1HL_26A_R2.fq -rw-r--r-- 1 root root 759M Jul 20 17:23 1HL_27A.assembled.fastq -rw-r--r-- 1 root root 384M Jul 20 17:23 1HL_27A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:21 1HL_27A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:21 1HL_27A.unassembled.reverse.fastq -rw-r--r-- 1 root root 960M Jul 20 08:43 1HL_27A_R1.fq -rw-r--r-- 1 root root 1023M Jul 20 08:44 1HL_27A_R2.fq -rw-r--r-- 1 root root 847M Jul 20 17:25 1HL_28A.assembled.fastq -rw-r--r-- 1 root root 472M Jul 20 17:25 1HL_28A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:23 1HL_28A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:23 1HL_28A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:44 1HL_28A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 20 08:45 1HL_28A_R2.fq -rw-r--r-- 1 root root 776M Jul 20 17:27 1HL_29A.assembled.fastq -rw-r--r-- 1 root root 376M Jul 20 17:27 1HL_29A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:25 1HL_29A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:25 1HL_29A.unassembled.reverse.fastq -rw-r--r-- 1 root root 981M Jul 20 08:45 1HL_29A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:46 1HL_29A_R2.fq -rw-r--r-- 1 root root 664M Jul 20 17:29 1HL_2A.assembled.fastq -rw-r--r-- 1 root root 325M Jul 20 17:29 1HL_2A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:27 1HL_2A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:27 1HL_2A.unassembled.reverse.fastq -rw-r--r-- 1 root root 838M Jul 20 08:24 1HL_2A_R1.fq -rw-r--r-- 1 root root 886M Jul 20 08:25 1HL_2A_R2.fq -rw-r--r-- 1 root root 579M Jul 20 17:31 1HL_31A.assembled.fastq -rw-r--r-- 1 root root 297M Jul 20 17:31 1HL_31A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:29 1HL_31A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:29 1HL_31A.unassembled.reverse.fastq -rw-r--r-- 1 root root 736M Jul 20 08:46 1HL_31A_R1.fq -rw-r--r-- 1 root root 784M Jul 20 08:46 1HL_31A_R2.fq -rw-r--r-- 1 root root 624M Jul 20 17:32 1HL_33A.assembled.fastq -rw-r--r-- 1 root root 333M Jul 20 17:32 1HL_33A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:31 1HL_33A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:31 1HL_33A.unassembled.reverse.fastq -rw-r--r-- 1 root root 799M Jul 20 08:47 1HL_33A_R1.fq -rw-r--r-- 1 root root 851M Jul 20 08:47 1HL_33A_R2.fq -rw-r--r-- 1 root root 654M Jul 20 17:34 1HL_34A.assembled.fastq -rw-r--r-- 1 root root 360M Jul 20 17:34 1HL_34A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:32 1HL_34A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:32 1HL_34A.unassembled.reverse.fastq -rw-r--r-- 1 root root 843M Jul 20 08:47 1HL_34A_R1.fq -rw-r--r-- 1 root root 899M Jul 20 08:48 1HL_34A_R2.fq -rw-r--r-- 1 root root 591M Jul 20 17:36 1HL_35A.assembled.fastq -rw-r--r-- 1 root root 328M Jul 20 17:36 1HL_35A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:34 1HL_35A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:34 1HL_35A.unassembled.reverse.fastq -rw-r--r-- 1 root root 764M Jul 20 08:48 1HL_35A_R1.fq -rw-r--r-- 1 root root 814M Jul 20 08:49 1HL_35A_R2.fq -rw-r--r-- 1 root root 525M Jul 20 17:37 1HL_3A.assembled.fastq -rw-r--r-- 1 root root 272M Jul 20 17:37 1HL_3A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:36 1HL_3A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:36 1HL_3A.unassembled.reverse.fastq -rw-r--r-- 1 root root 674M Jul 20 08:25 1HL_3A_R1.fq -rw-r--r-- 1 root root 712M Jul 20 08:25 1HL_3A_R2.fq -rw-r--r-- 1 root root 648M Jul 20 17:39 1HL_4A.assembled.fastq -rw-r--r-- 1 root root 310M Jul 20 17:39 1HL_4A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:37 1HL_4A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:37 1HL_4A.unassembled.reverse.fastq -rw-r--r-- 1 root root 819M Jul 20 08:26 1HL_4A_R1.fq -rw-r--r-- 1 root root 866M Jul 20 08:26 1HL_4A_R2.fq -rw-r--r-- 1 root root 497M Jul 20 17:40 1HL_5A.assembled.fastq -rw-r--r-- 1 root root 246M Jul 20 17:40 1HL_5A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:39 1HL_5A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:39 1HL_5A.unassembled.reverse.fastq -rw-r--r-- 1 root root 635M Jul 20 08:26 1HL_5A_R1.fq -rw-r--r-- 1 root root 671M Jul 20 08:27 1HL_5A_R2.fq -rw-r--r-- 1 root root 473M Jul 20 17:42 1HL_6A.assembled.fastq -rw-r--r-- 1 root root 269M Jul 20 17:42 1HL_6A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:40 1HL_6A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:40 1HL_6A.unassembled.reverse.fastq -rw-r--r-- 1 root root 621M Jul 20 08:27 1HL_6A_R1.fq -rw-r--r-- 1 root root 657M Jul 20 08:27 1HL_6A_R2.fq -rw-r--r-- 1 root root 671M Jul 20 17:44 1HL_7A.assembled.fastq -rw-r--r-- 1 root root 346M Jul 20 17:44 1HL_7A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:42 1HL_7A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:42 1HL_7A.unassembled.reverse.fastq -rw-r--r-- 1 root root 861M Jul 20 08:28 1HL_7A_R1.fq -rw-r--r-- 1 root root 910M Jul 20 08:28 1HL_7A_R2.fq -rw-r--r-- 1 root root 610M Jul 20 17:45 1HL_8A.assembled.fastq -rw-r--r-- 1 root root 327M Jul 20 17:45 1HL_8A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:44 1HL_8A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:44 1HL_8A.unassembled.reverse.fastq -rw-r--r-- 1 root root 794M Jul 20 08:28 1HL_8A_R1.fq -rw-r--r-- 1 root root 840M Jul 20 08:29 1HL_8A_R2.fq -rw-r--r-- 1 root root 687M Jul 20 17:47 1HL_9A.assembled.fastq -rw-r--r-- 1 root root 345M Jul 20 17:47 1HL_9A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:45 1HL_9A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:45 1HL_9A.unassembled.reverse.fastq -rw-r--r-- 1 root root 876M Jul 20 08:29 1HL_9A_R1.fq -rw-r--r-- 1 root root 926M Jul 20 08:30 1HL_9A_R2.fq -rw-r--r-- 1 root root 518M Jul 20 17:49 1NF_10A.assembled.fastq -rw-r--r-- 1 root root 272M Jul 20 17:49 1NF_10A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:47 1NF_10A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:47 1NF_10A.unassembled.reverse.fastq -rw-r--r-- 1 root root 681M Jul 20 07:39 1NF_10A_R1.fq -rw-r--r-- 1 root root 703M Jul 20 07:40 1NF_10A_R2.fq -rw-r--r-- 1 root root 535M Jul 20 17:50 1NF_11A.assembled.fastq -rw-r--r-- 1 root root 285M Jul 20 17:50 1NF_11A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:49 1NF_11A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:49 1NF_11A.unassembled.reverse.fastq -rw-r--r-- 1 root root 703M Jul 20 07:40 1NF_11A_R1.fq -rw-r--r-- 1 root root 725M Jul 20 07:40 1NF_11A_R2.fq -rw-r--r-- 1 root root 540M Jul 20 17:52 1NF_12A.assembled.fastq -rw-r--r-- 1 root root 268M Jul 20 17:52 1NF_12A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:50 1NF_12A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:50 1NF_12A.unassembled.reverse.fastq -rw-r--r-- 1 root root 701M Jul 20 07:41 1NF_12A_R1.fq -rw-r--r-- 1 root root 723M Jul 20 07:41 1NF_12A_R2.fq -rw-r--r-- 1 root root 970M Jul 20 17:54 1NF_13A.assembled.fastq -rw-r--r-- 1 root root 460M Jul 20 17:54 1NF_13A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:52 1NF_13A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:52 1NF_13A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.2G Jul 20 07:41 1NF_13A_R1.fq -rw-r--r-- 1 root root 1.3G Jul 20 07:42 1NF_13A_R2.fq -rw-r--r-- 1 root root 625M Jul 20 17:56 1NF_14A.assembled.fastq -rw-r--r-- 1 root root 345M Jul 20 17:56 1NF_14A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:54 1NF_14A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:54 1NF_14A.unassembled.reverse.fastq -rw-r--r-- 1 root root 822M Jul 20 07:42 1NF_14A_R1.fq -rw-r--r-- 1 root root 855M Jul 20 07:42 1NF_14A_R2.fq -rw-r--r-- 1 root root 610M Jul 20 17:58 1NF_15A.assembled.fastq -rw-r--r-- 1 root root 319M Jul 20 17:58 1NF_15A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:56 1NF_15A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:56 1NF_15A.unassembled.reverse.fastq -rw-r--r-- 1 root root 793M Jul 20 07:43 1NF_15A_R1.fq -rw-r--r-- 1 root root 825M Jul 20 07:43 1NF_15A_R2.fq -rw-r--r-- 1 root root 700M Jul 20 18:00 1NF_16A.assembled.fastq -rw-r--r-- 1 root root 339M Jul 20 18:00 1NF_16A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 17:58 1NF_16A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 17:58 1NF_16A.unassembled.reverse.fastq -rw-r--r-- 1 root root 904M Jul 20 07:44 1NF_16A_R1.fq -rw-r--r-- 1 root root 941M Jul 20 07:44 1NF_16A_R2.fq -rw-r--r-- 1 root root 752M Jul 20 18:02 1NF_17A.assembled.fastq -rw-r--r-- 1 root root 388M Jul 20 18:02 1NF_17A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:00 1NF_17A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:00 1NF_17A.unassembled.reverse.fastq -rw-r--r-- 1 root root 977M Jul 20 07:44 1NF_17A_R1.fq -rw-r--r-- 1 root root 1017M Jul 20 07:45 1NF_17A_R2.fq -rw-r--r-- 1 root root 396M Jul 20 18:03 1NF_18A.assembled.fastq -rw-r--r-- 1 root root 221M Jul 20 18:03 1NF_18A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:02 1NF_18A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:02 1NF_18A.unassembled.reverse.fastq -rw-r--r-- 1 root root 521M Jul 20 07:45 1NF_18A_R1.fq -rw-r--r-- 1 root root 542M Jul 20 07:45 1NF_18A_R2.fq -rw-r--r-- 1 root root 472M Jul 20 18:04 1NF_19A.assembled.fastq -rw-r--r-- 1 root root 265M Jul 20 18:04 1NF_19A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:03 1NF_19A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:03 1NF_19A.unassembled.reverse.fastq -rw-r--r-- 1 root root 627M Jul 20 07:46 1NF_19A_R1.fq -rw-r--r-- 1 root root 653M Jul 20 07:46 1NF_19A_R2.fq -rw-r--r-- 1 root root 310M Jul 20 18:05 1NF_1A.assembled.fastq -rw-r--r-- 1 root root 179M Jul 20 18:05 1NF_1A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:04 1NF_1A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:04 1NF_1A.unassembled.reverse.fastq -rw-r--r-- 1 root root 417M Jul 20 07:34 1NF_1A_R1.fq -rw-r--r-- 1 root root 430M Jul 20 07:34 1NF_1A_R2.fq -rw-r--r-- 1 root root 596M Jul 20 18:07 1NF_20A.assembled.fastq -rw-r--r-- 1 root root 308M Jul 20 18:07 1NF_20A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:05 1NF_20A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:05 1NF_20A.unassembled.reverse.fastq -rw-r--r-- 1 root root 776M Jul 20 07:46 1NF_20A_R1.fq -rw-r--r-- 1 root root 808M Jul 20 07:47 1NF_20A_R2.fq -rw-r--r-- 1 root root 764M Jul 20 18:09 1NF_21A.assembled.fastq -rw-r--r-- 1 root root 357M Jul 20 18:09 1NF_21A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:07 1NF_21A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:07 1NF_21A.unassembled.reverse.fastq -rw-r--r-- 1 root root 966M Jul 20 07:47 1NF_21A_R1.fq -rw-r--r-- 1 root root 1005M Jul 20 07:47 1NF_21A_R2.fq -rw-r--r-- 1 root root 706M Jul 20 18:11 1NF_22A.assembled.fastq -rw-r--r-- 1 root root 351M Jul 20 18:11 1NF_22A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:09 1NF_22A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:09 1NF_22A.unassembled.reverse.fastq -rw-r--r-- 1 root root 908M Jul 20 07:48 1NF_22A_R1.fq -rw-r--r-- 1 root root 944M Jul 20 07:48 1NF_22A_R2.fq -rw-r--r-- 1 root root 813M Jul 20 18:13 1NF_23A.assembled.fastq -rw-r--r-- 1 root root 386M Jul 20 18:13 1NF_23A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:11 1NF_23A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:11 1NF_23A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 07:48 1NF_23A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 07:49 1NF_23A_R2.fq -rw-r--r-- 1 root root 709M Jul 20 18:15 1NF_24A.assembled.fastq -rw-r--r-- 1 root root 376M Jul 20 18:15 1NF_24A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:13 1NF_24A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:13 1NF_24A.unassembled.reverse.fastq -rw-r--r-- 1 root root 922M Jul 20 07:49 1NF_24A_R1.fq -rw-r--r-- 1 root root 959M Jul 20 07:50 1NF_24A_R2.fq -rw-r--r-- 1 root root 503M Jul 20 18:17 1NF_25A.assembled.fastq -rw-r--r-- 1 root root 283M Jul 20 18:17 1NF_25A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:15 1NF_25A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:15 1NF_25A.unassembled.reverse.fastq -rw-r--r-- 1 root root 666M Jul 20 07:50 1NF_25A_R1.fq -rw-r--r-- 1 root root 693M Jul 20 07:50 1NF_25A_R2.fq -rw-r--r-- 1 root root 695M Jul 20 18:19 1NF_26A.assembled.fastq -rw-r--r-- 1 root root 360M Jul 20 18:19 1NF_26A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:17 1NF_26A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:17 1NF_26A.unassembled.reverse.fastq -rw-r--r-- 1 root root 900M Jul 20 07:51 1NF_26A_R1.fq -rw-r--r-- 1 root root 937M Jul 20 07:51 1NF_26A_R2.fq -rw-r--r-- 1 root root 567M Jul 20 18:20 1NF_27A.assembled.fastq -rw-r--r-- 1 root root 284M Jul 20 18:20 1NF_27A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:19 1NF_27A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:19 1NF_27A.unassembled.reverse.fastq -rw-r--r-- 1 root root 733M Jul 20 07:51 1NF_27A_R1.fq -rw-r--r-- 1 root root 762M Jul 20 07:52 1NF_27A_R2.fq -rw-r--r-- 1 root root 583M Jul 20 18:22 1NF_28A.assembled.fastq -rw-r--r-- 1 root root 297M Jul 20 18:22 1NF_28A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:20 1NF_28A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:20 1NF_28A.unassembled.reverse.fastq -rw-r--r-- 1 root root 758M Jul 20 07:52 1NF_28A_R1.fq -rw-r--r-- 1 root root 788M Jul 20 07:52 1NF_28A_R2.fq -rw-r--r-- 1 root root 929M Jul 20 18:24 1NF_29A.assembled.fastq -rw-r--r-- 1 root root 485M Jul 20 18:24 1NF_29A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:22 1NF_29A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:22 1NF_29A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.2G Jul 20 07:53 1NF_29A_R1.fq -rw-r--r-- 1 root root 1.3G Jul 20 07:53 1NF_29A_R2.fq -rw-r--r-- 1 root root 834M Jul 20 18:26 1NF_2A.assembled.fastq -rw-r--r-- 1 root root 425M Jul 20 18:26 1NF_2A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:24 1NF_2A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:24 1NF_2A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 07:34 1NF_2A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 07:34 1NF_2A_R2.fq -rw-r--r-- 1 root root 725M Jul 20 18:28 1NF_30A.assembled.fastq -rw-r--r-- 1 root root 335M Jul 20 18:28 1NF_30A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:26 1NF_30A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:26 1NF_30A.unassembled.reverse.fastq -rw-r--r-- 1 root root 930M Jul 20 07:54 1NF_30A_R1.fq -rw-r--r-- 1 root root 967M Jul 20 07:54 1NF_30A_R2.fq -rw-r--r-- 1 root root 848M Jul 20 18:30 1NF_31A.assembled.fastq -rw-r--r-- 1 root root 439M Jul 20 18:30 1NF_31A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:28 1NF_31A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:28 1NF_31A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 07:55 1NF_31A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 20 07:55 1NF_31A_R2.fq -rw-r--r-- 1 root root 860M Jul 20 18:33 1NF_32A.assembled.fastq -rw-r--r-- 1 root root 420M Jul 20 18:33 1NF_32A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:30 1NF_32A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:30 1NF_32A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 07:56 1NF_32A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 20 07:56 1NF_32A_R2.fq -rw-r--r-- 1 root root 649M Jul 20 18:34 1NF_33A.assembled.fastq -rw-r--r-- 1 root root 321M Jul 20 18:34 1NF_33A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:33 1NF_33A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:33 1NF_33A.unassembled.reverse.fastq -rw-r--r-- 1 root root 837M Jul 20 07:56 1NF_33A_R1.fq -rw-r--r-- 1 root root 870M Jul 20 07:57 1NF_33A_R2.fq -rw-r--r-- 1 root root 868M Jul 20 18:37 1NF_4A.assembled.fastq -rw-r--r-- 1 root root 438M Jul 20 18:37 1NF_4A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:34 1NF_4A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:34 1NF_4A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 07:35 1NF_4A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 20 07:35 1NF_4A_R2.fq -rw-r--r-- 1 root root 651M Jul 20 18:38 1NF_5A.assembled.fastq -rw-r--r-- 1 root root 353M Jul 20 18:38 1NF_5A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:37 1NF_5A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:37 1NF_5A.unassembled.reverse.fastq -rw-r--r-- 1 root root 860M Jul 20 07:35 1NF_5A_R1.fq -rw-r--r-- 1 root root 887M Jul 20 07:36 1NF_5A_R2.fq -rw-r--r-- 1 root root 590M Jul 20 18:40 1NF_6A.assembled.fastq -rw-r--r-- 1 root root 301M Jul 20 18:40 1NF_6A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:38 1NF_6A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:38 1NF_6A.unassembled.reverse.fastq -rw-r--r-- 1 root root 767M Jul 20 07:36 1NF_6A_R1.fq -rw-r--r-- 1 root root 791M Jul 20 07:37 1NF_6A_R2.fq -rw-r--r-- 1 root root 642M Jul 20 18:42 1NF_7A.assembled.fastq -rw-r--r-- 1 root root 319M Jul 20 18:42 1NF_7A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:40 1NF_7A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:40 1NF_7A.unassembled.reverse.fastq -rw-r--r-- 1 root root 831M Jul 20 07:37 1NF_7A_R1.fq -rw-r--r-- 1 root root 858M Jul 20 07:37 1NF_7A_R2.fq -rw-r--r-- 1 root root 651M Jul 20 18:44 1NF_8A.assembled.fastq -rw-r--r-- 1 root root 351M Jul 20 18:44 1NF_8A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:42 1NF_8A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:42 1NF_8A.unassembled.reverse.fastq -rw-r--r-- 1 root root 858M Jul 20 07:38 1NF_8A_R1.fq -rw-r--r-- 1 root root 886M Jul 20 07:38 1NF_8A_R2.fq -rw-r--r-- 1 root root 802M Jul 20 18:46 1NF_9A.assembled.fastq -rw-r--r-- 1 root root 419M Jul 20 18:46 1NF_9A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:44 1NF_9A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:44 1NF_9A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 07:39 1NF_9A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 07:39 1NF_9A_R2.fq -rw-r--r-- 1 root root 631M Jul 20 18:47 1SN_10A.assembled.fastq -rw-r--r-- 1 root root 308M Jul 20 18:47 1SN_10A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:46 1SN_10A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:46 1SN_10A.unassembled.reverse.fastq -rw-r--r-- 1 root root 804M Jul 20 08:05 1SN_10A_R1.fq -rw-r--r-- 1 root root 843M Jul 20 08:05 1SN_10A_R2.fq -rw-r--r-- 1 root root 647M Jul 20 18:49 1SN_11A.assembled.fastq -rw-r--r-- 1 root root 308M Jul 20 18:49 1SN_11A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:47 1SN_11A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:47 1SN_11A.unassembled.reverse.fastq -rw-r--r-- 1 root root 823M Jul 20 08:06 1SN_11A_R1.fq -rw-r--r-- 1 root root 863M Jul 20 08:06 1SN_11A_R2.fq -rw-r--r-- 1 root root 809M Jul 20 18:51 1SN_12A.assembled.fastq -rw-r--r-- 1 root root 412M Jul 20 18:51 1SN_12A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:49 1SN_12A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:49 1SN_12A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:06 1SN_12A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:07 1SN_12A_R2.fq -rw-r--r-- 1 root root 706M Jul 20 18:53 1SN_13A.assembled.fastq -rw-r--r-- 1 root root 361M Jul 20 18:53 1SN_13A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:51 1SN_13A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:51 1SN_13A.unassembled.reverse.fastq -rw-r--r-- 1 root root 906M Jul 20 08:08 1SN_13A_R1.fq -rw-r--r-- 1 root root 950M Jul 20 08:08 1SN_13A_R2.fq -rw-r--r-- 1 root root 850M Jul 20 18:55 1SN_14A.assembled.fastq -rw-r--r-- 1 root root 429M Jul 20 18:55 1SN_14A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:53 1SN_14A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:53 1SN_14A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:08 1SN_14A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 20 08:09 1SN_14A_R2.fq -rw-r--r-- 1 root root 572M Jul 20 18:57 1SN_15A.assembled.fastq -rw-r--r-- 1 root root 311M Jul 20 18:57 1SN_15A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:55 1SN_15A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:55 1SN_15A.unassembled.reverse.fastq -rw-r--r-- 1 root root 743M Jul 20 08:09 1SN_15A_R1.fq -rw-r--r-- 1 root root 780M Jul 20 08:10 1SN_15A_R2.fq -rw-r--r-- 1 root root 615M Jul 20 18:58 1SN_16A.assembled.fastq -rw-r--r-- 1 root root 325M Jul 20 18:58 1SN_16A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:57 1SN_16A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:57 1SN_16A.unassembled.reverse.fastq -rw-r--r-- 1 root root 797M Jul 20 08:10 1SN_16A_R1.fq -rw-r--r-- 1 root root 835M Jul 20 08:10 1SN_16A_R2.fq -rw-r--r-- 1 root root 688M Jul 20 19:00 1SN_17A.assembled.fastq -rw-r--r-- 1 root root 371M Jul 20 19:00 1SN_17A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 18:58 1SN_17A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 18:58 1SN_17A.unassembled.reverse.fastq -rw-r--r-- 1 root root 893M Jul 20 08:11 1SN_17A_R1.fq -rw-r--r-- 1 root root 937M Jul 20 08:11 1SN_17A_R2.fq -rw-r--r-- 1 root root 782M Jul 20 19:02 1SN_18A.assembled.fastq -rw-r--r-- 1 root root 391M Jul 20 19:02 1SN_18A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:00 1SN_18A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:00 1SN_18A.unassembled.reverse.fastq -rw-r--r-- 1 root root 998M Jul 20 08:12 1SN_18A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:12 1SN_18A_R2.fq -rw-r--r-- 1 root root 730M Jul 20 19:04 1SN_19A.assembled.fastq -rw-r--r-- 1 root root 367M Jul 20 19:04 1SN_19A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:02 1SN_19A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:02 1SN_19A.unassembled.reverse.fastq -rw-r--r-- 1 root root 934M Jul 20 08:12 1SN_19A_R1.fq -rw-r--r-- 1 root root 980M Jul 20 08:13 1SN_19A_R2.fq -rw-r--r-- 1 root root 712M Jul 20 19:06 1SN_1A.assembled.fastq -rw-r--r-- 1 root root 383M Jul 20 19:06 1SN_1A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:04 1SN_1A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:04 1SN_1A.unassembled.reverse.fastq -rw-r--r-- 1 root root 931M Jul 20 07:57 1SN_1A_R1.fq -rw-r--r-- 1 root root 968M Jul 20 07:58 1SN_1A_R2.fq -rw-r--r-- 1 root root 555M Jul 20 19:07 1SN_20A.assembled.fastq -rw-r--r-- 1 root root 261M Jul 20 19:07 1SN_20A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:06 1SN_20A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:06 1SN_20A.unassembled.reverse.fastq -rw-r--r-- 1 root root 706M Jul 20 08:13 1SN_20A_R1.fq -rw-r--r-- 1 root root 740M Jul 20 08:14 1SN_20A_R2.fq -rw-r--r-- 1 root root 819M Jul 20 19:10 1SN_21A.assembled.fastq -rw-r--r-- 1 root root 435M Jul 20 19:10 1SN_21A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:07 1SN_21A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:07 1SN_21A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:14 1SN_21A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:14 1SN_21A_R2.fq -rw-r--r-- 1 root root 746M Jul 20 19:11 1SN_22A.assembled.fastq -rw-r--r-- 1 root root 367M Jul 20 19:11 1SN_22A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:10 1SN_22A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:10 1SN_22A.unassembled.reverse.fastq -rw-r--r-- 1 root root 950M Jul 20 08:15 1SN_22A_R1.fq -rw-r--r-- 1 root root 996M Jul 20 08:15 1SN_22A_R2.fq -rw-r--r-- 1 root root 877M Jul 20 19:14 1SN_23A.assembled.fastq -rw-r--r-- 1 root root 439M Jul 20 19:14 1SN_23A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:11 1SN_23A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:11 1SN_23A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:16 1SN_23A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 20 08:16 1SN_23A_R2.fq -rw-r--r-- 1 root root 432M Jul 20 19:15 1SN_24A.assembled.fastq -rw-r--r-- 1 root root 241M Jul 20 19:15 1SN_24A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:14 1SN_24A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:14 1SN_24A.unassembled.reverse.fastq -rw-r--r-- 1 root root 566M Jul 20 08:17 1SN_24A_R1.fq -rw-r--r-- 1 root root 599M Jul 20 08:17 1SN_24A_R2.fq -rw-r--r-- 1 root root 690M Jul 20 19:17 1SN_25A.assembled.fastq -rw-r--r-- 1 root root 380M Jul 20 19:17 1SN_25A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:15 1SN_25A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:15 1SN_25A.unassembled.reverse.fastq -rw-r--r-- 1 root root 896M Jul 20 08:17 1SN_25A_R1.fq -rw-r--r-- 1 root root 947M Jul 20 08:18 1SN_25A_R2.fq -rw-r--r-- 1 root root 638M Jul 20 19:18 1SN_26A.assembled.fastq -rw-r--r-- 1 root root 339M Jul 20 19:18 1SN_26A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:17 1SN_26A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:17 1SN_26A.unassembled.reverse.fastq -rw-r--r-- 1 root root 821M Jul 20 08:18 1SN_26A_R1.fq -rw-r--r-- 1 root root 868M Jul 20 08:19 1SN_26A_R2.fq -rw-r--r-- 1 root root 598M Jul 20 19:20 1SN_27A.assembled.fastq -rw-r--r-- 1 root root 332M Jul 20 19:20 1SN_27A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:18 1SN_27A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:18 1SN_27A.unassembled.reverse.fastq -rw-r--r-- 1 root root 781M Jul 20 08:19 1SN_27A_R1.fq -rw-r--r-- 1 root root 825M Jul 20 08:19 1SN_27A_R2.fq -rw-r--r-- 1 root root 662M Jul 20 19:22 1SN_28A.assembled.fastq -rw-r--r-- 1 root root 352M Jul 20 19:22 1SN_28A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:20 1SN_28A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:20 1SN_28A.unassembled.reverse.fastq -rw-r--r-- 1 root root 854M Jul 20 08:20 1SN_28A_R1.fq -rw-r--r-- 1 root root 903M Jul 20 08:20 1SN_28A_R2.fq -rw-r--r-- 1 root root 505M Jul 20 19:23 1SN_29A.assembled.fastq -rw-r--r-- 1 root root 301M Jul 20 19:23 1SN_29A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:22 1SN_29A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:22 1SN_29A.unassembled.reverse.fastq -rw-r--r-- 1 root root 674M Jul 20 08:20 1SN_29A_R1.fq -rw-r--r-- 1 root root 713M Jul 20 08:21 1SN_29A_R2.fq -rw-r--r-- 1 root root 738M Jul 20 19:25 1SN_2A.assembled.fastq -rw-r--r-- 1 root root 375M Jul 20 19:25 1SN_2A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:23 1SN_2A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:23 1SN_2A.unassembled.reverse.fastq -rw-r--r-- 1 root root 948M Jul 20 07:58 1SN_2A_R1.fq -rw-r--r-- 1 root root 994M Jul 20 07:59 1SN_2A_R2.fq -rw-r--r-- 1 root root 917M Jul 20 19:28 1SN_30A.assembled.fastq -rw-r--r-- 1 root root 452M Jul 20 19:28 1SN_30A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:25 1SN_30A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:25 1SN_30A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.2G Jul 20 08:21 1SN_30A_R1.fq -rw-r--r-- 1 root root 1.2G Jul 20 08:22 1SN_30A_R2.fq -rw-r--r-- 1 root root 790M Jul 20 19:30 1SN_31A.assembled.fastq -rw-r--r-- 1 root root 409M Jul 20 19:30 1SN_31A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:28 1SN_31A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:28 1SN_31A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1012M Jul 20 08:22 1SN_31A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:23 1SN_31A_R2.fq -rw-r--r-- 1 root root 596M Jul 20 19:31 1SN_32A.assembled.fastq -rw-r--r-- 1 root root 317M Jul 20 19:31 1SN_32A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:30 1SN_32A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:30 1SN_32A.unassembled.reverse.fastq -rw-r--r-- 1 root root 770M Jul 20 08:23 1SN_32A_R1.fq -rw-r--r-- 1 root root 814M Jul 20 08:23 1SN_32A_R2.fq -rw-r--r-- 1 root root 557M Jul 20 19:33 1SN_3A.assembled.fastq -rw-r--r-- 1 root root 312M Jul 20 19:33 1SN_3A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:31 1SN_3A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:31 1SN_3A.unassembled.reverse.fastq -rw-r--r-- 1 root root 730M Jul 20 07:59 1SN_3A_R1.fq -rw-r--r-- 1 root root 765M Jul 20 07:59 1SN_3A_R2.fq -rw-r--r-- 1 root root 790M Jul 20 19:35 1SN_4A.assembled.fastq -rw-r--r-- 1 root root 388M Jul 20 19:35 1SN_4A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:33 1SN_4A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:33 1SN_4A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1003M Jul 20 08:00 1SN_4A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:00 1SN_4A_R2.fq -rw-r--r-- 1 root root 716M Jul 20 19:37 1SN_5A.assembled.fastq -rw-r--r-- 1 root root 359M Jul 20 19:37 1SN_5A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:35 1SN_5A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:35 1SN_5A.unassembled.reverse.fastq -rw-r--r-- 1 root root 912M Jul 20 08:01 1SN_5A_R1.fq -rw-r--r-- 1 root root 957M Jul 20 08:01 1SN_5A_R2.fq -rw-r--r-- 1 root root 806M Jul 20 19:39 1SN_6A.assembled.fastq -rw-r--r-- 1 root root 402M Jul 20 19:39 1SN_6A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:37 1SN_6A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:37 1SN_6A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:01 1SN_6A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:02 1SN_6A_R2.fq -rw-r--r-- 1 root root 717M Jul 20 19:41 1SN_7A.assembled.fastq -rw-r--r-- 1 root root 358M Jul 20 19:41 1SN_7A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:39 1SN_7A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:39 1SN_7A.unassembled.reverse.fastq -rw-r--r-- 1 root root 915M Jul 20 08:02 1SN_7A_R1.fq -rw-r--r-- 1 root root 959M Jul 20 08:03 1SN_7A_R2.fq -rw-r--r-- 1 root root 813M Jul 20 19:43 1SN_8A.assembled.fastq -rw-r--r-- 1 root root 398M Jul 20 19:43 1SN_8A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:41 1SN_8A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:41 1SN_8A.unassembled.reverse.fastq -rw-r--r-- 1 root root 1.1G Jul 20 08:03 1SN_8A_R1.fq -rw-r--r-- 1 root root 1.1G Jul 20 08:04 1SN_8A_R2.fq -rw-r--r-- 1 root root 767M Jul 20 19:45 1SN_9A.assembled.fastq -rw-r--r-- 1 root root 379M Jul 20 19:45 1SN_9A.discarded.fastq -rw-r--r-- 1 root root 0 Jul 20 19:43 1SN_9A.unassembled.forward.fastq -rw-r--r-- 1 root root 0 Jul 20 19:43 1SN_9A.unassembled.reverse.fastq -rw-r--r-- 1 root root 976M Jul 20 08:04 1SN_9A_R1.fq -rw-r--r-- 1 root root 1023M Jul 20 08:04 1SN_9A_R2.fq
Set the data location to the de-multiplexed & merged data¶
In [30]:
%%bash
## set location of demultiplexed data that are 'pear' filtered
sed -i '/## 18./c\/data/analysis/fastq/*.assembled.fastq ## 18. demulti data loc ' /data/analysis/params.txt
In [31]:
%%bash
cat /data/analysis/params.txt
==** parameter inputs for pyRAD version 3.0.66 **======================== affected step ==
/data/analysis/ ## 1. Working directory (all)
/data/analysis/ ## 2. Loc. of non-demultiplexed files (if not line 18) (s1)
/data/analysis/barcodes.txt ## 3. Loc. of barcode file (if not line 18) (s1)
/usr/local/bioinformatics/vsearch-1.11.1-linux-x86_64/bin/vsearch ## 4. command (or path) to call vsearch (or usearch) (s3,s6)
/usr/local/bioinformatics/muscle3.8.31_i86linux64 ## 5. command (or path) to call muscle (s3,s7)
CWGC ## 6. Restriction overhang (e.g., C|TGCAG -> TGCAG) (s1,s2)
8 ## 7. N processors
6 ## 8. Mindepth: min coverage for a cluster (s4,s5)
4 ## 9. NQual: max # sites with qual < 20 (or see line 20)(s2)
.88 ## 10. Wclust: clustering threshold as a decimal (s3,s6)
pairgbs ## 11. data type
4 ## 12. MinCov: min samples in a final locus (s7)
3 ## 13. MaxSH: max inds with shared hetero site (s7)
oly_gbs_pyrad ## 14. Prefix name for final output (no spaces) (s7)
==== optional params below this line =================================== affected step ==
## 15.opt.: select subset (prefix* only selector) (s2-s7)
## 16.opt.: add-on (outgroup) taxa (list or prefix*) (s6,s7)
## 17.opt.: exclude taxa (list or prefix*) (s7)
/data/analysis/fastq/*.assembled.fastq ## 18. demulti data loc
## 19.opt.: maxM: N mismatches in barcodes (def= 1) (s1)
## 20.opt.: phred Qscore offset (def= 33) (s2)
2 ## 21.opt.: filter: def=0=NQual 1=NQual+adapters. 2=strict (s2)
## 22.opt.: a priori E,H (def= 0.001,0.01, if not estimated) (s5)
4 ## 23.opt.: maxN: max Ns in a cons seq (def=5) (s5)
8 ## 24.opt.: maxH: max heterozyg. sites in cons seq (def=5) (s5)
## 25.opt.: ploidy: max alleles in cons seq (def=2;see docs) (s4,s5)
## 26.opt.: maxSNPs: (def=100). Paired (def=100,100) (s7)
## 27.opt.: maxIndels: within-clust,across-clust (def. 3,99) (s3,s7)
## 28.opt.: random number seed (def. 112233) (s3,s6,s7)
## 29.opt.: trim overhang left,right on final loci, def(0,0) (s7)
* ## 30.opt.: output formats: p,n,a,s,v,u,t,m,k,g,* (see docs) (s7)
## 31.opt.: maj. base call at depth>x<mindepth (def.x=mindepth) (s5)
50 ## 32.opt.: keep trimmed reads (def=0). Enter min length. (s2)
## 33.opt.: max stack size (int), def= max(500,mean+2*SD) (s3)
## 34.opt.: minDerep: exclude dereps with <= N copies, def=1 (s3)
## 35.opt.: use hierarchical clustering (def.=0, 1=yes) (s6)
## 36.opt.: repeat masking (def.=1='dust' method, 0=no) (s3,s6)
16 ## 37.opt.: vsearch max threads per job (def.=6; see docs) (s3,s6)
==== optional: list group/clade assignments below this line (see docs) ==================
1HL 5 1HL*
1NF 5 1NF*
1SN 5 1SN*
Set the datatype to "merged"¶
In [32]:
%%bash
sed -i '/## 11./c\merged ## 11. data type ' /data/analysis/params.txt
In [33]:
%%bash
cat /data/analysis/params.txt
==** parameter inputs for pyRAD version 3.0.66 **======================== affected step ==
/data/analysis/ ## 1. Working directory (all)
/data/analysis/ ## 2. Loc. of non-demultiplexed files (if not line 18) (s1)
/data/analysis/barcodes.txt ## 3. Loc. of barcode file (if not line 18) (s1)
/usr/local/bioinformatics/vsearch-1.11.1-linux-x86_64/bin/vsearch ## 4. command (or path) to call vsearch (or usearch) (s3,s6)
/usr/local/bioinformatics/muscle3.8.31_i86linux64 ## 5. command (or path) to call muscle (s3,s7)
CWGC ## 6. Restriction overhang (e.g., C|TGCAG -> TGCAG) (s1,s2)
8 ## 7. N processors
6 ## 8. Mindepth: min coverage for a cluster (s4,s5)
4 ## 9. NQual: max # sites with qual < 20 (or see line 20)(s2)
.88 ## 10. Wclust: clustering threshold as a decimal (s3,s6)
merged ## 11. data type
4 ## 12. MinCov: min samples in a final locus (s7)
3 ## 13. MaxSH: max inds with shared hetero site (s7)
oly_gbs_pyrad ## 14. Prefix name for final output (no spaces) (s7)
==== optional params below this line =================================== affected step ==
## 15.opt.: select subset (prefix* only selector) (s2-s7)
## 16.opt.: add-on (outgroup) taxa (list or prefix*) (s6,s7)
## 17.opt.: exclude taxa (list or prefix*) (s7)
/data/analysis/fastq/*.assembled.fastq ## 18. demulti data loc
## 19.opt.: maxM: N mismatches in barcodes (def= 1) (s1)
## 20.opt.: phred Qscore offset (def= 33) (s2)
2 ## 21.opt.: filter: def=0=NQual 1=NQual+adapters. 2=strict (s2)
## 22.opt.: a priori E,H (def= 0.001,0.01, if not estimated) (s5)
4 ## 23.opt.: maxN: max Ns in a cons seq (def=5) (s5)
8 ## 24.opt.: maxH: max heterozyg. sites in cons seq (def=5) (s5)
## 25.opt.: ploidy: max alleles in cons seq (def=2;see docs) (s4,s5)
## 26.opt.: maxSNPs: (def=100). Paired (def=100,100) (s7)
## 27.opt.: maxIndels: within-clust,across-clust (def. 3,99) (s3,s7)
## 28.opt.: random number seed (def. 112233) (s3,s6,s7)
## 29.opt.: trim overhang left,right on final loci, def(0,0) (s7)
* ## 30.opt.: output formats: p,n,a,s,v,u,t,m,k,g,* (see docs) (s7)
## 31.opt.: maj. base call at depth>x<mindepth (def.x=mindepth) (s5)
50 ## 32.opt.: keep trimmed reads (def=0). Enter min length. (s2)
## 33.opt.: max stack size (int), def= max(500,mean+2*SD) (s3)
## 34.opt.: minDerep: exclude dereps with <= N copies, def=1 (s3)
## 35.opt.: use hierarchical clustering (def.=0, 1=yes) (s6)
## 36.opt.: repeat masking (def.=1='dust' method, 0=no) (s3,s6)
16 ## 37.opt.: vsearch max threads per job (def.=6; see docs) (s3,s6)
==== optional: list group/clade assignments below this line (see docs) ==================
1HL 5 1HL*
1NF 5 1NF*
1SN 5 1SN*
Step 2: Quality Filtering¶
In [34]:
%%bash
time pyrad -p /data/analysis/params.txt -s 2
/data/analysis/edits/1HL_17A.assembled already in edits/ /data/analysis/edits/1HL_11A.assembled already in edits/ /data/analysis/edits/1HL_10A.assembled already in edits/ /data/analysis/edits/1HL_13A.assembled already in edits/ /data/analysis/edits/1HL_12A.assembled already in edits/ /data/analysis/edits/1HL_16A.assembled already in edits/ /data/analysis/edits/1HL_15A.assembled already in edits/ /data/analysis/edits/1HL_14A.assembled already in edits/
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
sorted .fastq from /data/analysis/fastq/*.assembled.fastq being used
step 2: editing raw reads
........................................................................................
real 70m16.887s
user 453m14.412s
sys 1m16.616s
Let's take a look at the results¶
In [35]:
%%bash
cat /data/analysis/stats/s2.rawedit.txt
sample Nreads passed passed.w.trim passed.total
1HL_10A.assembled 2890999 2127357 214294 2341651
1HL_11A.assembled 3088425 2242089 246561 2488650
1HL_12A.assembled 2323231 1728934 163227 1892161
1HL_13A.assembled 2701305 2075000 175929 2250929
1HL_14A.assembled 1865526 1407146 129089 1536235
1HL_15A.assembled 1837684 1342114 139898 1482012
1HL_16A.assembled 2100838 1568118 147901 1716019
1HL_17A.assembled 3106139 2259592 237499 2497091
Nreads = total number of reads for a sample
passed = retained reads that passed quality filtering at full length
passed.w.trim= retained reads that were trimmed due to detection of adapters
passed.total = total kept reads of sufficient length
note: you can set the option in params file to include trimmed reads of xx length.
sample Nreads passed passed.w.trim passed.total
1HL_19A.assembled 3174884 2334130 238750 2572880
1HL_1A.assembled 2098741 1574224 145441 1719665
1HL_20A.assembled 2516582 1852015 188703 2040718
1HL_21A.assembled 2203056 1616145 167632 1783777
1HL_22A.assembled 1807508 1343661 129836 1473497
1HL_23A.assembled 3117475 2274454 240445 2514899
1HL_24A.assembled 2290319 1671535 175429 1846964
1HL_25A.assembled 2606955 1914854 193248 2108102
1HL_26A.assembled 2915184 2177424 206507 2383931
1HL_27A.assembled 2887147 2121325 216634 2337959
1HL_28A.assembled 3182673 2272478 240890 2513368
1HL_29A.assembled 3004016 2288387 202722 2491109
1HL_2A.assembled 2525685 1858833 189246 2048079
1HL_31A.assembled 2203255 1636345 159656 1796001
1HL_33A.assembled 2357587 1733974 178510 1912484
1HL_34A.assembled 2461367 1795163 191006 1986169
1HL_35A.assembled 2222135 1621437 168996 1790433
1HL_3A.assembled 1994744 1473692 146406 1620098
1HL_4A.assembled 2492589 1890737 171986 2062723
1HL_5A.assembled 1914213 1462615 127309 1589924
1HL_6A.assembled 1778884 1322008 130346 1452354
1HL_7A.assembled 2555275 1862044 181746 2043790
1HL_8A.assembled 2330048 1756156 162264 1918420
1HL_9A.assembled 2620475 1941412 189818 2131230
1NF_10A.assembled 1955600 1455878 144024 1599902
1NF_11A.assembled 2004118 1477037 151957 1628994
1NF_12A.assembled 2051515 1531004 146564 1677568
1NF_13A.assembled 3671926 2725655 269586 2995241
1NF_14A.assembled 2333572 1708721 178913 1887634
1NF_15A.assembled 2296320 1697717 170815 1868532
1NF_16A.assembled 2701311 2082692 176785 2259477
1NF_17A.assembled 2846779 2102287 210452 2312739
1NF_18A.assembled 1469499 1065959 116806 1182765
1NF_19A.assembled 1775039 1310095 133947 1444042
1NF_1A.assembled 1157346 850786 89996 940782
1NF_20A.assembled 2261913 1689225 164421 1853646
1NF_21A.assembled 2903774 2133792 221401 2355193
1NF_22A.assembled 2674529 1978558 197203 2175761
1NF_23A.assembled 3106447 2328959 222988 2551947
1NF_24A.assembled 2650885 1941071 205135 2146206
1NF_25A.assembled 1879642 1378557 146071 1524628
1NF_26A.assembled 2616210 1900022 206048 2106070
1NF_27A.assembled 2158729 1630790 151335 1782125
1NF_28A.assembled 2221247 1685941 149698 1835639
1NF_29A.assembled 3514784 2617734 257383 2875117
1NF_2A.assembled 3135623 2306002 239165 2545167
1NF_30A.assembled 2821297 2180592 179533 2360125
1NF_31A.assembled 3191552 2338303 244951 2583254
1NF_32A.assembled 3264224 2413332 239270 2652602
1NF_33A.assembled 2473981 1850165 178316 2028481
1NF_4A.assembled 3260931 2396178 247926 2644104
1NF_5A.assembled 2436600 1796651 190767 1987418
1NF_6A.assembled 2221155 1650625 164512 1815137
1NF_7A.assembled 2431311 1825704 174568 2000272
1NF_8A.assembled 2436005 1788363 187953 1976316
1NF_9A.assembled 3008281 2205187 231508 2436695
1SN_10A.assembled 2404256 1798762 169244 1968006
1SN_11A.assembled 2486121 1856100 176791 2032891
1SN_12A.assembled 3042628 2223439 229758 2453197
1SN_13A.assembled 2667803 1973010 195834 2168844
1SN_14A.assembled 3219771 2373958 236954 2610912
1SN_15A.assembled 2138651 1557006 167519 1724525
1SN_16A.assembled 2321176 1702395 174707 1877102
1SN_17A.assembled 2578853 1887817 197186 2085003
1SN_18A.assembled 2962022 2185019 219331 2404350
1SN_19A.assembled 2768323 2042906 208850 2251756
1SN_1A.assembled 2667969 1966508 204226 2170734
1SN_20A.assembled 2144390 1629218 144236 1773454
1SN_21A.assembled 3074446 2240437 241641 2482078
1SN_22A.assembled 2834716 2101705 206100 2307805
1SN_23A.assembled 3302973 2390188 260212 2650400
1SN_24A.assembled 1634612 1212113 118238 1330351
1SN_25A.assembled 2592529 1894147 197899 2092046
1SN_26A.assembled 2405605 1760829 182699 1943528
1SN_27A.assembled 2257668 1681310 160915 1842225
1SN_28A.assembled 2505410 1846471 186214 2032685
1SN_29A.assembled 1902038 1395597 145369 1540966
1SN_2A.assembled 2797966 2073626 202843 2276469
1SN_30A.assembled 3478736 2550190 265223 2815413
1SN_31A.assembled 2993047 2215634 218769 2434403
1SN_32A.assembled 2259662 1666918 167017 1833935
1SN_3A.assembled 2078939 1527695 157980 1685675
1SN_4A.assembled 2993530 2177510 234644 2412154
1SN_5A.assembled 2699104 1976652 205514 2182166
1SN_6A.assembled 3050907 2243129 227550 2470679
1SN_7A.assembled 2716067 1990379 203176 2193555
1SN_8A.assembled 3117883 2332672 221072 2553744
1SN_9A.assembled 2905903 2126562 223282 2349844
Nreads = total number of reads for a sample
passed = retained reads that passed quality filtering at full length
passed.w.trim= retained reads that were trimmed due to detection of adapters
passed.total = total kept reads of sufficient length
note: you can set the option in params file to include trimmed reads of xx length.
Step 3: Clustering¶
In [36]:
%%bash
time pyrad -p /data/analysis/params.txt -s 3
skipping 1HL_17A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_11A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_10A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_13A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_12A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_16A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_14A.assembled.clustS.gz already exists in /data/analysis/clust.88 skipping 1HL_15A.assembled.clustS.gz already exists in /data/analysis/clust.88
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
de-replicating files for clustering...
step 3: within-sample clustering of 96 samples at
'.88' similarity. Running 8 parallel jobs
with up to 16 threads per job. If needed,
adjust to avoid CPU and MEM limits
sample 1NF_29A finished, 97131 loci
sample 1NF_31A finished, 109056 loci
sample 1NF_4A finished, 110918 loci
sample 1NF_32A finished, 115584 loci
sample 1SN_14A finished, 110527 loci
sample 1SN_23A finished, 118344 loci
sample 1SN_30A finished, 121564 loci
sample 1NF_13A finished, 127575 loci
sample 1HL_19A finished, 111346 loci
sample 1HL_23A finished, 94229 loci
sample 1NF_2A finished, 102160 loci
sample 1SN_8A finished, 102124 loci
sample 1NF_23A finished, 122732 loci
sample 1HL_28A finished, 104605 loci
sample 1SN_21A finished, 131359 loci
sample 1SN_12A finished, 116140 loci
sample 1HL_29A finished, 93081 loci
sample 1SN_6A finished, 107796 loci
sample 1NF_9A finished, 103833 loci
sample 1SN_31A finished, 103367 loci
sample 1SN_18A finished, 113397 loci
sample 1HL_26A finished, 109215 loci
sample 1SN_4A finished, 111639 loci
sample 1NF_21A finished, 161246 loci
sample 1NF_30A finished, 92056 loci
sample 1SN_9A finished, 126666 loci
sample 1HL_27A finished, 99679 loci
sample 1NF_17A finished, 108371 loci
sample 1SN_22A finished, 107304 loci
sample 1SN_2A finished, 106745 loci
sample 1SN_19A finished, 120143 loci
sample 1NF_16A finished, 101457 loci
sample 1NF_22A finished, 108890 loci
sample 1SN_1A finished, 95372 loci
sample 1SN_13A finished, 106500 loci
sample 1SN_7A finished, 107945 loci
sample 1SN_5A finished, 121723 loci
sample 1HL_25A finished, 89569 loci
sample 1NF_24A finished, 109692 loci
sample 1NF_26A finished, 105799 loci
sample 1SN_25A finished, 120465 loci
sample 1HL_9A finished, 113633 loci
sample 1SN_17A finished, 106838 loci
sample 1HL_20A finished, 96673 loci
sample 1HL_4A finished, 93231 loci
sample 1HL_2A finished, 93559 loci
sample 1SN_28A finished, 96449 loci
sample 1NF_33A finished, 91154 loci
sample 1SN_11A finished, 104454 loci
sample 1HL_34A finished, 90703 loci
sample 1NF_5A finished, 97753 loci
sample 1HL_7A finished, 100958 loci
sample 1NF_7A finished, 96247 loci
sample 1NF_8A finished, 98509 loci
sample 1SN_10A finished, 107961 loci
sample 1SN_26A finished, 100318 loci
sample 1HL_33A finished, 90312 loci
sample 1NF_14A finished, 99152 loci
sample 1HL_8A finished, 87336 loci
sample 1NF_15A finished, 93177 loci
sample 1HL_24A finished, 88576 loci
sample 1SN_16A finished, 96334 loci
sample 1NF_20A finished, 91488 loci
sample 1SN_27A finished, 96854 loci
sample 1NF_28A finished, 92468 loci
sample 1SN_32A finished, 117048 loci
sample 1NF_6A finished, 94940 loci
sample 1HL_21A finished, 104256 loci
sample 1HL_31A finished, 95291 loci
sample 1NF_27A finished, 90432 loci
sample 1HL_35A finished, 95108 loci
sample 1SN_15A finished, 97901 loci
sample 1SN_20A finished, 110647 loci
sample 1NF_12A finished, 83679 loci
sample 1HL_1A finished, 89466 loci
sample 1NF_10A finished, 90811 loci
sample 1NF_11A finished, 88047 loci
sample 1SN_3A finished, 123243 loci
sample 1HL_3A finished, 113436 loci
sample 1HL_5A finished, 87447 loci
sample 1SN_29A finished, 88849 loci
sample 1NF_25A finished, 83423 loci
sample 1NF_1A finished, 62149 loci
sample 1NF_19A finished, 75939 loci
sample 1SN_24A finished, 83024 loci
sample 1HL_22A finished, 92252 loci
sample 1HL_6A finished, 85254 loci
sample 1NF_18A finished, 87277 loci
real 1003m49.345s
user 4473m36.422s
sys 3013m44.243s
Let's take a look at the results¶
In [37]:
%%bash
cat /data/analysis/stats/s3.clusters.txt
taxa total dpt.me dpt.sd d>5.tot d>5.me d>5.sd badpairs
## total = total number of clusters, including singletons
## dpt.me = mean depth of clusters
## dpt.sd = standard deviation of cluster depth
## >N.tot = number of clusters with depth greater than N
## >N.me = mean depth of clusters with depth greater than N
## >N.sd = standard deviation of cluster depth for clusters with depth greater than N
## badpairs = mismatched 1st & 2nd reads (only for paired ddRAD data)
HISTOGRAMS
taxa total dpt.me dpt.sd d>5.tot d>5.me d>5.sd badpairs
## total = total number of clusters, including singletons
## dpt.me = mean depth of clusters
## dpt.sd = standard deviation of cluster depth
## >N.tot = number of clusters with depth greater than N
## >N.me = mean depth of clusters with depth greater than N
## >N.sd = standard deviation of cluster depth for clusters with depth greater than N
## badpairs = mismatched 1st & 2nd reads (only for paired ddRAD data)
HISTOGRAMS
taxa total dpt.me dpt.sd d>5.tot d>5.me d>5.sd badpairs
1HL_11A.assembled 125407 10.143 106.054 39824 27.904 186.958 0
## total = total number of clusters, including singletons
## dpt.me = mean depth of clusters
## dpt.sd = standard deviation of cluster depth
## >N.tot = number of clusters with depth greater than N
## >N.me = mean depth of clusters with depth greater than N
## >N.sd = standard deviation of cluster depth for clusters with depth greater than N
## badpairs = mismatched 1st & 2nd reads (only for paired ddRAD data)
HISTOGRAMS
sample: 1HL_11A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 81060
5 ***** 15999
10 *** 8344
15 ** 5099
20 ** 3282
25 ** 2270
30 * 1646
35 * 1247
40 * 1846
50 ** 3628
100 * 738
250 * 160
500 * 88
taxa total dpt.me dpt.sd d>5.tot d>5.me d>5.sd badpairs
1HL_19A.assembled 111346 11.783 258.313 38571 30.329 438.285 0
1HL_1A.assembled 89466 9.948 347.089 29477 26.013 604.363 0
1HL_20A.assembled 96673 10.944 193.672 32643 28.529 332.587 0
1HL_21A.assembled 104256 9.054 192.249 30870 26.107 352.712 0
1HL_22A.assembled 92252 8.599 259.696 27812 23.907 472.615 0
1HL_23A.assembled 94229 13.942 180.382 39737 30.281 276.936 0
1HL_24A.assembled 88576 10.82 149.102 30992 27.192 251.243 0
1HL_25A.assembled 89569 12.118 226.818 34969 27.81 362.446 0
1HL_26A.assembled 109215 11.438 420.149 36553 30.405 725.87 0
1HL_27A.assembled 99679 12.006 223.061 38529 27.889 358.205 0
1HL_28A.assembled 104605 12.051 197.342 38344 29.491 325.205 0
1HL_29A.assembled 93081 13.658 750.0 35000 32.963 1222.843 0
1HL_2A.assembled 93559 11.288 201.279 34228 27.314 332.162 0
1HL_31A.assembled 95291 9.835 251.465 31182 25.924 439.152 0
1HL_33A.assembled 90312 10.396 79.055 32034 25.631 131.365 0
1HL_34A.assembled 90703 11.697 246.572 34024 27.797 402.071 0
1HL_35A.assembled 95108 10.032 228.329 32759 25.228 388.592 0
1HL_3A.assembled 113436 7.502 184.256 29682 23.511 359.717 0
1HL_4A.assembled 93231 11.593 537.487 33129 28.966 901.401 0
1HL_5A.assembled 87447 9.52 487.743 27033 26.22 877.004 0
1HL_6A.assembled 85254 8.901 277.694 27005 23.706 493.074 0
1HL_7A.assembled 100958 10.385 361.718 34619 26.445 617.387 0
1HL_8A.assembled 87336 13.276 960.795 30092 34.623 1636.609 0
1HL_9A.assembled 113633 9.831 316.472 33918 28.622 578.821 0
1NF_10A.assembled 90811 9.312 299.189 27231 26.366 545.981 0
1NF_11A.assembled 88047 9.677 196.397 28562 25.554 344.278 0
1NF_12A.assembled 83679 10.531 310.007 28530 26.891 530.534 0
1NF_13A.assembled 127575 11.923 359.562 43922 30.945 612.343 0
1NF_14A.assembled 99152 9.885 154.482 32182 26.324 270.413 0
1NF_15A.assembled 93177 10.451 264.876 32279 26.332 449.593 0
1NF_16A.assembled 101457 11.024 607.705 33035 29.739 1064.747 0
1NF_17A.assembled 108371 11.106 255.998 37017 28.707 437.478 0
1NF_18A.assembled 87277 7.406 89.423 24585 21.141 167.693 0
1NF_19A.assembled 75939 9.628 190.419 24889 25.087 332.074 0
1NF_1A.assembled 62149 8.108 174.832 18717 22.047 318.139 0
1NF_20A.assembled 91488 10.527 312.509 31878 26.372 529.052 0
1NF_21A.assembled 161246 7.618 219.851 39461 25.754 443.92 0
1NF_22A.assembled 108890 9.774 85.584 35333 26.057 148.922 0
1NF_23A.assembled 122732 10.619 411.238 40204 28.507 718.184 0
1NF_24A.assembled 109692 10.192 149.97 36529 26.586 259.098 0
1NF_25A.assembled 83423 9.536 148.404 27776 24.477 256.531 0
1NF_26A.assembled 105799 10.345 146.292 37361 25.612 245.441 0
1NF_27A.assembled 90432 10.227 440.639 29792 26.848 767.433 0
1NF_28A.assembled 92468 10.505 517.33 30337 27.845 902.936 0
1NF_29A.assembled 97131 15.049 542.183 38679 34.741 858.809 0
1NF_2A.assembled 102160 12.238 124.042 39165 28.671 199.235 0
1NF_30A.assembled 92056 13.305 858.877 33366 33.131 1426.392 0
1NF_31A.assembled 109056 12.184 211.596 40183 29.644 347.89 0
1NF_32A.assembled 115584 11.831 295.114 42115 29.092 488.418 0
1NF_33A.assembled 91154 11.593 412.218 33851 27.73 676.131 0
1NF_4A.assembled 110918 12.304 250.034 42340 29.001 404.131 0
1NF_5A.assembled 97753 10.474 250.119 34708 25.809 419.318 0
1NF_6A.assembled 94940 9.368 118.351 30978 24.514 206.359 0
1NF_7A.assembled 96247 10.695 394.478 32142 27.94 682.291 0
1NF_8A.assembled 98509 10.389 241.035 33332 26.768 413.876 0
1NF_9A.assembled 103833 11.85 94.91 38787 28.323 153.878 0
1SN_10A.assembled 107961 9.501 325.43 32953 26.629 588.677 0
1SN_11A.assembled 104454 10.049 303.261 33415 27.275 535.767 0
1SN_12A.assembled 116140 10.748 152.998 41460 26.503 255.311 0
1SN_13A.assembled 106500 10.499 239.77 36704 26.636 407.936 0
1SN_14A.assembled 110527 12.04 247.494 39987 29.769 410.869 0
1SN_15A.assembled 97901 9.263 79.841 32175 24.095 138.079 0
1SN_16A.assembled 96334 10.109 192.251 32132 26.332 332.283 0
1SN_17A.assembled 106838 10.042 126.215 36986 25.224 213.683 0
1SN_18A.assembled 113397 11.024 258.755 37830 29.143 447.44 0
1SN_19A.assembled 120143 9.694 213.952 37066 27.205 384.612 0
1SN_1A.assembled 95372 11.692 243.657 34549 28.667 404.267 0
1SN_20A.assembled 110647 8.501 460.876 28105 27.957 914.175 0
1SN_21A.assembled 131359 9.694 116.903 40472 27.254 209.541 0
1SN_22A.assembled 107304 11.036 304.238 37147 28.069 516.65 0
1SN_23A.assembled 118344 11.696 134.392 43564 28.418 220.498 0
1SN_24A.assembled 83024 8.485 194.621 24894 23.525 354.961 0
1SN_25A.assembled 120465 8.972 171.923 34291 26.879 321.535 0
1SN_26A.assembled 100318 10.0 154.535 33532 25.88 266.578 0
1SN_27A.assembled 96854 10.007 270.905 30897 27.075 479.192 0
1SN_28A.assembled 96449 11.092 223.44 34014 27.717 375.682 0
1SN_29A.assembled 88849 9.162 283.094 23680 29.309 547.853 0
1SN_2A.assembled 106745 11.138 256.265 36309 28.841 438.851 0
1SN_30A.assembled 121564 11.995 182.973 43872 29.826 303.755 0
1SN_31A.assembled 103367 11.972 280.474 37483 29.472 465.245 0
1SN_32A.assembled 117048 8.287 217.588 30447 26.647 426.085 0
1SN_3A.assembled 123243 7.339 191.784 30283 24.092 386.409 0
1SN_4A.assembled 111639 10.976 67.766 40547 26.738 110.685 0
1SN_5A.assembled 121723 9.342 129.073 38686 25.228 228.135 0
1SN_6A.assembled 107796 11.845 209.367 39610 28.769 344.727 0
1SN_7A.assembled 107945 11.257 414.056 37281 28.845 704.22 0
1SN_8A.assembled 102124 12.804 475.968 39290 30.02 767.048 0
1SN_9A.assembled 126666 9.535 155.002 39406 26.521 277.138 0
## total = total number of clusters, including singletons
## dpt.me = mean depth of clusters
## dpt.sd = standard deviation of cluster depth
## >N.tot = number of clusters with depth greater than N
## >N.me = mean depth of clusters with depth greater than N
## >N.sd = standard deviation of cluster depth for clusters with depth greater than N
## badpairs = mismatched 1st & 2nd reads (only for paired ddRAD data)
HISTOGRAMS
sample: 1HL_19A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 68447
5 ***** 15348
10 *** 8028
15 ** 4717
20 ** 3018
25 ** 2157
30 * 1601
35 * 1296
40 ** 1860
50 ** 3880
100 * 762
250 * 138
500 * 94
sample: 1HL_1A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 55800
5 ****** 14663
10 *** 6265
15 ** 3286
20 ** 2205
25 ** 1626
30 * 1261
35 * 963
40 * 1453
50 * 1454
100 * 369
250 * 71
500 * 49
sample: 1HL_20A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 59978
5 ***** 14290
10 *** 6839
15 ** 3593
20 ** 2286
25 ** 1754
30 * 1352
35 * 1130
40 ** 1702
50 ** 2969
100 * 584
250 * 122
500 * 74
sample: 1HL_21A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 69366
5 ***** 14289
10 *** 6552
15 ** 3529
20 ** 2351
25 * 1616
30 * 1265
35 * 1038
40 * 1625
50 ** 2010
100 * 463
250 * 93
500 * 59
sample: 1HL_22A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 60207
5 ****** 14409
10 *** 6005
15 ** 3259
20 ** 2225
25 ** 1558
30 * 1193
35 * 939
40 * 1060
50 * 959
100 * 333
250 * 65
500 * 40
sample: 1HL_23A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ***************** 50797
5 ***** 13857
10 *** 7823
15 *** 5087
20 ** 3695
25 ** 2718
30 ** 2039
35 * 1464
40 ** 2053
50 ** 3627
100 * 794
250 * 175
500 * 100
sample: 1HL_24A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 53755
5 ****** 13696
10 *** 6422
15 ** 3364
20 ** 2299
25 ** 1676
30 * 1329
35 * 1106
40 ** 1662
50 ** 2599
100 * 500
250 * 103
500 * 65
sample: 1HL_25A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ****************** 50626
5 ****** 14673
10 *** 7408
15 ** 4284
20 ** 2640
25 ** 1913
30 * 1452
35 * 1159
40 ** 1714
50 ** 2900
100 * 612
250 * 115
500 * 73
sample: 1HL_26A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 68621
5 ***** 14751
10 *** 7815
15 ** 4704
20 ** 2935
25 ** 2068
30 * 1503
35 * 1186
40 ** 1854
50 ** 2966
100 * 597
250 * 139
500 * 75
sample: 1HL_27A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ****************** 57155
5 ****** 15022
10 *** 8191
15 *** 5108
20 ** 3421
25 ** 2289
30 ** 1699
35 * 1257
40 ** 1791
50 ** 2886
100 * 651
250 * 134
500 * 75
sample: 1HL_28A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 62056
5 ***** 14868
10 *** 7895
15 ** 4977
20 ** 3150
25 ** 2085
30 * 1592
35 * 1265
40 ** 1889
50 ** 3830
100 * 774
250 * 138
500 * 86
sample: 1HL_29A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ****************** 54102
5 ****** 14459
10 *** 7471
15 ** 4356
20 ** 2782
25 ** 1895
30 * 1421
35 * 1099
40 ** 1734
50 ** 2963
100 * 599
250 * 127
500 * 72
sample: 1HL_2A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 55293
5 ****** 14690
10 *** 7496
15 ** 4060
20 ** 2452
25 ** 1830
30 * 1370
35 * 1114
40 ** 1686
50 ** 2813
100 * 569
250 * 114
500 * 72
sample: 1HL_31A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 59878
5 ****** 14849
10 *** 6772
15 ** 3561
20 ** 2301
25 ** 1618
30 * 1274
35 * 1035
40 * 1558
50 ** 1887
100 * 415
250 * 90
500 * 53
sample: 1HL_33A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 54388
5 ****** 14080
10 *** 6826
15 ** 3683
20 ** 2289
25 ** 1839
30 * 1318
35 * 1072
40 ** 1591
50 ** 2543
100 * 520
250 * 97
500 * 66
sample: 1HL_34A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ****************** 52753
5 ****** 14439
10 *** 7250
15 ** 4027
20 ** 2581
25 ** 1893
30 * 1352
35 * 1138
40 ** 1660
50 ** 2834
100 * 588
250 * 119
500 * 69
sample: 1HL_35A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 57982
5 ****** 15201
10 *** 7157
15 ** 3969
20 ** 2415
25 ** 1786
30 * 1319
35 * 1086
40 ** 1642
50 ** 1970
100 * 446
250 * 78
500 * 57
sample: 1HL_3A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************** 79456
5 ***** 14740
10 *** 6701
15 ** 3537
20 ** 2253
25 * 1605
30 * 1215
35 * 954
40 * 1325
50 * 1194
100 * 340
250 * 71
500 * 45
sample: 1HL_4A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 56050
5 ****** 14669
10 *** 7245
15 ** 4170
20 ** 2624
25 ** 1814
30 * 1321
35 * 1090
40 ** 1644
50 ** 1954
100 * 485
250 * 108
500 * 56
sample: 1HL_5A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 56227
5 ****** 14257
10 *** 5890
15 ** 3100
20 ** 2004
25 ** 1555
30 * 1217
35 * 933
40 * 971
50 * 935
100 * 258
250 * 60
500 * 39
sample: 1HL_6A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 54262
5 ****** 13951
10 *** 5618
15 ** 3062
20 ** 2158
25 ** 1600
30 * 1277
35 * 924
40 * 1057
50 * 968
100 * 287
250 * 55
500 * 35
sample: 1HL_7A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 61941
5 ****** 15596
10 *** 7832
15 ** 4635
20 ** 2781
25 ** 1897
30 * 1388
35 * 1043
40 * 1592
50 * 1657
100 * 438
250 * 98
500 * 59
sample: 1HL_8A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 53227
5 ****** 13942
10 *** 6123
15 ** 3357
20 ** 2316
25 ** 1653
30 * 1288
35 * 1023
40 ** 1609
50 ** 2152
100 * 484
250 * 99
500 * 62
sample: 1HL_9A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 75819
5 ***** 14427
10 *** 7230
15 ** 4104
20 ** 2675
25 * 1887
30 * 1413
35 * 1112
40 * 1690
50 ** 2544
100 * 559
250 * 106
500 * 66
sample: 1NF_10A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 59486
5 ****** 13838
10 *** 5285
15 ** 2995
20 ** 2099
25 ** 1521
30 * 1247
35 * 1041
40 * 1476
50 * 1344
100 * 357
250 * 72
500 * 50
sample: 1NF_11A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 55342
5 ****** 13998
10 *** 5684
15 ** 3071
20 ** 2136
25 ** 1624
30 * 1374
35 * 1074
40 ** 1572
50 ** 1641
100 * 401
250 * 83
500 * 47
sample: 1NF_12A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 51252
5 ****** 13806
10 *** 5740
15 ** 3176
20 ** 2028
25 ** 1603
30 * 1241
35 * 989
40 ** 1554
50 ** 1760
100 * 387
250 * 91
500 * 52
sample: 1NF_13A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 78889
5 ***** 17043
10 *** 8832
15 ** 5810
20 ** 3960
25 ** 2690
30 * 1945
35 * 1382
40 * 2066
50 ** 3833
100 * 867
250 * 157
500 * 100
sample: 1NF_14A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 62650
5 ****** 14980
10 *** 6750
15 ** 3424
20 ** 2326
25 ** 1813
30 * 1316
35 * 1155
40 ** 1761
50 ** 2314
100 * 485
250 * 107
500 * 71
sample: 1NF_15A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 56710
5 ****** 14879
10 *** 7031
15 ** 3623
20 ** 2384
25 ** 1746
30 * 1378
35 * 1053
40 ** 1712
50 ** 2044
100 * 447
250 * 105
500 * 65
sample: 1NF_16A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 63999
5 ****** 15463
10 *** 7182
15 ** 3826
20 ** 2460
25 ** 1773
30 * 1381
35 * 1111
40 * 1679
50 ** 1976
100 * 442
250 * 108
500 * 56
sample: 1NF_17A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 66920
5 ***** 15679
10 *** 7793
15 ** 4436
20 ** 2751
25 ** 2007
30 * 1587
35 * 1203
40 ** 1834
50 ** 3288
100 * 661
250 * 133
500 * 79
sample: 1NF_18A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 58644
5 ****** 13303
10 *** 5111
15 ** 2923
20 ** 1980
25 ** 1457
30 * 1091
35 * 839
40 * 859
50 * 724
100 * 252
250 * 62
500 * 32
sample: 1NF_19A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 47341
5 ****** 12391
10 *** 4652
15 ** 2727
20 ** 2058
25 ** 1526
30 * 1215
35 * 1002
40 ** 1356
50 * 1225
100 * 341
250 * 69
500 * 36
sample: 1NF_1A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 40415
5 ****** 9764
10 *** 3824
15 ** 2502
20 ** 1812
25 ** 1283
30 * 850
35 * 495
40 * 467
50 * 481
100 * 198
250 * 33
500 * 25
sample: 1NF_20A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 55363
5 ****** 14974
10 *** 6816
15 ** 3668
20 ** 2391
25 ** 1692
30 * 1364
35 * 1163
40 ** 1653
50 ** 1825
100 * 430
250 * 85
500 * 64
sample: 1NF_21A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 *********************** 117090
5 **** 16991
10 *** 8855
15 ** 5584
20 ** 3381
25 * 2168
30 * 1504
35 * 1150
40 * 1820
50 * 1981
100 * 536
250 * 108
500 * 78
sample: 1NF_22A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 69222
5 ***** 15624
10 *** 7399
15 ** 4179
20 ** 2636
25 ** 1857
30 * 1442
35 * 1142
40 * 1743
50 ** 2885
100 * 558
250 * 138
500 * 65
sample: 1NF_23A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 78024
5 ***** 16707
10 *** 8705
15 ** 5301
20 ** 3276
25 ** 2225
30 * 1603
35 * 1291
40 * 1869
50 ** 2898
100 * 613
250 * 132
500 * 87
sample: 1NF_24A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 68448
5 ****** 16514
10 *** 7667
15 ** 4237
20 ** 2554
25 ** 1923
30 * 1491
35 * 1193
40 ** 1833
50 ** 3029
100 * 602
250 * 127
500 * 74
sample: 1NF_25A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 51529
5 ****** 13841
10 *** 5522
15 ** 3049
20 ** 2085
25 ** 1613
30 * 1250
35 * 1063
40 ** 1562
50 * 1389
100 * 393
250 * 82
500 * 45
sample: 1NF_26A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 63962
5 ****** 16117
10 *** 8166
15 ** 4751
20 ** 2925
25 ** 2048
30 * 1479
35 * 1152
40 ** 1787
50 ** 2625
100 * 590
250 * 122
500 * 75
sample: 1NF_27A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 56374
5 ****** 15004
10 *** 6276
15 ** 3441
20 ** 2277
25 ** 1624
30 * 1293
35 * 1056
40 * 1314
50 * 1290
100 * 350
250 * 77
500 * 55
sample: 1NF_28A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 57828
5 ****** 14884
10 *** 6545
15 ** 3515
20 ** 2293
25 ** 1737
30 * 1324
35 * 1045
40 * 1415
50 * 1397
100 * 356
250 * 85
500 * 43
sample: 1NF_29A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ****************** 54455
5 ****** 14658
10 *** 7747
15 ** 4706
20 ** 2984
25 ** 2020
30 * 1597
35 * 1348
40 ** 1991
50 ** 4385
100 * 944
250 * 195
500 * 100
sample: 1NF_2A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ****************** 58774
5 ***** 15245
10 *** 8081
15 ** 5015
20 ** 3276
25 ** 2233
30 * 1592
35 * 1288
40 ** 1940
50 ** 3691
100 * 771
250 * 170
500 * 84
sample: 1NF_30A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 54566
5 ****** 14727
10 *** 7037
15 ** 3924
20 ** 2587
25 ** 1891
30 * 1367
35 * 1090
40 ** 1742
50 ** 2416
100 * 534
250 * 99
500 * 75
sample: 1NF_31A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 64382
5 ***** 15952
10 *** 8419
15 ** 5021
20 ** 3255
25 ** 2280
30 * 1765
35 * 1299
40 ** 1947
50 ** 3724
100 * 749
250 * 167
500 * 96
sample: 1NF_32A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 69054
5 ***** 16199
10 *** 8932
15 ** 5662
20 ** 3875
25 ** 2672
30 * 1860
35 * 1294
40 ** 1984
50 ** 3095
100 * 716
250 * 148
500 * 93
sample: 1NF_33A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ****************** 53161
5 ****** 14931
10 *** 7387
15 ** 4128
20 ** 2666
25 ** 1788
30 * 1436
35 * 1085
40 ** 1745
50 ** 2167
100 * 491
250 * 102
500 * 66
sample: 1NF_4A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ****************** 64058
5 ***** 16236
10 *** 8800
15 *** 5600
20 ** 3795
25 ** 2670
30 * 1819
35 * 1346
40 ** 2014
50 ** 3583
100 * 734
250 * 163
500 * 100
sample: 1NF_5A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 58654
5 ****** 15786
10 *** 7511
15 ** 4235
20 ** 2597
25 ** 1915
30 * 1509
35 * 1129
40 ** 1762
50 ** 2018
100 * 467
250 * 106
500 * 64
sample: 1NF_6A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 59692
5 ****** 15080
10 *** 6491
15 ** 3424
20 ** 2300
25 ** 1690
30 * 1330
35 * 1138
40 * 1546
50 ** 1702
100 * 401
250 * 86
500 * 60
sample: 1NF_7A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 59628
5 ****** 15414
10 *** 6662
15 ** 3563
20 ** 2376
25 ** 1727
30 * 1299
35 * 1093
40 ** 1691
50 ** 2186
100 * 438
250 * 106
500 * 63
sample: 1NF_8A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 60915
5 ****** 15163
10 *** 7188
15 ** 3650
20 ** 2356
25 ** 1752
30 * 1409
35 * 1101
40 ** 1802
50 ** 2511
100 * 492
250 * 108
500 * 62
sample: 1NF_9A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 60663
5 ****** 15870
10 *** 8026
15 ** 4718
20 ** 3031
25 ** 2111
30 * 1623
35 * 1202
40 ** 1861
50 ** 3700
100 * 780
250 * 162
500 * 86
sample: 1SN_10A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 70432
5 ***** 15850
10 *** 7226
15 ** 3644
20 ** 2291
25 * 1757
30 * 1395
35 * 1023
40 * 1733
50 ** 2009
100 * 443
250 * 101
500 * 56
sample: 1SN_11A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 66764
5 ***** 14959
10 *** 7366
15 ** 4114
20 ** 2520
25 ** 1756
30 * 1378
35 * 1059
40 * 1612
50 ** 2261
100 * 500
250 * 103
500 * 62
sample: 1SN_12A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 69908
5 ***** 17243
10 *** 8883
15 ** 5452
20 ** 3474
25 ** 2344
30 * 1669
35 * 1335
40 ** 2003
50 ** 2933
100 * 681
250 * 134
500 * 81
sample: 1SN_13A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 65156
5 ****** 16331
10 *** 8085
15 ** 4570
20 ** 2775
25 ** 1889
30 * 1462
35 * 1140
40 ** 1854
50 ** 2500
100 * 534
250 * 128
500 * 76
sample: 1SN_14A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 66093
5 ***** 16195
10 *** 8253
15 ** 5072
20 ** 3073
25 ** 2166
30 * 1667
35 * 1265
40 ** 1950
50 ** 3773
100 * 759
250 * 175
500 * 86
sample: 1SN_15A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 61429
5 ****** 15221
10 *** 6915
15 ** 3617
20 ** 2340
25 ** 1732
30 * 1359
35 * 1122
40 ** 1694
50 ** 1866
100 * 458
250 * 89
500 * 59
sample: 1SN_16A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 60032
5 ****** 14828
10 *** 6726
15 ** 3745
20 ** 2286
25 ** 1713
30 * 1302
35 * 1090
40 ** 1700
50 ** 2274
100 * 472
250 * 102
500 * 64
sample: 1SN_17A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 65231
5 ****** 16536
10 *** 8216
15 ** 4531
20 ** 2770
25 ** 1893
30 * 1471
35 * 1148
40 ** 1852
50 ** 2432
100 * 566
250 * 121
500 * 71
sample: 1SN_18A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 71002
5 ***** 16111
10 *** 7962
15 ** 4690
20 ** 2809
25 ** 1930
30 * 1497
35 * 1204
40 * 1860
50 ** 3430
100 * 664
250 * 155
500 * 83
sample: 1SN_19A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 78620
5 ***** 15808
10 *** 8065
15 ** 4719
20 ** 2879
25 ** 2015
30 * 1466
35 * 1146
40 * 1788
50 ** 2826
100 * 614
250 * 120
500 * 77
sample: 1SN_1A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 56574
5 ****** 15056
10 *** 7145
15 ** 3874
20 ** 2457
25 ** 1910
30 * 1471
35 * 1174
40 ** 1845
50 ** 3081
100 * 588
250 * 123
500 * 74
sample: 1SN_20A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************** 78249
5 ***** 14274
10 *** 5786
15 ** 3176
20 ** 2132
25 * 1572
30 * 1213
35 * 1014
40 * 1420
50 * 1326
100 * 363
250 * 80
500 * 41
sample: 1SN_21A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 86121
5 ***** 16817
10 *** 8526
15 ** 5034
20 ** 3269
25 * 2178
30 * 1663
35 * 1262
40 * 1908
50 ** 3627
100 * 727
250 * 147
500 * 80
sample: 1SN_22A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 65559
5 ****** 16330
10 *** 7913
15 ** 4519
20 ** 2797
25 ** 2023
30 * 1454
35 * 1178
40 ** 1851
50 ** 2883
100 * 596
250 * 131
500 * 70
sample: 1SN_23A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 70248
5 ***** 16403
10 *** 8889
15 ** 5871
20 ** 4049
25 ** 2695
30 * 1948
35 * 1411
40 ** 2062
50 ** 3678
100 * 797
250 * 185
500 * 108
sample: 1SN_24A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 54175
5 ****** 12989
10 *** 5054
15 ** 2834
20 ** 1985
25 ** 1504
30 * 1183
35 * 907
40 * 1057
50 * 973
100 * 272
250 * 53
500 * 38
sample: 1SN_25A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 81776
5 ***** 15548
10 *** 7196
15 ** 3891
20 ** 2423
25 * 1764
30 * 1380
35 * 1108
40 * 1825
50 ** 2812
100 * 560
250 * 119
500 * 63
sample: 1SN_26A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 62225
5 ****** 15646
10 *** 7183
15 ** 3857
20 ** 2467
25 ** 1745
30 * 1319
35 * 1110
40 ** 1674
50 ** 2421
100 * 488
250 * 115
500 * 68
sample: 1SN_27A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 61562
5 ****** 15045
10 *** 6281
15 ** 3303
20 ** 2256
25 ** 1702
30 * 1270
35 * 1083
40 ** 1632
50 ** 2146
100 * 419
250 * 95
500 * 60
sample: 1SN_28A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 58170
5 ****** 15131
10 *** 7200
15 ** 3936
20 ** 2501
25 ** 1813
30 * 1410
35 * 1099
40 ** 1700
50 ** 2734
100 * 560
250 * 128
500 * 67
sample: 1SN_29A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************** 61992
5 ***** 10749
10 ** 4344
15 ** 2512
20 ** 1897
25 ** 1564
30 * 1215
35 * 939
40 * 1401
50 ** 1744
100 * 377
250 * 70
500 * 45
sample: 1SN_2A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 65816
5 ****** 16161
10 *** 7702
15 ** 4282
20 ** 2555
25 ** 1828
30 * 1419
35 * 1154
40 ** 1820
50 ** 3135
100 * 647
250 * 143
500 * 83
sample: 1SN_30A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 73268
5 ***** 15757
10 *** 8439
15 ** 5612
20 ** 4288
25 ** 2976
30 ** 2274
35 * 1628
40 ** 2260
50 ** 3888
100 * 881
250 * 199
500 * 94
sample: 1SN_31A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 61571
5 ****** 15614
10 *** 7864
15 ** 4528
20 ** 2742
25 ** 1990
30 * 1504
35 * 1220
40 ** 1839
50 ** 3626
100 * 642
250 * 142
500 * 85
sample: 1SN_32A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************** 82376
5 ***** 14587
10 *** 6195
15 ** 3447
20 ** 2205
25 * 1630
30 * 1348
35 * 1087
40 * 1650
50 * 1948
100 * 413
250 * 99
500 * 63
sample: 1SN_3A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************** 88130
5 ***** 15945
10 *** 6261
15 ** 3358
20 ** 2263
25 * 1647
30 * 1354
35 * 1095
40 * 1382
50 * 1300
100 * 369
250 * 98
500 * 41
sample: 1SN_4A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 66525
5 ***** 16198
10 *** 8510
15 ** 5400
20 ** 3534
25 ** 2383
30 * 1695
35 * 1220
40 ** 1861
50 ** 3351
100 * 701
250 * 175
500 * 86
sample: 1SN_5A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************** 78276
5 ***** 17217
10 *** 8370
15 ** 4862
20 ** 3089
25 ** 2090
30 * 1523
35 * 1202
40 * 1765
50 ** 2582
100 * 558
250 * 113
500 * 76
sample: 1SN_6A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 63699
5 ***** 16069
10 *** 8480
15 ** 5081
20 ** 3165
25 ** 2176
30 * 1587
35 * 1268
40 ** 1886
50 ** 3463
100 * 681
250 * 153
500 * 88
sample: 1SN_7A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ******************* 66184
5 ***** 16029
10 *** 8005
15 ** 4767
20 ** 3007
25 ** 2051
30 * 1525
35 * 1145
40 ** 1807
50 ** 2632
100 * 587
250 * 126
500 * 79
sample: 1SN_8A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ****************** 58494
5 ****** 15867
10 *** 8341
15 *** 5216
20 ** 3433
25 ** 2249
30 * 1596
35 * 1282
40 ** 1962
50 ** 2791
100 * 686
250 * 132
500 * 74
sample: 1SN_9A.assembled
bins depth_histogram cnts
: 0------------50-------------100%
0 ********************* 82655
5 ***** 16474
10 *** 8607
15 ** 5154
20 ** 3303
25 ** 2210
30 * 1521
35 * 1194
40 * 1810
50 ** 2886
100 * 644
250 * 125
500 * 83
Step 4: Parameter estimation¶
In [38]:
%%bash
time pyrad -p /data/analysis/params.txt -s 4
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
step 4: estimating error rate and heterozygosity
................................................................................................
real 1033m58.941s
user 7810m54.513s
sys 0m26.031s
In [1]:
%%bash
cat /data/analysis/stats/Pi_E_estimate.txt
taxa H E 1HL_13A.assembled 0.01234178 0.00144187 1HL_10A.assembled 0.01399484 0.00139753 1HL_15A.assembled 0.01242723 0.0011879 1HL_12A.assembled 0.01240698 0.00157456 1HL_16A.assembled 0.01381188 0.00054237 1HL_17A.assembled 0.01332716 0.00127763 1HL_14A.assembled 0.01266762 0.00203182 1HL_11A.assembled 0.01216978 0.00165352 1HL_13A.assembled 0.01234178 0.00144187 1HL_10A.assembled 0.01399484 0.00139753 1HL_15A.assembled 0.01242723 0.0011879 1HL_12A.assembled 0.01240698 0.00157456 1HL_16A.assembled 0.01381188 0.00054237 1HL_17A.assembled 0.01332716 0.00127763 1HL_14A.assembled 0.01266762 0.00203182 1NF_1A.assembled 0.01302939 0.00167192 1NF_19A.assembled 0.01302296 0.00156732 1SN_24A.assembled 0.01280068 0.00172812 1NF_18A.assembled 0.01298092 0.00169576 1SN_29A.assembled 0.01288545 0.0014629 1HL_6A.assembled 0.01312247 0.00152568 1NF_25A.assembled 0.01280702 0.00162634 1NF_12A.assembled 0.01233271 0.00160106 1HL_5A.assembled 0.01210494 0.00151496 1NF_10A.assembled 0.01256406 0.00160585 1NF_11A.assembled 0.01268246 0.00165147 1HL_22A.assembled 0.01296414 0.00141904 1HL_1A.assembled 0.01264807 0.00153935 1NF_27A.assembled 0.01276751 0.00161456 1HL_8A.assembled 0.01278796 0.00153183 1NF_28A.assembled 0.01270732 0.0013666 1HL_24A.assembled 0.01238308 0.00167162 1NF_20A.assembled 0.01238014 0.0016131 1HL_33A.assembled 0.01272048 0.00149538 1NF_6A.assembled 0.01228499 0.00146483 1NF_15A.assembled 0.01227426 0.0016802 1NF_33A.assembled 0.0122457 0.0014486 1HL_31A.assembled 0.01280596 0.00156314 1NF_30A.assembled 0.01249949 0.00142214 1HL_4A.assembled 0.0124557 0.00147379 1HL_25A.assembled 0.01237222 0.00147768 1NF_7A.assembled 0.01265262 0.0015292 1SN_16A.assembled 0.01240499 0.00171252 1HL_34A.assembled 0.0125749 0.00155212 1SN_27A.assembled 0.01287353 0.00165881 1HL_35A.assembled 0.01271276 0.00166711 1SN_15A.assembled 0.01229991 0.00183908 1HL_2A.assembled 0.01213818 0.00163783 1HL_20A.assembled 0.01231824 0.00169476 1NF_14A.assembled 0.01255169 0.00174661 1NF_8A.assembled 0.01200827 0.00145281 1NF_5A.assembled 0.01213863 0.00174276 1SN_20A.assembled 0.01273133 0.00142206 1HL_29A.assembled 0.01255651 0.00138799 1SN_1A.assembled 0.01224028 0.00159679 1HL_21A.assembled 0.01272641 0.00165356 1SN_28A.assembled 0.01232066 0.00158669 1NF_16A.assembled 0.01270907 0.00133209 1SN_26A.assembled 0.01229891 0.00164706 1HL_3A.assembled 0.01232275 0.00170013 1SN_11A.assembled 0.01217015 0.0017145 1SN_10A.assembled 0.01259095 0.00153324 1HL_7A.assembled 0.01250226 0.00159767 1HL_27A.assembled 0.01243331 0.00153194 1SN_32A.assembled 0.01259383 0.00168319 1NF_26A.assembled 0.0123278 0.0016356 1HL_23A.assembled 0.0120483 0.00141033 1NF_22A.assembled 0.012368 0.00153584 1SN_13A.assembled 0.01218486 0.00168494 1SN_8A.assembled 0.0121543 0.00123362 1SN_17A.assembled 0.01209382 0.00169839 1HL_9A.assembled 0.01225037 0.00138481 1NF_29A.assembled 0.01233428 0.00133912 1SN_2A.assembled 0.01243021 0.00158393 1SN_31A.assembled 0.01246079 0.00163411 1SN_7A.assembled 0.01225447 0.00159903 1NF_2A.assembled 0.01183364 0.00155891 1SN_22A.assembled 0.01251389 0.00158949 1NF_9A.assembled 0.01216885 0.00156536 1HL_26A.assembled 0.01233653 0.00150096 1NF_17A.assembled 0.01222267 0.00155723 1NF_24A.assembled 0.01246688 0.00156561 1SN_3A.assembled 0.01289977 0.00173733 1SN_6A.assembled 0.01147223 0.00148751 1SN_18A.assembled 0.01207448 0.00169724 1SN_25A.assembled 0.01237121 0.00153697 1NF_31A.assembled 0.0120439 0.00169293 1SN_4A.assembled 0.01186629 0.00181512 1HL_19A.assembled 0.01249681 0.001469 1SN_14A.assembled 0.01188004 0.00151311 1HL_28A.assembled 0.01284234 0.00148895 1SN_19A.assembled 0.01226379 0.00146784 1NF_4A.assembled 0.01189766 0.00151837 1SN_5A.assembled 0.01218246 0.00154433 1NF_32A.assembled 0.01184959 0.00150245 1SN_12A.assembled 0.01194628 0.0015164 1NF_23A.assembled 0.01202496 0.00159625 1SN_9A.assembled 0.01227092 0.00147231 1SN_23A.assembled 0.0119076 0.00165804 1HL_11A.assembled 0.01216978 0.00165352 1SN_30A.assembled 0.01212247 0.00153805 1SN_21A.assembled 0.01214048 0.00152167 1NF_13A.assembled 0.01200208 0.00144921 1NF_21A.assembled 0.01227234 0.00160427
Step 5: Consenus base calling¶
In [2]:
%%bash
time pyrad -p /data/analysis/params.txt -s 5
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
step 5: creating consensus seqs for 96 samples, using H=0.01247 E=0.00154
................................................................................................
real 283m28.123s
user 2225m48.454s
sys 0m21.896s
In [3]:
%%bash
cat /data/analysis/stats/s5.consens.txt
taxon nloci f1loci f2loci nsites npoly poly
1SN_9A.assembled 126666 39351 36531 3608975 27926 0.0077379
1NF_21A.assembled 161246 39443 36671 3636146 28281 0.0077777
1HL_11A.assembled 125407 39736 36789 3659499 28748 0.0078557
1SN_21A.assembled 131359 40405 37391 3710615 29515 0.0079542
1NF_23A.assembled 122732 40195 37205 3685221 27872 0.0075632
1SN_23A.assembled 118344 43478 40280 4003823 30280 0.0075628
1SN_30A.assembled 121564 43820 40498 4028393 30831 0.0076534
1NF_13A.assembled 127575 43904 40669 4061429 31655 0.0077941
1SN_19A.assembled 120143 37039 34302 3391779 26288 0.0077505
1SN_5A.assembled 121723 38622 35831 3535911 27199 0.0076922
1SN_12A.assembled 116140 41408 38556 3826417 29483 0.0077051
1HL_28A.assembled 104605 38307 35400 3494480 29496 0.0084407
1NF_32A.assembled 115584 42091 39211 3910432 29914 0.0076498
1NF_4A.assembled 110918 42309 39302 3937693 30501 0.0077459
1HL_19A.assembled 111346 38543 35529 3516253 28770 0.008182
1SN_14A.assembled 110527 39960 37012 3672750 28933 0.0078777
1SN_3A.assembled 123243 30267 27902 2757712 22382 0.0081161
1SN_25A.assembled 120465 34264 31627 3121698 24600 0.0078803
1SN_4A.assembled 111639 40461 37537 3700940 28499 0.0077005
1NF_31A.assembled 109056 40135 37206 3699649 29395 0.0079453
1SN_18A.assembled 113397 37806 34886 3437679 26836 0.0078064
1NF_24A.assembled 109692 36482 33637 3323227 26392 0.0079417
1SN_6A.assembled 107796 39568 36696 3633773 28379 0.0078098
1NF_17A.assembled 108371 36993 34269 3396706 26808 0.0078924
1HL_26A.assembled 109215 36539 33940 3355241 27093 0.0080748
1NF_9A.assembled 103833 38701 35800 3566248 27750 0.0077813
1SN_22A.assembled 107304 37128 34313 3381298 27082 0.0080094
1SN_7A.assembled 107945 37270 34523 3405066 26766 0.0078606
1SN_31A.assembled 103367 37462 34522 3408204 27267 0.0080004
1NF_2A.assembled 102160 39089 36185 3613096 27291 0.0075534
1SN_2A.assembled 106745 36280 33499 3295202 26624 0.0080796
1NF_29A.assembled 97131 38665 35705 3572818 28739 0.0080438
1HL_9A.assembled 113633 33903 31474 3108523 24533 0.0078922
1SN_17A.assembled 106838 36925 34316 3397345 26326 0.007749
1SN_32A.assembled 117048 30430 28152 2765465 22416 0.0081057
1NF_22A.assembled 108890 35268 32593 3205990 25077 0.0078219
1SN_13A.assembled 106500 36679 34119 3377673 26667 0.0078951
1SN_8A.assembled 102124 39277 36419 3606017 27214 0.0075468
1NF_26A.assembled 105799 37307 34584 3417125 27144 0.0079435
1HL_23A.assembled 94229 39679 36679 3670297 28098 0.0076555
1HL_27A.assembled 99679 38496 35515 3506773 27942 0.007968
1HL_3A.assembled 113436 29668 27610 2706497 21429 0.0079176
1HL_7A.assembled 100958 34603 32043 3133581 25103 0.008011
1SN_10A.assembled 107961 32943 30498 2992413 23627 0.0078956
1SN_11A.assembled 104454 33399 30952 3032091 23582 0.0077775
1NF_16A.assembled 101457 33028 30579 3023331 24401 0.0080709
1SN_26A.assembled 100318 33495 31062 3054944 24463 0.0080077
1SN_28A.assembled 96449 33988 31452 3103839 24776 0.0079824
1SN_20A.assembled 110647 28100 26023 2546672 20322 0.0079798
1HL_21A.assembled 104256 30845 28517 2804488 22863 0.0081523
1SN_1A.assembled 95372 34524 31959 3175357 25129 0.0079138
1NF_5A.assembled 97753 34686 32182 3206890 24794 0.0077315
1HL_29A.assembled 93081 34994 32305 3196604 25791 0.0080682
1NF_14A.assembled 99152 32144 29593 2920320 23617 0.0080871
1NF_8A.assembled 98509 33312 30789 3052509 23625 0.0077395
1HL_20A.assembled 96673 32611 30108 2953428 23741 0.0080385
1SN_15A.assembled 97901 32116 29737 2929488 22906 0.0078191
1HL_2A.assembled 93559 34198 31675 3106805 24414 0.0078582
1HL_35A.assembled 95108 32745 30314 2981191 24792 0.0083161
1SN_27A.assembled 96854 30880 28482 2808518 22876 0.0081452
1SN_16A.assembled 96334 32102 29553 2900317 22978 0.0079226
1NF_7A.assembled 96247 32132 29763 2948720 23567 0.0079923
1HL_34A.assembled 90703 34004 31368 3103956 24993 0.008052
1HL_25A.assembled 89569 34944 32390 3192139 25739 0.0080632
1HL_4A.assembled 93231 33120 30738 3033354 23946 0.0078942
1NF_30A.assembled 92056 33360 30959 3056959 24605 0.0080488
1HL_31A.assembled 95291 31165 28789 2817102 23125 0.0082088
1NF_33A.assembled 91154 33841 31504 3106099 24494 0.0078858
1NF_6A.assembled 94940 30921 28724 2843186 22585 0.0079436
1NF_15A.assembled 93177 32266 29930 2959421 22973 0.0077627
1HL_33A.assembled 90312 31968 29601 2908044 23843 0.008199
1NF_20A.assembled 91488 31865 29615 2925249 23684 0.0080964
1HL_24A.assembled 88576 30943 28613 2823744 22660 0.0080248
1NF_28A.assembled 92468 30327 28091 2782839 21855 0.0078535
1HL_8A.assembled 87336 30088 27767 2756324 22606 0.0082015
1HL_22A.assembled 92252 27801 25710 2521484 20380 0.0080825
1NF_27A.assembled 90432 29783 27582 2715903 21908 0.0080666
1HL_1A.assembled 89466 29467 27230 2658920 21428 0.0080589
1NF_11A.assembled 88047 28538 26386 2609746 21160 0.0081081
1NF_10A.assembled 90811 27220 25173 2485274 19734 0.0079404
1HL_5A.assembled 87447 27026 25137 2467239 18819 0.0076276
1NF_18A.assembled 87277 24553 22760 2237750 18717 0.0083642
1SN_29A.assembled 88849 23673 21743 2168624 17003 0.0078405
1NF_12A.assembled 83679 28518 26486 2601756 20544 0.0078962
1NF_25A.assembled 83423 27750 25672 2533230 20930 0.0082622
1HL_16A.assembled 1494 791 736 73247 611 0.0083416
1HL_6A.assembled 85254 26998 24895 2465456 20399 0.0082739
1HL_12A.assembled 1492 747 689 69158 509 0.00736
1HL_15A.assembled 1495 759 702 69999 542 0.007743
1HL_17A.assembled 2990 1793 1631 162299 1403 0.0086445
1HL_13A.assembled 497 266 247 24456 188 0.0076873
1HL_10A.assembled 997 528 489 48543 414 0.0085285
1SN_24A.assembled 83024 24881 22983 2252342 18213 0.0080862
1HL_14A.assembled 6986 3553 3301 328955 2940 0.0089374
1NF_19A.assembled 75939 24872 22971 2267086 18505 0.0081625
1NF_1A.assembled 62149 18708 17365 1733112 14104 0.008138
## nloci = number of loci
## f1loci = number of loci with >N depth coverage
## f2loci = number of loci with >N depth and passed paralog filter
## nsites = number of sites across f loci
## npoly = number of polymorphic sites in nsites
## poly = frequency of polymorphic sites
Cluster across samples¶
In [4]:
%%bash
time pyrad -p /data/analysis/params.txt -s 6
vsearch v1.11.1_linux_x86_64, 14.7GB RAM, 8 cores https://github.com/torognes/vsearch finished clustering
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
step 6: clustering across 96 samples at '.88' similarity
Reading file /data/analysis/clust.88/cat.haplos_ 100%
291106371 nt in 2830448 seqs, min 32, max 234, avg 103
Counting unique k-mers 100%
Clustering 100%
Sorting clusters 100%
Writing clusters 100%
Clusters: 125434 Size min 1, max 2641, avg 22.6
Singletons: 25519, 0.9% of seqs, 20.3% of clusters
real 38m13.548s
user 60m4.549s
sys 0m14.356s
Assemble final data set¶
In [5]:
%%bash
time pyrad -p /data/analysis/params.txt -s 7
ingroup 1HL_10A.assembled,1HL_11A.assembled,1HL_12A.assembled,1HL_13A.assembled,1HL_14A.assembled,1HL_15A.assembled,1HL_16A.assembled,1HL_17A.assembled,1HL_19A.assembled,1HL_1A.assembled,1HL_20A.assembled,1HL_21A.assembled,1HL_22A.assembled,1HL_23A.assembled,1HL_24A.assembled,1HL_25A.assembled,1HL_26A.assembled,1HL_27A.assembled,1HL_28A.assembled,1HL_29A.assembled,1HL_2A.assembled,1HL_31A.assembled,1HL_33A.assembled,1HL_34A.assembled,1HL_35A.assembled,1HL_3A.assembled,1HL_4A.assembled,1HL_5A.assembled,1HL_6A.assembled,1HL_7A.assembled,1HL_8A.assembled,1HL_9A.assembled,1NF_10A.assembled,1NF_11A.assembled,1NF_12A.assembled,1NF_13A.assembled,1NF_14A.assembled,1NF_15A.assembled,1NF_16A.assembled,1NF_17A.assembled,1NF_18A.assembled,1NF_19A.assembled,1NF_1A.assembled,1NF_20A.assembled,1NF_21A.assembled,1NF_22A.assembled,1NF_23A.assembled,1NF_24A.assembled,1NF_25A.assembled,1NF_26A.assembled,1NF_27A.assembled,1NF_28A.assembled,1NF_29A.assembled,1NF_2A.assembled,1NF_30A.assembled,1NF_31A.assembled,1NF_32A.assembled,1NF_33A.assembled,1NF_4A.assembled,1NF_5A.assembled,1NF_6A.assembled,1NF_7A.assembled,1NF_8A.assembled,1NF_9A.assembled,1SN_10A.assembled,1SN_11A.assembled,1SN_12A.assembled,1SN_13A.assembled,1SN_14A.assembled,1SN_15A.assembled,1SN_16A.assembled,1SN_17A.assembled,1SN_18A.assembled,1SN_19A.assembled,1SN_1A.assembled,1SN_20A.assembled,1SN_21A.assembled,1SN_22A.assembled,1SN_23A.assembled,1SN_24A.assembled,1SN_25A.assembled,1SN_26A.assembled,1SN_27A.assembled,1SN_28A.assembled,1SN_29A.assembled,1SN_2A.assembled,1SN_30A.assembled,1SN_31A.assembled,1SN_32A.assembled,1SN_3A.assembled,1SN_4A.assembled,1SN_5A.assembled,1SN_6A.assembled,1SN_7A.assembled,1SN_8A.assembled,1SN_9A.assembled addon exclude final stats written to: /data/analysis/stats/oly_gbs_pyrad.stats output files being written to: /data/analysis/outfiles/ directory filtering & writing to phylip file writing nexus file Writing gphocs file + writing full SNPs file + writing unlinked bi-allelic SNPs file + writing STRUCTURE file + writing geno file + writing treemix file data set reduced for group coverage minimums 1HL ['1HL_5A.assembled', '1HL_31A.assembled', '1HL_11A.assembled', '1HL_20A.assembled', '1HL_4A.assembled', '1HL_2A.assembled', '1HL_9A.assembled', '1HL_29A.assembled', '1HL_17A.assembled', '1HL_35A.assembled', '1HL_27A.assembled', '1HL_6A.assembled', '1HL_16A.assembled', '1HL_21A.assembled', '1HL_12A.assembled', '1HL_26A.assembled', '1HL_10A.assembled', '1HL_23A.assembled', '1HL_22A.assembled', '1HL_7A.assembled', '1HL_8A.assembled', '1HL_15A.assembled', '1HL_24A.assembled', '1HL_25A.assembled', '1HL_1A.assembled', '1HL_19A.assembled', '1HL_13A.assembled', '1HL_33A.assembled', '1HL_3A.assembled', '1HL_34A.assembled', '1HL_28A.assembled', '1HL_14A.assembled'] minimum= 5 1NF ['1NF_29A.assembled', '1NF_24A.assembled', '1NF_14A.assembled', '1NF_31A.assembled', '1NF_5A.assembled', '1NF_6A.assembled', '1NF_9A.assembled', '1NF_10A.assembled', '1NF_18A.assembled', '1NF_21A.assembled', '1NF_15A.assembled', '1NF_17A.assembled', '1NF_7A.assembled', '1NF_8A.assembled', '1NF_32A.assembled', '1NF_1A.assembled', '1NF_20A.assembled', '1NF_19A.assembled', '1NF_23A.assembled', '1NF_33A.assembled', '1NF_27A.assembled', '1NF_28A.assembled', '1NF_30A.assembled', '1NF_4A.assembled', '1NF_25A.assembled', '1NF_13A.assembled', '1NF_26A.assembled', '1NF_2A.assembled', '1NF_11A.assembled', '1NF_12A.assembled', '1NF_22A.assembled', '1NF_16A.assembled'] minimum= 5 1SN ['1SN_31A.assembled', '1SN_20A.assembled', '1SN_11A.assembled', '1SN_32A.assembled', '1SN_6A.assembled', '1SN_5A.assembled', '1SN_15A.assembled', '1SN_13A.assembled', '1SN_27A.assembled', '1SN_16A.assembled', '1SN_19A.assembled', '1SN_23A.assembled', '1SN_3A.assembled', '1SN_1A.assembled', '1SN_17A.assembled', '1SN_25A.assembled', '1SN_22A.assembled', '1SN_21A.assembled', '1SN_7A.assembled', '1SN_24A.assembled', '1SN_10A.assembled', '1SN_29A.assembled', '1SN_2A.assembled', '1SN_26A.assembled', '1SN_14A.assembled', '1SN_12A.assembled', '1SN_4A.assembled', '1SN_18A.assembled', '1SN_28A.assembled', '1SN_8A.assembled', '1SN_30A.assembled', '1SN_9A.assembled'] minimum= 5 writing vcf file writing migrate-n file data set reduced for group coverage minimums 1HL ['1HL_5A.assembled', '1HL_31A.assembled', '1HL_11A.assembled', '1HL_20A.assembled', '1HL_4A.assembled', '1HL_2A.assembled', '1HL_9A.assembled', '1HL_29A.assembled', '1HL_17A.assembled', '1HL_35A.assembled', '1HL_27A.assembled', '1HL_6A.assembled', '1HL_16A.assembled', '1HL_21A.assembled', '1HL_12A.assembled', '1HL_26A.assembled', '1HL_10A.assembled', '1HL_23A.assembled', '1HL_22A.assembled', '1HL_7A.assembled', '1HL_8A.assembled', '1HL_15A.assembled', '1HL_24A.assembled', '1HL_25A.assembled', '1HL_1A.assembled', '1HL_19A.assembled', '1HL_13A.assembled', '1HL_33A.assembled', '1HL_3A.assembled', '1HL_34A.assembled', '1HL_28A.assembled', '1HL_14A.assembled'] minimum= 5 1NF ['1NF_29A.assembled', '1NF_24A.assembled', '1NF_14A.assembled', '1NF_31A.assembled', '1NF_5A.assembled', '1NF_6A.assembled', '1NF_9A.assembled', '1NF_10A.assembled', '1NF_18A.assembled', '1NF_21A.assembled', '1NF_15A.assembled', '1NF_17A.assembled', '1NF_7A.assembled', '1NF_8A.assembled', '1NF_32A.assembled', '1NF_1A.assembled', '1NF_20A.assembled', '1NF_19A.assembled', '1NF_23A.assembled', '1NF_33A.assembled', '1NF_27A.assembled', '1NF_28A.assembled', '1NF_30A.assembled', '1NF_4A.assembled', '1NF_25A.assembled', '1NF_13A.assembled', '1NF_26A.assembled', '1NF_2A.assembled', '1NF_11A.assembled', '1NF_12A.assembled', '1NF_22A.assembled', '1NF_16A.assembled'] minimum= 5 1SN ['1SN_31A.assembled', '1SN_20A.assembled', '1SN_11A.assembled', '1SN_32A.assembled', '1SN_6A.assembled', '1SN_5A.assembled', '1SN_15A.assembled', '1SN_13A.assembled', '1SN_27A.assembled', '1SN_16A.assembled', '1SN_19A.assembled', '1SN_23A.assembled', '1SN_3A.assembled', '1SN_1A.assembled', '1SN_17A.assembled', '1SN_25A.assembled', '1SN_22A.assembled', '1SN_21A.assembled', '1SN_7A.assembled', '1SN_24A.assembled', '1SN_10A.assembled', '1SN_29A.assembled', '1SN_2A.assembled', '1SN_26A.assembled', '1SN_14A.assembled', '1SN_12A.assembled', '1SN_4A.assembled', '1SN_18A.assembled', '1SN_28A.assembled', '1SN_8A.assembled', '1SN_30A.assembled', '1SN_9A.assembled'] minimum= 5
------------------------------------------------------------
pyRAD : RADseq for phylogenetics & introgression analyses
------------------------------------------------------------
Cluster input file: using
/data/analysis/clust.88/cat.clust_.gz
........Traceback (most recent call last):
File "/usr/local/bin/pyrad", line 9, in <module>
load_entry_point('pyrad==3.0.66', 'console_scripts', 'pyrad')()
File "build/bdist.linux-x86_64/egg/pyrad/pyRAD.py", line 527, in main
File "build/bdist.linux-x86_64/egg/pyrad/alignable.py", line 1013, in main
File "build/bdist.linux-x86_64/egg/pyrad/loci2mig.py", line 75, in make
IndexError: list index out of range
real 62m40.523s
user 199m9.651s
sys 18m54.962s
Well, that error is a bit disconcerting... Let's look at the results and output files.¶
Examine results¶
In [6]:
%%bash
cat /data/analysis/stats/oly_gbs_pyrad.stats
71174 ## loci with > minsp containing data 46377 ## loci with > minsp containing data & paralogs removed 46377 ## loci with > minsp containing data & paralogs removed & final filtering ## number of loci recovered in final data set for each taxon. taxon nloci 1HL_10A.assembled 175 1HL_11A.assembled 13726 1HL_12A.assembled 250 1HL_13A.assembled 89 1HL_14A.assembled 1124 1HL_15A.assembled 242 1HL_16A.assembled 253 1HL_17A.assembled 614 1HL_19A.assembled 13237 1HL_1A.assembled 9589 1HL_20A.assembled 10722 1HL_21A.assembled 10058 1HL_22A.assembled 8946 1HL_23A.assembled 13897 1HL_24A.assembled 10120 1HL_25A.assembled 11994 1HL_26A.assembled 12644 1HL_27A.assembled 13331 1HL_28A.assembled 13299 1HL_29A.assembled 11531 1HL_2A.assembled 11332 1HL_31A.assembled 10212 1HL_33A.assembled 10466 1HL_34A.assembled 11384 1HL_35A.assembled 10929 1HL_3A.assembled 9789 1HL_4A.assembled 10970 1HL_5A.assembled 8488 1HL_6A.assembled 8568 1HL_7A.assembled 11704 1HL_8A.assembled 9702 1HL_9A.assembled 11520 1NF_10A.assembled 8772 1NF_11A.assembled 9182 1NF_12A.assembled 9173 1NF_13A.assembled 15824 1NF_14A.assembled 10389 1NF_15A.assembled 10584 1NF_16A.assembled 10908 1NF_17A.assembled 12588 1NF_18A.assembled 7653 1NF_19A.assembled 7740 1NF_1A.assembled 5730 1NF_20A.assembled 10530 1NF_21A.assembled 13420 1NF_22A.assembled 11876 1NF_23A.assembled 14146 1NF_24A.assembled 12340 1NF_25A.assembled 8823 1NF_26A.assembled 12830 1NF_27A.assembled 9581 1NF_28A.assembled 9856 1NF_29A.assembled 13202 1NF_2A.assembled 13564 1NF_30A.assembled 11282 1NF_31A.assembled 13992 1NF_32A.assembled 15263 1NF_33A.assembled 11421 1NF_4A.assembled 15121 1NF_5A.assembled 11616 1NF_6A.assembled 10245 1NF_7A.assembled 10526 1NF_8A.assembled 10983 1NF_9A.assembled 13320 1SN_10A.assembled 10945 1SN_11A.assembled 11161 1SN_12A.assembled 14540 1SN_13A.assembled 12521 1SN_14A.assembled 13881 1SN_15A.assembled 10552 1SN_16A.assembled 10531 1SN_17A.assembled 12615 1SN_18A.assembled 12986 1SN_19A.assembled 12675 1SN_1A.assembled 11588 1SN_20A.assembled 9021 1SN_21A.assembled 14061 1SN_22A.assembled 12667 1SN_23A.assembled 15501 1SN_24A.assembled 7868 1SN_25A.assembled 11435 1SN_26A.assembled 11144 1SN_27A.assembled 10070 1SN_28A.assembled 11328 1SN_29A.assembled 7278 1SN_2A.assembled 12322 1SN_30A.assembled 15723 1SN_31A.assembled 12763 1SN_32A.assembled 9938 1SN_3A.assembled 9641 1SN_4A.assembled 14322 1SN_5A.assembled 13373 1SN_6A.assembled 13776 1SN_7A.assembled 12741 1SN_8A.assembled 13723 1SN_9A.assembled 13737 ## nloci = number of loci with data for exactly ntaxa ## ntotal = number of loci for which at least ntaxa have data ntaxa nloci saved ntotal 1 - 2 - - 3 - - 4 5125 * 46377 5 4040 * 41252 6 3276 * 37212 7 2709 * 33936 8 2280 * 31227 9 1864 * 28947 10 1683 * 27083 11 1501 * 25400 12 1392 * 23899 13 1164 * 22507 14 1098 * 21343 15 1078 * 20245 16 832 * 19167 17 864 * 18335 18 773 * 17471 19 731 * 16698 20 611 * 15967 21 595 * 15356 22 593 * 14761 23 495 * 14168 24 479 * 13673 25 465 * 13194 26 411 * 12729 27 398 * 12318 28 368 * 11920 29 377 * 11552 30 338 * 11175 31 321 * 10837 32 313 * 10516 33 303 * 10203 34 302 * 9900 35 260 * 9598 36 256 * 9338 37 241 * 9082 38 234 * 8841 39 211 * 8607 40 228 * 8396 41 203 * 8168 42 211 * 7965 43 189 * 7754 44 179 * 7565 45 176 * 7386 46 162 * 7210 47 195 * 7048 48 154 * 6853 49 185 * 6699 50 177 * 6514 51 146 * 6337 52 152 * 6191 53 160 * 6039 54 144 * 5879 55 141 * 5735 56 140 * 5594 57 134 * 5454 58 137 * 5320 59 128 * 5183 60 146 * 5055 61 130 * 4909 62 129 * 4779 63 132 * 4650 64 128 * 4518 65 109 * 4390 66 117 * 4281 67 123 * 4164 68 137 * 4041 69 125 * 3904 70 128 * 3779 71 155 * 3651 72 122 * 3496 73 129 * 3374 74 123 * 3245 75 123 * 3122 76 133 * 2999 77 142 * 2866 78 156 * 2724 79 178 * 2568 80 165 * 2390 81 161 * 2225 82 165 * 2064 83 169 * 1899 84 192 * 1730 85 198 * 1538 86 195 * 1340 87 239 * 1145 88 283 * 906 89 462 * 623 90 142 * 161 91 17 * 19 92 2 * 2 ## nvar = number of loci containing n variable sites (pis+autapomorphies). ## sumvar = sum of variable sites (SNPs). ## pis = number of loci containing n parsimony informative sites. ## sumpis = sum of parsimony informative sites. nvar sumvar PIS sumPIS 0 21211 0 29874 0 1 7261 7261 5694 5694 2 3835 14931 2967 11628 3 2729 23118 2097 17919 4 2015 31178 1448 23711 5 1694 39648 1109 29256 6 1371 47874 880 34536 7 1176 56106 646 39058 8 997 64082 476 42866 9 787 71165 334 45872 10 634 77505 226 48132 11 508 83093 160 49892 12 386 87725 126 51404 13 337 92106 88 52548 14 276 95970 68 53500 15 208 99090 43 54145 16 186 102066 35 54705 17 140 104446 16 54977 18 120 106606 15 55247 19 90 108316 15 55532 20 70 109716 10 55732 21 56 110892 10 55942 22 43 111838 6 56074 23 46 112896 10 56304 24 35 113736 5 56424 25 27 114411 6 56574 26 26 115087 3 56652 27 22 115681 0 56652 28 11 115989 2 56708 29 12 116337 0 56708 30 17 116847 2 56768 31 9 117126 1 56799 32 4 117254 0 56799 33 4 117386 0 56799 34 5 117556 0 56799 35 3 117661 0 56799 36 3 117769 3 56907 37 4 117917 1 56944 38 3 118031 0 56944 39 2 118109 0 56944 40 1 118149 0 56944 41 0 118149 0 56944 42 2 118233 0 56944 43 3 118362 0 56944 44 0 118362 0 56944 45 2 118452 1 56989 46 0 118452 0 56989 47 0 118452 0 56989 48 0 118452 0 56989 49 2 118550 0 56989 50 1 118600 0 56989 51 0 118600 0 56989 52 1 118652 0 56989 53 0 118652 0 56989 54 1 118706 0 56989 55 1 118761 0 56989 total var= 118761 total pis= 56989 sampled unlinked SNPs= 25166 sampled unlinked bi-allelic SNPs= 25039
I'm not sure what all of that means...¶
Output files¶
In [7]:
%%bash
ls /data/analysis/outfiles/
oly_gbs_pyrad.alleles oly_gbs_pyrad.excluded_loci oly_gbs_pyrad.gphocs oly_gbs_pyrad.loci oly_gbs_pyrad.migrate oly_gbs_pyrad.nex oly_gbs_pyrad.phy oly_gbs_pyrad.phy.partitions oly_gbs_pyrad.snps oly_gbs_pyrad.snps.geno oly_gbs_pyrad.str oly_gbs_pyrad.treemix.gz oly_gbs_pyrad.unlinked_snps oly_gbs_pyrad.usnps.geno oly_gbs_pyrad.vcf
In [ ]: