Notebook

In [1]:

%%bash
echo "TODAY'S DATE:"
date
echo "------------"
echo ""
lsb_release -a
echo ""
echo "------------"
echo "HOSTNAME: "; hostname 
echo ""
echo "------------"
echo "Computer Specs:"
echo ""
lscpu
echo ""
echo "------------"
echo ""
echo "Memory Specs"
echo ""
free -mh

TODAY'S DATE:
Thu Nov 30 14:09:12 PST 2017
------------

Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial

------------
HOSTNAME: 
emu

------------
Computer Specs:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 26
Model name:            Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
Stepping:              5
CPU MHz:               2394.000
CPU max MHz:           2394.0000
CPU min MHz:           1596.0000
BogoMIPS:              4521.80
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm tpr_shadow vnmi flexpriority ept vpid dtherm ida

------------

Memory Specs

              total        used        free      shared  buff/cache   available
Mem:            47G        7.6G         29G        575M        9.8G         38G
Swap:           11G        5.6G        6.4G

No LSB modules are available.

In [2]:

%%bash
mkdir /home/sam/analyses/20171130_oly_pbjelly

In [3]:

ls /home/sam/analyses/20171120_oly_pbjelly/

assembly/            jelly.out.qual      output.err    support/
gap_fill_status.txt  liftOverTable.json  output.out
jelly.out.fasta      mapping/            Protocol.xml

In [4]:

ls /home/sam/analyses/20171130_oly_pbjelly/

In [5]:

%%bash
cp /home/sam/analyses/20171120_oly_pbjelly/Protocol.xml /home/sam/analyses/20171130_oly_pbjelly/

In [6]:

%%bash
cat /home/sam/analyses/20171130_oly_pbjelly/Protocol.xml

<jellyProtocol>
    <reference>/home/sam/data/oly_BGI_scaffolds.fasta</reference>  
    <outputDir>/home/sam/analyses/20171120_oly_pbjelly</outputDir>
    <blasr>-minMatch 8 -minPctIdentity 70 -bestn 1 -nCandidates 20 -maxScore -500 -nproc 48 -noSplitSubreads</blasr>
    <input baseDir="/home/sam/data/">
        <job>m130619_081336_42134_c100525122550000001823081109281326_s1_p0.fastq</job>
        <job>m170211_224036_42134_c101073082550000001823236402101737_s1_X0_filtered_subreads.fastq</job>
	<job>m170301_100013_42134_c101174162550000001823269408211761_s1_p0_filtered_subreads.fastq</job>
	<job>m170301_162825_42134_c101174162550000001823269408211762_s1_p0_filtered_subreads.fastq</job>
	<job>m170301_225711_42134_c101174162550000001823269408211763_s1_p0_filtered_subreads.fastq</job>
	<job>m170308_163922_42134_c101174252550000001823269408211742_s1_p0_filtered_subreads.fastq</job>
	<job>m170308_230815_42134_c101174252550000001823269408211743_s1_p0_filtered_subreads.fastq</job>
	<job>m170315_001112_42134_c101169372550000001823273008151717_s1_p0_filtered_subreads.fastq</job>
	<job>m170315_063041_42134_c101169382550000001823273008151700_s1_p0_filtered_subreads.fastq</job>
	<job>m170315_124938_42134_c101169382550000001823273008151701_s1_p0_filtered_subreads.fastq</job>
	<job>m170315_190851_42134_c101169382550000001823273008151702_s1_p0_filtered_subreads.fastq</job>
    </input>
</jellyProtocol>

In [7]:

%%bash
sed -i 's/20171120_oly_pbjelly/20171130_oly_pbjelly/g' /home/sam/analyses/20171130_oly_pbjelly/Protocol.xml

In [8]:

%%bash
cat /home/sam/analyses/20171130_oly_pbjelly/Protocol.xml

<jellyProtocol>
    <reference>/home/sam/data/oly_BGI_scaffolds.fasta</reference>  
    <outputDir>/home/sam/analyses/20171130_oly_pbjelly</outputDir>
    <blasr>-minMatch 8 -minPctIdentity 70 -bestn 1 -nCandidates 20 -maxScore -500 -nproc 48 -noSplitSubreads</blasr>
    <input baseDir="/home/sam/data/">
        <job>m130619_081336_42134_c100525122550000001823081109281326_s1_p0.fastq</job>
        <job>m170211_224036_42134_c101073082550000001823236402101737_s1_X0_filtered_subreads.fastq</job>
	<job>m170301_100013_42134_c101174162550000001823269408211761_s1_p0_filtered_subreads.fastq</job>
	<job>m170301_162825_42134_c101174162550000001823269408211762_s1_p0_filtered_subreads.fastq</job>
	<job>m170301_225711_42134_c101174162550000001823269408211763_s1_p0_filtered_subreads.fastq</job>
	<job>m170308_163922_42134_c101174252550000001823269408211742_s1_p0_filtered_subreads.fastq</job>
	<job>m170308_230815_42134_c101174252550000001823269408211743_s1_p0_filtered_subreads.fastq</job>
	<job>m170315_001112_42134_c101169372550000001823273008151717_s1_p0_filtered_subreads.fastq</job>
	<job>m170315_063041_42134_c101169382550000001823273008151700_s1_p0_filtered_subreads.fastq</job>
	<job>m170315_124938_42134_c101169382550000001823273008151701_s1_p0_filtered_subreads.fastq</job>
	<job>m170315_190851_42134_c101169382550000001823273008151702_s1_p0_filtered_subreads.fastq</job>
    </input>
</jellyProtocol>

In [ ]:

%%bash
source /home/shared/PBSuite_15.8.24/setup.sh
time python /home/shared/PBSuite_15.8.24/bin/Jelly.py setup /home/sam/analyses/20171130_oly_pbjelly/Protocol.xml
time python /home/shared/PBSuite_15.8.24/bin/Jelly.py mapping /home/sam/analyses/20171130_oly_pbjelly/Protocol.xml
time python /home/shared/PBSuite_15.8.24/bin/Jelly.py support /home/sam/analyses/20171130_oly_pbjelly/Protocol.xml
time python /home/shared/PBSuite_15.8.24/bin/Jelly.py extraction /home/sam/analyses/20171130_oly_pbjelly/Protocol.xml
time python /home/shared/PBSuite_15.8.24/bin/Jelly.py assembly /home/sam/analyses/20171130_oly_pbjelly/Protocol.xml
time python /home/shared/PBSuite_15.8.24/bin/Jelly.py output /home/sam/analyses/20171130_oly_pbjelly/Protocol.xml

Well, Firefox crashed while the command above was running (20180103). However, PB Jelly is still running. I'll report on output once it has completed.¶

In [ ]:

%%bash
source ~/.bashrc
source ~/.bash_aliases
emailme

In [12]:

%%bash
ls -lthr /home/sam/analyses/20171130_oly_pbjelly/

total 3.4G
-rw-rw-r-- 1 sam sam 1.4K Nov 30 14:13 Protocol.xml
drwxrwxr-x 2 sam sam 4.0K Dec  2 03:40 mapping
drwxrwxr-x 2 sam sam 4.0K Dec  2 05:26 support
drwxrwxr-x 1 sam sam  26M Dec  2 15:07 assembly
-rw-rw-r-- 1 sam sam    0 Dec 31 07:17 output.out
-rw-rw-r-- 1 sam sam  18M Dec 31 11:46 gap_fill_status.txt
-rw-rw-r-- 1 sam sam 1.2G Jan  9 04:43 jelly.out.qual
-rw-rw-r-- 1 sam sam 1.2G Jan  9 04:43 jelly.out.fasta
-rw-rw-r-- 1 sam sam  91M Jan  9 04:43 liftOverTable.json
-rw-rw-r-- 1 sam sam 862M Jan  9 04:43 output.err

In [13]:

%%bash
# Count the number of ">" (i.e. scaffolds) in the output file
tr -cd \> < /home/sam/analyses/20171130_oly_pbjelly/jelly.out.fasta | wc -c

In [14]:

%%bash
time cp -pR /home/sam/analyses/20171130_oly_pbjelly/ /mnt/owl/Athaliana/
source ~/.bashrc
source ~/.bash_aliases
emailme

cp: cannot create regular file '/mnt/owl/Athaliana/20171130_oly_pbjelly/Protocol.xml': Permission denied
cp: cannot create directory '/mnt/owl/Athaliana/20171130_oly_pbjelly/mapping': Permission denied
cp: cannot create regular file '/mnt/owl/Athaliana/20171130_oly_pbjelly/output.out': Permission denied
cp: cannot create regular file '/mnt/owl/Athaliana/20171130_oly_pbjelly/output.err': Permission denied
cp: cannot create regular file '/mnt/owl/Athaliana/20171130_oly_pbjelly/gap_fill_status.txt': Permission denied
cp: cannot create regular file '/mnt/owl/Athaliana/20171130_oly_pbjelly/jelly.out.fasta': Permission denied
cp: cannot create regular file '/mnt/owl/Athaliana/20171130_oly_pbjelly/jelly.out.qual': Permission denied
cp: cannot create regular file '/mnt/owl/Athaliana/20171130_oly_pbjelly/liftOverTable.json': Permission denied
cp: cannot create directory '/mnt/owl/Athaliana/20171130_oly_pbjelly/assembly': Permission denied
cp: cannot create directory '/mnt/owl/Athaliana/20171130_oly_pbjelly/support': Permission denied
cp: preserving permissions for '/mnt/owl/Athaliana/20171130_oly_pbjelly': Operation not permitted

real	0m1.368s
user	0m0.000s
sys	0m0.000s
bash: line 4: emailme: command not found

Used sudo outside of notebook for copying.

In [16]:

%%bash
ls -ltrh /mnt/owl/Athaliana/20171130_oly_pbjelly/

total 3.4G
-rw-rw-r-- 1 sam users 1.4K Nov 30 14:13 Protocol.xml
drwxrwxr-x 2 sam sam      0 Dec  2 03:40 mapping
drwxrwxr-x 2 sam sam      0 Dec  2 05:26 support
drwxrwxr-x 1 sam sam      0 Dec  2 15:07 assembly
-rw-rw-r-- 1 sam users    0 Dec 31 07:17 output.out
-rw-rw-r-- 1 sam users  18M Dec 31 11:46 gap_fill_status.txt
-rw-rw-r-- 1 sam users 1.2G Jan  9 04:43 jelly.out.qual
-rw-rw-r-- 1 sam users 1.2G Jan  9 04:43 jelly.out.fasta
-rw-rw-r-- 1 sam users  91M Jan  9 04:43 liftOverTable.json
-rw-rw-r-- 1 sam users 862M Jan  9 04:43 output.err

Started this notebook again to run Quast.

In [1]:

!python /home/sam/software/quast-4.5/quast.py \
-t 24 \
/mnt/owl/Athaliana/20171130_oly_pbjelly/jelly.out.fasta

/home/sam/software/quast-4.5/quast.py -t 24 /mnt/owl/Athaliana/20171130_oly_pbjelly/jelly.out.fasta

Version: 4.5, 15ca3b9

System information:
  OS: Linux-4.4.0-109-generic-x86_64-with-Ubuntu-16.04-xenial (linux_64)
  Python version: 2.7.12
  CPUs number: 24

Started: 2018-01-16 09:35:09

Logging to /home/sam/gitrepos/LabDocs/jupyter_nbs/sam/quast_results/results_2018_01_16_09_35_09/quast.log

CWD: /home/sam/gitrepos/LabDocs/jupyter_nbs/sam
Main parameters: 
  Threads: 24, minimum contig length: 500, ambiguity: one, threshold for extensive misassembly size: 1000

WARNING: Can't draw plots: python-matplotlib is missing or corrupted.

Contigs:
  Pre-processing...
  /mnt/owl/Athaliana/20171130_oly_pbjelly/jelly.out.fasta ==> jelly.out

2018-01-16 09:36:13
Running Basic statistics processor...
  Contig files: 
    jelly.out
  Calculating N50 and L50...
    jelly.out, N50 = 12433, L50 = 26241, Total length = 1180563613, GC % = 36.57, # N's per 100 kbp =  6580.58
Done.

NOTICE: Genes are not predicted by default. Use --gene-finding option to enable it.

2018-01-16 09:37:14
Creating large visual summaries...
This may take a while: press Ctrl-C to skip this step..
  1 of 1: Creating Icarus viewers...
Done

2018-01-16 09:37:32
RESULTS:
  Text versions of total report are saved to /home/sam/gitrepos/LabDocs/jupyter_nbs/sam/quast_results/results_2018_01_16_09_35_09/report.txt, report.tsv, and report.tex
  Text versions of transposed total report are saved to /home/sam/gitrepos/LabDocs/jupyter_nbs/sam/quast_results/results_2018_01_16_09_35_09/transposed_report.txt, transposed_report.tsv, and transposed_report.tex
  HTML version (interactive tables and plots) saved to /home/sam/gitrepos/LabDocs/jupyter_nbs/sam/quast_results/results_2018_01_16_09_35_09/report.html
  Icarus (contig browser) is saved to /home/sam/gitrepos/LabDocs/jupyter_nbs/sam/quast_results/results_2018_01_16_09_35_09/icarus.html
  Log saved to /home/sam/gitrepos/LabDocs/jupyter_nbs/sam/quast_results/results_2018_01_16_09_35_09/quast.log

Finished: 2018-01-16 09:37:32
Elapsed time: 0:02:23.101954
NOTICEs: 1; WARNINGs: 1; non-fatal ERRORs: 0

Thank you for using QUAST!

In [2]:

%%bash
cat quast_results/results_2018_01_16_09_35_09/report.txt

All statistics are based on contigs of size >= 500 bp, unless otherwise noted (e.g., "# contigs (>= 0 bp)" and "Total length (>= 0 bp)" include all contigs).
Suggestion: assembly jelly.out contains continuous fragments of N's of length >= 10 bp. You may consider rerunning QUAST using --scaffolds (-s) option!

Assembly                    jelly.out 
# contigs (>= 0 bp)         696946    
# contigs (>= 1000 bp)      159429    
# contigs (>= 5000 bp)      68750     
# contigs (>= 10000 bp)     35320     
# contigs (>= 25000 bp)     7048      
# contigs (>= 50000 bp)     894       
Total length (>= 0 bp)      1253001795
Total length (>= 1000 bp)   1140787867
Total length (>= 5000 bp)   932263178 
Total length (>= 10000 bp)  691523275 
Total length (>= 25000 bp)  261425921 
Total length (>= 50000 bp)  57741906  
# contigs                   213264    
Largest contig              194507    
Total length                1180563613
GC (%)                      36.57     
N50                         12433     
N75                         5983      
L50                         26241     
L75                         60202     
# N's per 100 kbp           6580.58

In [3]:

%%bash
cp -pR quast_results/results_2018_01_16_09_35_09/ /mnt/owl/Athaliana/quast_results/

cp: cannot create directory '/mnt/owl/Athaliana/quast_results/results_2018_01_16_09_35_09': Permission denied

Needed to use `sudo` to cp, so did that outside of notebook.¶

In [4]:

%%bash
ls /mnt/owl/Athaliana/quast_results/results_2018_01_16_09_35_09/

basic_stats
icarus.html
icarus_viewers
quast.log
report.html
report.tex
report.tsv
report.txt
transposed_report.tex
transposed_report.tsv
transposed_report.txt

In [ ]:

Well, Firefox crashed while the command above was running (20180103). However, PB Jelly is still running. I'll report on output once it has completed.¶

Needed to use sudo to cp, so did that outside of notebook.¶

Needed to use `sudo` to cp, so did that outside of notebook.¶