Notebook

In [1]:

echo "TODAY'S DATE:"
date
echo "------------"
echo ""
#Display operating system info
lsb_release -a
echo ""
echo "------------"
echo "HOSTNAME: "; hostname 
echo ""
echo "------------"
echo "Computer Specs:"
echo ""
lscpu
echo ""
echo "------------"
echo ""
echo "Memory Specs"
echo ""
free -mh

  File "<ipython-input-1-dcd5c9efe8d4>", line 1
    echo "TODAY'S DATE:"
                       ^
SyntaxError: invalid syntax

In [2]:

%%bash
echo "TODAY'S DATE:"
date
echo "------------"
echo ""
#Display operating system info
lsb_release -a
echo ""
echo "------------"
echo "HOSTNAME: "; hostname 
echo ""
echo "------------"
echo "Computer Specs:"
echo ""
lscpu
echo ""
echo "------------"
echo ""
echo "Memory Specs"
echo ""
free -mh

TODAY'S DATE:
Mon Jul  2 08:32:22 PDT 2018
------------

Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial

------------
HOSTNAME: 
roadrunner

------------
Computer Specs:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 26
Model name:            Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
Stepping:              5
CPU MHz:               1596.000
CPU max MHz:           2394.0000
CPU min MHz:           1596.0000
BogoMIPS:              4521.78
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm kaiser tpr_shadow vnmi flexpriority ept vpid dtherm ida

------------

Memory Specs

              total        used        free      shared  buff/cache   available
Mem:            47G        518M         14G        509M         31G         45G
Swap:           47G        4.8M         47G

No LSB modules are available.

Assembly `Olurida_v081.fa` details are here: https://github.com/RobertsLab/resources/wiki/Genomic-Resources#genome-1 ¶

Make output directories¶

In [3]:

%%bash
mkdir /home/sam/analyses/20180702_oly_repeatmasker_defaults
mkdir /home/sam/analyses/20180702_oly_repeatmasker_Cgigas
mkdir /home/sam/analyses/20180702_oly_repeatmasker_Cvirginica
mkdir /home/sam/analyses/20180702_oly_repeatmasker_Olurida

Run RepeatMasker with default setting and following options:¶

-par 15 : Use 15 CPU threads

-gff : Create GFF output file (in addition to default files)

-excln : Adjusts output table calculations to exclude sequence runs of >=25Ns. Useful for draft genome assemblies.

-1> : Send stdout to file instead of printing to notebook.

-2> : Send stderr to file instead of printing to notebook.

In [4]:

%%bash
wd="/home/sam/analyses/20180702_oly_repeatmasker_defaults"
cd "$wd"
time \
/home/shared/RepeatMasker-4.0.7/RepeatMasker \
/home/sam/data/oly_assemblies/Olurida_v081.fa \
-par 15 \
-gff \
-excln \
1> "$wd"/stdout.out \
2> "$wd"/stderr.err
sed '/^Subject:/ s/ / repeatmasker defaults JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

real	252m34.218s
user	3228m1.044s
sys	539m4.084s

Run RepeatMasker with Crassostrea gigas species setting and following options:¶

-species "crassostrea gigas" : Sets species to Crassostrea gigas

-par 15 : Use 15 CPU threads

-gff : Create GFF output file (in addition to default files)

-excln : Adjusts output table calculations to exclude sequence runs of >=25Ns. Useful for draft genome assemblies.

-1> : Send stdout to file instead of printing to notebook.

-2> : Send stderr to file instead of printing to notebook.

In [5]:

%%bash
wd="/home/sam/analyses/20180702_oly_repeatmasker_Cgigas"
cd "$wd"
time \
/home/shared/RepeatMasker-4.0.7/RepeatMasker \
/home/sam/data/oly_assemblies/Olurida_v081.fa \
-species "crassostrea gigas" \
-par 15 \
-gff \
-excln \
1> "$wd"/stdout.out \
2> "$wd"/stderr.err
sed '/^Subject:/ s/ / repeatmasker gigas JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

real	248m53.801s
user	3021m41.000s
sys	399m32.716s

Run RepeatMasker with Crassostrea virginica species setting and following options:¶

-species "crassostrea virginica" : Sets species to Crassostrea virginica

-par 15 : Use 15 CPU threads

-gff : Create GFF output file (in addition to default files)

-excln : Adjusts output table calculations to exclude sequence runs of >=25Ns. Useful for draft genome assemblies.

-1> : Send stdout to file instead of printing to notebook.

-2> : Send stderr to file instead of printing to notebook.

In [6]:

%%bash
wd="/home/sam/analyses/20180702_oly_repeatmasker_Cvirginica"
cd "$wd"
time \
/home/shared/RepeatMasker-4.0.7/RepeatMasker \
/home/sam/data/oly_assemblies/Olurida_v081.fa \
-species "crassostrea virginica" \
-par 15 \
-gff \
-excln \
1> "$wd"/stdout.out \
2> "$wd"/stderr.err
sed '/^Subject:/ s/ / repeatmasker virginica JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

real	151m26.032s
user	1605m31.120s
sys	426m29.152s

Run RepeatMasker with Ostrea lurida species setting and following options:¶

-species "ostrea lurida" : Sets species to Ostrea lurida

-par 15 : Use 15 CPU threads

-gff : Create GFF output file (in addition to default files)

-excln : Adjusts output table calculations to exclude sequence runs of >=25Ns. Useful for draft genome assemblies.

-1> : Send stdout to file instead of printing to notebook.

-2> : Send stderr to file instead of printing to notebook.

In [7]:

%%bash
wd="/home/sam/analyses/20180702_oly_repeatmasker_Olurida"
cd "$wd"
time \
/home/shared/RepeatMasker-4.0.7/RepeatMasker \
/home/sam/data/oly_assemblies/Olurida_v081.fa \
-species "ostrea lurida" \
-par 15 \
-gff \
-excln \
1> "$wd"/stdout.out \
2> "$wd"/stderr.err
sed '/^Subject:/ s/ / repeatmasker lurida JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

real	146m27.018s
user	1575m41.164s
sys	428m13.936s

Re-run RepeatMasker with Crassostrea gigas species setting and following options:¶

-species "crassostrea gigas" : Sets species to Crassostrea gigas

-par 15 : Use 15 CPU threads

-gff : Create GFF output file (in addition to default files)

-excln : Adjusts output table calculations to exclude sequence runs of >=25Ns. Useful for draft genome assemblies.

-1> : Send stdout to file instead of printing to notebook.

-2> : Send stderr to file instead of printing to notebook.

Re-ran this due to problems with output files being ovewritten by subsequent job(s). This is an irritating aspect of RepeatMasker...

In [8]:

%%bash
wd="/home/sam/analyses/20180702_oly_repeatmasker_Cgigas"
cd "$wd"
time \
/home/shared/RepeatMasker-4.0.7/RepeatMasker \
/home/sam/data/oly_assemblies/Olurida_v081.fa \
-species "crassostrea gigas" \
-par 15 \
-gff \
-excln \
1> "$wd"/stdout.out \
2> "$wd"/stderr.err
sed '/^Subject:/ s/ / repeatmasker gigas JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

real	247m19.775s
user	3003m2.852s
sys	397m16.468s

Re-run RepeatMasker with Ostrea lurida species setting and following options:¶

-species "ostrea lurida" : Sets species to Ostrea lurida

-par 15 : Use 15 CPU threads

-gff : Create GFF output file (in addition to default files)

-excln : Adjusts output table calculations to exclude sequence runs of >=25Ns. Useful for draft genome assemblies.

-1> : Send stdout to file instead of printing to notebook.

-2> : Send stderr to file instead of printing to notebook.

Re-ran this due to problems with output files being ovewritten by subsequent job(s). This is an irritating aspect of RepeatMasker...

In [9]:

%%bash
wd="/home/sam/analyses/20180702_oly_repeatmasker_Olurida"
cd "$wd"
time \
/home/shared/RepeatMasker-4.0.7/RepeatMasker \
/home/sam/data/oly_assemblies/Olurida_v081.fa \
-species "ostrea lurida" \
-par 15 \
-gff \
-excln \
1> "$wd"/stdout.out \
2> "$wd"/stderr.err
sed '/^Subject:/ s/ / repeatmasker lurida JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

real	147m27.320s
user	1578m28.308s
sys	437m29.592s

SUMMARY TABLE (default settings)¶

In [10]:

%%bash
cat /home/sam/analyses/20180702_oly_repeatmasker_defaults/Olurida_v081.fa.tbl

==================================================
file name: Olurida_v081.fa          
sequences:        159429
total length: 1140787867 bp  (1077373535 bp excl N/X-runs)
GC level:         36.58 %
bases masked:   17954347 bp ( 1.67 %)
==================================================
               number of      length   percentage
               elements*    occupied  of sequence
--------------------------------------------------
SINEs:            16599       978030 bp    0.09 %
      ALUs            1          292 bp    0.00 %
      MIRs          937        72873 bp    0.01 %

LINEs:             3279       752631 bp    0.07 %
      LINE1         172        10882 bp    0.00 %
      LINE2         646        67827 bp    0.01 %
      L3/CR1        659        60327 bp    0.01 %

LTR elements:       569       127808 bp    0.01 %
      ERVL           32         1949 bp    0.00 %
      ERVL-MaLRs     10          490 bp    0.00 %
      ERV_classI    165        17699 bp    0.00 %
      ERV_classII    26         1590 bp    0.00 %

DNA elements:      1911       161957 bp    0.02 %
     hAT-Charlie     74         4216 bp    0.00 %
     TcMar-Tigger   584        24985 bp    0.00 %

Unclassified:        78         9834 bp    0.00 %

Total interspersed repeats:  2030260 bp    0.19 %


Small RNA:         5592       409456 bp    0.04 %

Satellites:         117        21278 bp    0.00 %
Simple repeats:  270784     12935570 bp    1.20 %
Low complexity:   42130      2568284 bp    0.24 %
==================================================

* most repeats fragmented by insertions or deletions
  have been counted as one element
  Runs of >=20 X/Ns in query were excluded in % calcs


The query species was assumed to be homo sapiens  
RepeatMasker Combined Database: Dfam_Consensus-20170127, RepBase-20170127
        
run with rmblastn version 2.6.0+

SUMMARY TABLE (Crassostrea gigas)¶

In [11]:

%%bash
cat /home/sam/analyses/20180702_oly_repeatmasker_Cgigas/Olurida_v081.fa.tbl

==================================================
file name: Olurida_v081.fa          
sequences:        159429
total length: 1140787867 bp  (1077373535 bp excl N/X-runs)
GC level:         36.58 %
bases masked:  152816516 bp ( 14.18 %)
==================================================
               number of      length   percentage
               elements*    occupied  of sequence
--------------------------------------------------
Retroelements       193250     67253771 bp    6.24 %
   SINEs:             2087       284274 bp    0.03 %
   Penelope         158576     56080082 bp    5.21 %
   LINEs:           179430     61300904 bp    5.69 %
    CRE/SLACS            0            0 bp    0.00 %
     L2/CR1/Rex        675       348273 bp    0.03 %
     R1/LOA/Jockey       0            0 bp    0.00 %
     R2/R4/NeSL          7        10781 bp    0.00 %
     RTE/Bov-B        7051      1827344 bp    0.17 %
     L1/CIN4             0            0 bp    0.00 %
   LTR elements:     11733      5668593 bp    0.53 %
     BEL/Pao          1517       871288 bp    0.08 %
     Ty1/Copia          78        72481 bp    0.01 %
     Gypsy/DIRS1      9151      4445789 bp    0.41 %
       Retroviral        0            0 bp    0.00 %

DNA transposons     233691     33727339 bp    3.13 %
   hobo-Activator    17578      1886743 bp    0.18 %
   Tc1-IS630-Pogo    39184      6403235 bp    0.59 %
   En-Spm                0            0 bp    0.00 %
   MuDR-IS905            0            0 bp    0.00 %
   PiggyBac           7261      1003937 bp    0.09 %
   Tourist/Harbinger  8635       823434 bp    0.08 %
   Other (Mirage,        0            0 bp    0.00 %
    P-element, Transib)

Rolling-circles          0            0 bp    0.00 %

Unclassified:       157855     36675484 bp    3.40 %

Total interspersed repeats:   137656594 bp   12.78 %


Small RNA:             222        72690 bp    0.01 %

Satellites:           6260      1238331 bp    0.11 %
Simple repeats:     241081     11662466 bp    1.08 %
Low complexity:      38915      2347827 bp    0.22 %
==================================================

* most repeats fragmented by insertions or deletions
  have been counted as one element
  Runs of >=20 X/Ns in query were excluded in % calcs


The query species was assumed to be crassostrea gigas
RepeatMasker Combined Database: Dfam_Consensus-20170127, RepBase-20170127
        
run with rmblastn version 2.6.0+

SUMMARY TABLE (Crassostrea virginica)¶

In [12]:

%%bash
cat /home/sam/analyses/20180702_oly_repeatmasker_Cvirginica/Olurida_v081.fa.tbl

==================================================
file name: Olurida_v081.fa          
sequences:        159429
total length: 1140787867 bp  (1077373535 bp excl N/X-runs)
GC level:         36.58 %
bases masked:   36996910 bp ( 3.43 %)
==================================================
               number of      length   percentage
               elements*    occupied  of sequence
--------------------------------------------------
Retroelements        59806      9886111 bp    0.92 %
   SINEs:            59806      9886111 bp    0.92 %
   Penelope              0            0 bp    0.00 %
   LINEs:                0            0 bp    0.00 %
    CRE/SLACS            0            0 bp    0.00 %
     L2/CR1/Rex          0            0 bp    0.00 %
     R1/LOA/Jockey       0            0 bp    0.00 %
     R2/R4/NeSL          0            0 bp    0.00 %
     RTE/Bov-B           0            0 bp    0.00 %
     L1/CIN4             0            0 bp    0.00 %
   LTR elements:         0            0 bp    0.00 %
     BEL/Pao             0            0 bp    0.00 %
     Ty1/Copia           0            0 bp    0.00 %
     Gypsy/DIRS1         0            0 bp    0.00 %
       Retroviral        0            0 bp    0.00 %

DNA transposons       8720      2230426 bp    0.21 %
   hobo-Activator        0            0 bp    0.00 %
   Tc1-IS630-Pogo        0            0 bp    0.00 %
   En-Spm                0            0 bp    0.00 %
   MuDR-IS905            0            0 bp    0.00 %
   PiggyBac              0            0 bp    0.00 %
   Tourist/Harbinger     0            0 bp    0.00 %
   Other (Mirage,        0            0 bp    0.00 %
    P-element, Transib)

Rolling-circles          0            0 bp    0.00 %

Unclassified:        47005      9434652 bp    0.88 %

Total interspersed repeats:    21551189 bp    2.00 %


Small RNA:           60030      9959172 bp    0.92 %

Satellites:              8         5100 bp    0.00 %
Simple repeats:     259134     12795379 bp    1.19 %
Low complexity:      42184      2581162 bp    0.24 %
==================================================

* most repeats fragmented by insertions or deletions
  have been counted as one element
  Runs of >=20 X/Ns in query were excluded in % calcs


The query species was assumed to be crassostrea virginica
RepeatMasker Combined Database: Dfam_Consensus-20170127, RepBase-20170127
        
run with rmblastn version 2.6.0+

SUMMARY TABLE (Ostrea lurida)¶

In [13]:

%%bash
cat /home/sam/analyses/20180702_oly_repeatmasker_Olurida/Olurida_v081.fa.tbl

cat: /home/sam/analyses/20180702_oly_repeatmasker_Olurida/Olurida_v081.fa.tbl: No such file or directory

In [14]:

%%bash
cat /home/sam/analyses/20180702_oly_repeatmasker_Olurida/Olurida_v081.fa.tbl

==================================================
file name: Olurida_v081.fa          
sequences:        159429
total length: 1140787867 bp  (1077373535 bp excl N/X-runs)
GC level:         36.58 %
bases masked:   15918797 bp ( 1.48 %)
==================================================
               number of      length   percentage
               elements*    occupied  of sequence
--------------------------------------------------
Retroelements            0            0 bp    0.00 %
   SINEs:                0            0 bp    0.00 %
   Penelope              0            0 bp    0.00 %
   LINEs:                0            0 bp    0.00 %
    CRE/SLACS            0            0 bp    0.00 %
     L2/CR1/Rex          0            0 bp    0.00 %
     R1/LOA/Jockey       0            0 bp    0.00 %
     R2/R4/NeSL          0            0 bp    0.00 %
     RTE/Bov-B           0            0 bp    0.00 %
     L1/CIN4             0            0 bp    0.00 %
   LTR elements:         0            0 bp    0.00 %
     BEL/Pao             0            0 bp    0.00 %
     Ty1/Copia           0            0 bp    0.00 %
     Gypsy/DIRS1         0            0 bp    0.00 %
       Retroviral        0            0 bp    0.00 %

DNA transposons          0            0 bp    0.00 %
   hobo-Activator        0            0 bp    0.00 %
   Tc1-IS630-Pogo        0            0 bp    0.00 %
   En-Spm                0            0 bp    0.00 %
   MuDR-IS905            0            0 bp    0.00 %
   PiggyBac              0            0 bp    0.00 %
   Tourist/Harbinger     0            0 bp    0.00 %
   Other (Mirage,        0            0 bp    0.00 %
    P-element, Transib)

Rolling-circles          0            0 bp    0.00 %

Unclassified:            3          189 bp    0.00 %

Total interspersed repeats:         189 bp    0.00 %


Small RNA:             224        73061 bp    0.01 %

Satellites:              8         5100 bp    0.00 %
Simple repeats:     273098     13256460 bp    1.23 %
Low complexity:      42443      2592212 bp    0.24 %
==================================================

* most repeats fragmented by insertions or deletions
  have been counted as one element
  Runs of >=20 X/Ns in query were excluded in % calcs


The query species was assumed to be ostrea lurida 
RepeatMasker Combined Database: Dfam_Consensus-20170127, RepBase-20170127
        
run with rmblastn version 2.6.0+

Had to use sudo to copy files to Owl.

In [15]:

%%bash
ls /mnt/owl/Athaliana/20180702_oly_repeatmasker*

/mnt/owl/Athaliana/20180702_oly_repeatmasker_Cgigas:
Olurida_v081.fa.cat.gz
Olurida_v081.fa.masked
Olurida_v081.fa.out
Olurida_v081.fa.out.gff
Olurida_v081.fa.tbl
readme.txt
stderr.err
stdout.out

/mnt/owl/Athaliana/20180702_oly_repeatmasker_Cvirginica:
Olurida_v081.fa.cat.gz
Olurida_v081.fa.masked
Olurida_v081.fa.out
Olurida_v081.fa.out.gff
Olurida_v081.fa.tbl
readme.txt
stderr.err
stdout.out

/mnt/owl/Athaliana/20180702_oly_repeatmasker_defaults:
Olurida_v081.fa.cat.gz
Olurida_v081.fa.masked
Olurida_v081.fa.out
Olurida_v081.fa.out.gff
Olurida_v081.fa.tbl
readme.txt
stderr.err
stdout.out

/mnt/owl/Athaliana/20180702_oly_repeatmasker_Olurida:
Olurida_v081.fa.cat.gz
Olurida_v081.fa.masked
Olurida_v081.fa.out
Olurida_v081.fa.out.gff
Olurida_v081.fa.tbl
readme.txt
stderr.err
stdout.out

In [ ]:

Assembly Olurida_v081.fa details are here: https://github.com/RobertsLab/resources/wiki/Genomic-Resources#genome-1¶

Make output directories¶

Run RepeatMasker with default setting and following options:¶

Run RepeatMasker with Crassostrea gigas species setting and following options:¶

Run RepeatMasker with Crassostrea virginica species setting and following options:¶

Run RepeatMasker with Ostrea lurida species setting and following options:¶

Re-run RepeatMasker with Crassostrea gigas species setting and following options:¶

Re-run RepeatMasker with Ostrea lurida species setting and following options:¶

SUMMARY TABLE (default settings)¶

SUMMARY TABLE (Crassostrea gigas)¶

SUMMARY TABLE (Crassostrea virginica)¶

SUMMARY TABLE (Ostrea lurida)¶

Assembly `Olurida_v081.fa` details are here: https://github.com/RobertsLab/resources/wiki/Genomic-Resources#genome-1 ¶