Notebook

In [1]:

%%bash
echo "TODAY'S DATE:"
date
echo "------------"
echo ""
#Display operating system info
lsb_release -a
echo ""
echo "------------"
echo "HOSTNAME: "; hostname 
echo ""
echo "------------"
echo "Computer Specs:"
echo ""
lscpu
echo ""
echo "------------"
echo ""
echo "Memory Specs"
echo ""

TODAY'S DATE:
Mon Oct 22 14:15:28 PDT 2018
------------

Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.5 LTS
Release:	16.04
Codename:	xenial

------------
HOSTNAME: 
emu

------------
Computer Specs:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 26
Model name:            Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
Stepping:              5
CPU MHz:               2394.000
CPU max MHz:           2394.0000
CPU min MHz:           1596.0000
BogoMIPS:              4521.80
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm tpr_shadow vnmi flexpriority ept vpid dtherm ida

------------

Memory Specs

No LSB modules are available.

In [2]:

%%bash
free -mh

              total        used        free      shared  buff/cache   available
Mem:            47G        1.5G         28G        597M         17G         44G
Swap:           11G          0B         11G

Create working directories for both genome versions¶

In [3]:

%%bash
mkdir /home/sam/analyses/20181022_Olurida_v080_repeatmodeler
mkdir /home/sam/analyses/20181022_Olurida_v081_repeatmodeler

Create RepeatModeler databases¶

In [4]:

%%bash
# RepeatModeler path
rptm=/home/shared/RepeatModeler-open-1.0.11/

# Genome paths
Olurida_v080=/home/sam/data/genomes/oly/Olurida_v080.fa
Olurida_v081=/home/sam/data/genomes/oly/Olurida_v081.fa

# Run on v080
echo "------------------------------------------------------------------------"
echo "Begin v080 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v080_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-name Ostrea_lurida_v080 \
${Olurida_v080} \
1> stdout.out \
2> stderr.err

sed '/^Subject:/ s/ / repeatmodeler v080 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

echo "------------------------------------------------------------------------"
echo ""
echo "------------------------------------------------------------------------"
echo "Begin v081 RepeatModeler"

# Run on v081
echo "------------------------------------------------------------------------"
echo "Begin v080 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v081_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-name Ostrea_lurida_v081 \
${Olurida_v081} \
1> stdout.out \
2> stderr.err

sed '/^Subject:/ s/ / repeatmodeler v081 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

------------------------------------------------------------------------
Begin v080 RepeatModeler
------------------------------------------------------------------------

------------------------------------------------------------------------
Begin v081 RepeatModeler
------------------------------------------------------------------------
Begin v080 RepeatModeler

real	1m12.758s
user	1m4.244s
sys	0m3.320s

real	0m45.485s
user	0m37.752s
sys	0m2.672s

In [5]:

%%bash
ls /home/sam/analyses/20181022_Olurida_v080_repeatmodeler

Ostrea_lurida_v080.nhr
Ostrea_lurida_v080.nin
Ostrea_lurida_v080.nnd
Ostrea_lurida_v080.nni
Ostrea_lurida_v080.nog
Ostrea_lurida_v080.nsq
Ostrea_lurida_v080.translation
stderr.err
stdout.out

In [6]:

%%bash
ls -ltrh /home/sam/analyses/20181022_Olurida_v080_repeatmodeler

total 358M
-rw-rw-r-- 1 sam sam    0 Oct 22 14:51 stderr.err
-rw-rw-r-- 1 sam sam  14M Oct 22 14:51 Ostrea_lurida_v080.translation
-rw-rw-r-- 1 sam sam 303M Oct 22 14:52 Ostrea_lurida_v080.nsq
-rw-rw-r-- 1 sam sam 8.0M Oct 22 14:52 Ostrea_lurida_v080.nin
-rw-rw-r-- 1 sam sam  26M Oct 22 14:52 Ostrea_lurida_v080.nhr
-rw-rw-r-- 1 sam sam  22K Oct 22 14:52 Ostrea_lurida_v080.nni
-rw-rw-r-- 1 sam sam 5.4M Oct 22 14:52 Ostrea_lurida_v080.nnd
-rw-rw-r-- 1 sam sam 2.7M Oct 22 14:52 Ostrea_lurida_v080.nog
-rw-rw-r-- 1 sam sam  157 Oct 22 14:52 stdout.out

In [7]:

%%bash
tail /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/stdout.out

Building database Ostrea_lurida_v080:
  Adding /home/sam/data/genomes/oly/Olurida_v080.fa to database
Number of sequences (bp) added to database: 0 ( 0 bp )

Well, that doesn't seem right... Let's try this again. Maybe need to add -engine? Although, NCBI is already configured as default...

In [8]:

%%bash
# RepeatModeler path
rptm=/home/shared/RepeatModeler-open-1.0.11/

# Genome paths
Olurida_v080=/home/sam/data/genomes/oly/Olurida_v080.fa
Olurida_v081=/home/sam/data/genomes/oly/Olurida_v081.fa

# Run on v080
echo "------------------------------------------------------------------------"
echo "Begin v080 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v080_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-name Ostrea_lurida_v080 \
-engine ncbi \
${Olurida_v080} \
1> stdout.out \
2> stderr.err

sed '/^Subject:/ s/ / repeatmodeler v080 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

echo "------------------------------------------------------------------------"
echo ""
echo "------------------------------------------------------------------------"

# Run on v081
echo "Begin v081 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v081_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-name Ostrea_lurida_v081 \
-engine ncbi \
${Olurida_v081} \
1> stdout.out \
2> stderr.err

sed '/^Subject:/ s/ / repeatmodeler v081 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

------------------------------------------------------------------------
Begin v080 RepeatModeler
------------------------------------------------------------------------

------------------------------------------------------------------------
Begin v081 RepeatModeler

real	1m11.241s
user	1m5.240s
sys	0m3.044s

real	0m44.712s
user	0m37.732s
sys	0m2.732s

In [9]:

%%bash
ls -ltrh /home/sam/analyses/20181022_Olurida_v080_repeatmodeler
tail /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/stdout.out

total 358M
-rw-rw-r-- 1 sam sam    0 Oct 22 15:03 stderr.err
-rw-rw-r-- 1 sam sam  14M Oct 22 15:04 Ostrea_lurida_v080.translation
-rw-rw-r-- 1 sam sam 303M Oct 22 15:05 Ostrea_lurida_v080.nsq
-rw-rw-r-- 1 sam sam 8.0M Oct 22 15:05 Ostrea_lurida_v080.nin
-rw-rw-r-- 1 sam sam  26M Oct 22 15:05 Ostrea_lurida_v080.nhr
-rw-rw-r-- 1 sam sam  22K Oct 22 15:05 Ostrea_lurida_v080.nni
-rw-rw-r-- 1 sam sam 5.4M Oct 22 15:05 Ostrea_lurida_v080.nnd
-rw-rw-r-- 1 sam sam 2.7M Oct 22 15:05 Ostrea_lurida_v080.nog
-rw-rw-r-- 1 sam sam  157 Oct 22 15:05 stdout.out
Building database Ostrea_lurida_v080:
  Adding /home/sam/data/genomes/oly/Olurida_v080.fa to database
Number of sequences (bp) added to database: 0 ( 0 bp )

That didn't seem to change anything. Let's just try to run RepeatModeler and see what happens.

In [10]:

%%bash
# RepeatModeler path
rptm=/home/shared/RepeatModeler-open-1.0.11/

# Run on v080
echo "------------------------------------------------------------------------"
echo "Begin v080 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v080_repeatmodeler
time \
perl ${rptm}RepeatModeler \
-database /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/Ostrea_lurida_v080 \
-engine ncbi \
-pa 16 \
>& run.out

sed '/^Subject:/ s/ / repeatmodeler v080 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

echo "------------------------------------------------------------------------"
echo ""
echo "------------------------------------------------------------------------"

# Run on v081
echo "Begin v081 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v081_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-database /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/Ostrea_lurida_v081 \
-engine ncbi \
-pa 16 \
>& run.out

sed '/^Subject:/ s/ / repeatmodeler v081 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

------------------------------------------------------------------------
Begin v080 RepeatModeler
------------------------------------------------------------------------

------------------------------------------------------------------------
Begin v081 RepeatModeler

real	0m0.148s
user	0m0.132s
sys	0m0.008s

real	0m0.113s
user	0m0.104s
sys	0m0.008s

Re-configured RepeatModeler installation to point to entire BLAST suite of tools.¶

In [11]:

%%bash
# RepeatModeler path
rptm=/home/shared/RepeatModeler-open-1.0.11/

# Run on v080
echo "------------------------------------------------------------------------"
echo "Begin v080 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v080_repeatmodeler
time \
perl ${rptm}RepeatModeler \
-database /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/Ostrea_lurida_v080 \
-engine ncbi \
-pa 16 \
>& database_build_run.out

sed '/^Subject:/ s/ / repeatmodeler v080 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

echo "------------------------------------------------------------------------"
echo ""
echo "------------------------------------------------------------------------"

# Run on v081
echo "Begin v081 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v081_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-database /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/Ostrea_lurida_v081 \
-engine ncbi \
-pa 16 \
>& database_build_run.out

sed '/^Subject:/ s/ / repeatmodeler v081 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

Process is interrupted.

Whoops! Ran the wrong program!!

In [12]:

%%bash
# RepeatModeler path
rptm=/home/shared/RepeatModeler-open-1.0.11/

# Genome paths
Olurida_v080=/home/sam/data/genomes/oly/Olurida_v080.fa
Olurida_v081=/home/sam/data/genomes/oly/Olurida_v081.fa

# Run on v080
echo "------------------------------------------------------------------------"
echo "Begin v080 RepeatModeler database builder"
cd /home/sam/analyses/20181022_Olurida_v080_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-name Ostrea_lurida_v080 \
-engine ncbi \
${Olurida_v080} \
>& database_build_run.out

sed '/^Subject:/ s/ / repeatmodeler v080 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

echo "------------------------------------------------------------------------"
echo ""
echo "------------------------------------------------------------------------"

# Run on v081
echo "Begin v081 RepeatModeler database builder"
cd /home/sam/analyses/20181022_Olurida_v081_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-name Ostrea_lurida_v081 \
-engine ncbi \
${Olurida_v081} \
>& database_build_run.out

sed '/^Subject:/ s/ / repeatmodeler v081 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

------------------------------------------------------------------------
Begin v080 RepeatModeler database builder
------------------------------------------------------------------------

------------------------------------------------------------------------
Begin v081 RepeatModeler database builder

real	1m4.328s
user	0m59.520s
sys	0m3.116s

real	0m41.983s
user	0m36.252s
sys	0m2.716s

In [13]:

$$bash
cat /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/database_build_run.out
echo""
cat /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/database_build_run.out

  File "<ipython-input-13-8bf47baf8486>", line 1
    $$bash
    ^
SyntaxError: invalid syntax

In [14]:

%%bash
cat /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/database_build_run.out
echo""
cat /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/database_build_run.out

Building database Ostrea_lurida_v080:
  Adding /home/sam/data/genomes/oly/Olurida_v080.fa to database
Number of sequences (bp) added to database: 696946 ( 1253001795 bp )

Building database Ostrea_lurida_v080:
  Adding /home/sam/data/genomes/oly/Olurida_v080.fa to database
Number of sequences (bp) added to database: 696946 ( 1253001795 bp )

Whoops! Printed the same output files. Let's actually look at the two different output files...

In [15]:

%%bash
cat /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/database_build_run.out
echo""
cat /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/database_build_run.out

Building database Ostrea_lurida_v080:
  Adding /home/sam/data/genomes/oly/Olurida_v080.fa to database
Number of sequences (bp) added to database: 696946 ( 1253001795 bp )

Building database Ostrea_lurida_v081:
  Adding /home/sam/data/genomes/oly/Olurida_v081.fa to database
Number of sequences (bp) added to database: 159429 ( 1140787867 bp )

We're in business!

Run RepeatModeler¶

In [16]:

%%bash
# RepeatModeler path
rptm=/home/shared/RepeatModeler-open-1.0.11/

# Genome paths
Olurida_v080=/home/sam/data/genomes/oly/Olurida_v080.fa
Olurida_v081=/home/sam/data/genomes/oly/Olurida_v081.fa

# Run on v080
echo "------------------------------------------------------------------------"
echo "Begin v080 RepeatModeler database builder"
cd /home/sam/analyses/20181022_Olurida_v080_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-name Ostrea_lurida_v080 \
-engine ncbi \
${Olurida_v080} \
>& database_build_run.out

sed '/^Subject:/ s/ / repeatmodeler v080 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

echo "------------------------------------------------------------------------"
echo ""
echo "------------------------------------------------------------------------"

# Run on v081
echo "Begin v081 RepeatModeler database builder"
cd /home/sam/analyses/20181022_Olurida_v081_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-name Ostrea_lurida_v081 \
-engine ncbi \
${Olurida_v081} \
>& database_build_run.out

sed '/^Subject:/ s/ / repeatmodeler v081 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

------------------------------------------------------------------------
Begin v080 RepeatModeler database builder
------------------------------------------------------------------------

------------------------------------------------------------------------
Begin v081 RepeatModeler database builder

real	1m4.566s
user	0m59.616s
sys	0m2.968s

real	0m40.985s
user	0m36.208s
sys	0m2.732s

Yeesh, did it again! Ran the wrong program. Copy and pasting is killing me today..

In [17]:

%%bash
# RepeatModeler path
rptm=/home/shared/RepeatModeler-open-1.0.11/

# Run on v080
echo "------------------------------------------------------------------------"
echo "Begin v080 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v080_repeatmodeler
time \
perl ${rptm}RepeatModeler \
-database /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/Ostrea_lurida_v080 \
-engine ncbi \
-pa 16 \
>& run.out

sed '/^Subject:/ s/ / repeatmodeler v080 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

echo "------------------------------------------------------------------------"
echo ""
echo "------------------------------------------------------------------------"

# Run on v081
echo "Begin v081 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v081_repeatmodeler
time \
perl ${rptm}BuildDatabase \
-database /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/Ostrea_lurida_v081 \
-engine ncbi \
-pa 16 \
>& run.out

sed '/^Subject:/ s/ / repeatmodeler v081 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

------------------------------------------------------------------------
Begin v080 RepeatModeler
------------------------------------------------------------------------

------------------------------------------------------------------------
Begin v081 RepeatModeler

real	2720m32.633s
user	26367m40.240s
sys	2371m25.528s

real	0m0.138s
user	0m0.092s
sys	0m0.008s

Yeesh! Forgot to change v081 code to actually run RepeatModeler!

In [18]:

%%bash
# RepeatModeler path
rptm=/home/shared/RepeatModeler-open-1.0.11/

# Run on v081
echo "------------------------------------------------------------------------"
echo "Begin v081 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v081_repeatmodeler
time \
perl ${rptm}RepeatModeler \
-database /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/Ostrea_lurida_v081 \
-engine ncbi \
-pa 16 \
>& run.out

sed '/^Subject:/ s/ / repeatmodeler v081 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

------------------------------------------------------------------------
Begin v081 RepeatModeler

real	891m3.411s
user	9443m43.164s
sys	146m48.432s

In [19]:

%%bash
tail -n 50 /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/run.out

  - Saving elements to a file...
    - 16 elements found.
Element Gathering: 00:00:00 (hh:mm:ss) Elapsed Time
Refining family-1505 model...
  - numRounds = 4
  - Consensus Length = 234 ( orig = 234 )
  - Avg Kimura Divergence = 0.01
  - Unaligned sequences = 0 ( orig = 0 )
  Build Consensus: 0:0:2 Elapsed Time
Refinement: 00:00:04 (hh:mm:ss) Elapsed Time

Processing RECON family: 3926
  - Saving elements to a file...
    - 16 elements found.
Element Gathering: 00:00:00 (hh:mm:ss) Elapsed Time
Refining family-3926 model...
  - numRounds = 8
  - Consensus Length = 1939 ( orig = 1902 )
  - Avg Kimura Divergence = 0.00
  - Unaligned sequences = 0 ( orig = 0 )
  Build Consensus: 0:0:5 Elapsed Time
Refinement: 00:00:07 (hh:mm:ss) Elapsed Time
Family Refinement: 00:38:15 (hh:mm:ss) Elapsed Time
Round Time: 21:13:16 (hh:mm:ss) Elapsed Time

Discovery complete: 2273 families found
Classifying Repeats...
RepeatClassifier Version open-1.0.11
===============================
Search Engine = ncbi
  - Looking for Simple and Low Complexity sequences..
  - Looking for similarity to known repeat proteins..
  - Looking for similarity to known repeat consensi..
Classification Time: 01:24:58 (hh:mm:ss) Elapsed Time
Program Time: 45:20:16 (hh:mm:ss) Elapsed Time
Working directory:  /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/RM_14392.TueOct231056022018
may be deleted unless there were problems with the run.

The results have been saved to:
  /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/Ostrea_lurida_v080-families.fa  - Consensus sequences for each family identified.
  /home/sam/analyses/20181022_Olurida_v080_repeatmodeler/Ostrea_lurida_v080-families.stk - Seed alignments for each family identified.

This version of RepeatModeler can upload families directly to the
open repeat database - Dfam_consensus.  Please consider uploading your final
curated library using the RepeatModeler "util/dfamConsensusTool.pl" script
( details at http://www.repeatmasker.org/RepeatModeler/dfamConsensusTool ) or
posting your raw (uncurated) RepeatModeler results to the TE Raw Dataset
Repository ( http://www.repeatmasker.org/Dfam_consensus/#/public/repository ).

In [20]:

%%bash
tail -n 50 /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/run.out

    - 62 elements found.
Element Gathering: 00:00:00 (hh:mm:ss) Elapsed Time
Refining family-945 model...
  - numRounds = 6
  - Consensus Length = 1553 ( orig = 1544 )
  - Avg Kimura Divergence = 0.00
  - Unaligned sequences = 19 ( orig = 21 )
  Build Consensus: 0:0:6 Elapsed Time
Refinement: 00:00:07 (hh:mm:ss) Elapsed Time

Processing RECON family: 1599
  - Saving elements to a file...
    - 62 elements found.
Element Gathering: 00:00:00 (hh:mm:ss) Elapsed Time
Refining family-1599 model...
  - numRounds = 6
  - Consensus Length = 1132 ( orig = 1136 )
  - Avg Kimura Divergence = 0.00
  - Unaligned sequences = 33 ( orig = 33 )
  Build Consensus: 0:0:5 Elapsed Time
Refinement: 00:00:06 (hh:mm:ss) Elapsed Time

Processing RECON family: 2008
  - Saving elements to a file...
    - 61 elements found.
Element Gathering: 00:00:00 (hh:mm:ss) Elapsed Time
Refining family-2008 model...
  - numRounds = 6
  - Consensus Length = 303 ( orig = 301 )
  - Avg Kimura Divergence = 0.00
  - Unaligned sequences = 7 ( orig = 17 )
  Build Consensus: 0:0:3 Elapsed Time
Refinement: 00:00:04 (hh:mm:ss) Elapsed Time

Processing RECON family: 1142
  - Saving elements to a file...
    - 61 elements found.
Element Gathering: 00:00:00 (hh:mm:ss) Elapsed Time
Refining family-1142 model...
  - numRounds = 9
  - Consensus Length = 969 ( orig = 969 )
  - Avg Kimura Divergence = 0.00
  - Unaligned sequences = 16 ( orig = 23 )
  Build Consensus: 0:0:11 Elapsed Time
Refinement: 00:00:12 (hh:mm:ss) Elapsed Time

Processing RECON family: 17
  - Saving elements to a file...
FastaDB::getSubstr - Error index out of bounds! (SeqID=gi|10431, offset=6970, length=918 actualSeqLen=7719)
 at /home/shared/RepeatModeler-open-1.0.11/RepeatModeler line 2069.

Well, based on this SeqAnswers thread, it looks like the best resoution to this error is to run RepeatModeler repeatedly until it works??!!

So, here we go...

In [21]:

%%bash
# RepeatModeler path
rptm=/home/shared/RepeatModeler-open-1.0.11/

# Run on v081
echo "------------------------------------------------------------------------"
echo "Begin v081 RepeatModeler"
cd /home/sam/analyses/20181022_Olurida_v081_repeatmodeler
time \
perl ${rptm}RepeatModeler \
-database /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/Ostrea_lurida_v081 \
-engine ncbi \
-pa 16 \
>& run.out

sed '/^Subject:/ s/ / repeatmodeler v081 JOB COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

------------------------------------------------------------------------
Begin v081 RepeatModeler

real	1946m38.614s
user	20614m12.632s
sys	276m15.616s

In [22]:

%%bash
tail -n 50 /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/run.out

  - Saving elements to a file...
    - 16 elements found.
Element Gathering: 00:00:00 (hh:mm:ss) Elapsed Time
Refining family-3957 model...
  - numRounds = 6
  - Consensus Length = 1289 ( orig = 1286 )
  - Avg Kimura Divergence = 0.00
  - Unaligned sequences = 0 ( orig = 1 )
  Build Consensus: 0:0:6 Elapsed Time
Refinement: 00:00:06 (hh:mm:ss) Elapsed Time

Processing RECON family: 8930
  - Saving elements to a file...
    - 16 elements found.
Element Gathering: 00:00:00 (hh:mm:ss) Elapsed Time
Refining family-8930 model...
  - numRounds = 6
  - Consensus Length = 382 ( orig = 403 )
  - Avg Kimura Divergence = 0.01
  - Unaligned sequences = 3 ( orig = 3 )
  Build Consensus: 0:0:1 Elapsed Time
Refinement: 00:00:02 (hh:mm:ss) Elapsed Time
Family Refinement: 00:24:11 (hh:mm:ss) Elapsed Time
Round Time: 15:35:32 (hh:mm:ss) Elapsed Time

Discovery complete: 2223 families found
Classifying Repeats...
RepeatClassifier Version open-1.0.11
===============================
Search Engine = ncbi
  - Looking for Simple and Low Complexity sequences..
  - Looking for similarity to known repeat proteins..
  - Looking for similarity to known repeat consensi..
Classification Time: 01:23:17 (hh:mm:ss) Elapsed Time
Program Time: 32:26:34 (hh:mm:ss) Elapsed Time
Working directory:  /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/RM_19440.MonOct290805562018
may be deleted unless there were problems with the run.

The results have been saved to:
  /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/Ostrea_lurida_v081-families.fa  - Consensus sequences for each family identified.
  /home/sam/analyses/20181022_Olurida_v081_repeatmodeler/Ostrea_lurida_v081-families.stk - Seed alignments for each family identified.

This version of RepeatModeler can upload families directly to the
open repeat database - Dfam_consensus.  Please consider uploading your final
curated library using the RepeatModeler "util/dfamConsensusTool.pl" script
( details at http://www.repeatmasker.org/RepeatModeler/dfamConsensusTool ) or
posting your raw (uncurated) RepeatModeler results to the TE Raw Dataset
Repository ( http://www.repeatmasker.org/Dfam_consensus/#/public/repository ).

In [23]:

%%bash
cd /home/sam/analyses/
rsync --archive --relative ./20181022_Olurida_v080_repeatmodeler gannet:/volume1/web/Atumefaciens
rsync --archive --relative ./20181022_Olurida_v081_repeatmodeler gannet:/volume1/web/Atumefaciens

sed '/^Subject:/ s/ / rsync JOBS COMPLETE/' ~/.default-subject.mail | msmtp "$EMAIL"

In [ ]: