INFO 2021-03-14 18:23:57 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/137.sorted.bam -O ../qc-processing/bowtie/deduplicated/137.dedup.bam -M ../qc-processing/bowtie/deduplicated/137.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:23:57.757 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:23:58 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/137.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/137.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/137.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:23:58 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:23:58 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:23:58 MarkDuplicates Start of doWork freeMemory: 67869712; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:23:58 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:23:58 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:24:01 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig6669:2,710
INFO 2021-03-14 18:24:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:03 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 2s. Last read position: Contig27126:25,668
INFO 2021-03-14 18:24:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:05 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig36557:11,314
INFO 2021-03-14 18:24:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:06 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig46967:1,169
INFO 2021-03-14 18:24:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:08 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig63991:717
INFO 2021-03-14 18:24:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:10 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig74566:11,537
INFO 2021-03-14 18:24:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:12 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,566
INFO 2021-03-14 18:24:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:13 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig203572:1,231
INFO 2021-03-14 18:24:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:14 MarkDuplicates Read 8215035 records. 0 pairs never matched.
INFO 2021-03-14 18:24:14 MarkDuplicates After buildSortedReadEndLists freeMemory: 1167565400; totalMemory: 1673527296; maxMemory: 4294967296
INFO 2021-03-14 18:24:14 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:24:15 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:24:15 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:24:15 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:24:15 MarkDuplicates After generateDuplicateIndexes freeMemory: 1615308160; totalMemory: 2748317696; maxMemory: 4294967296
INFO 2021-03-14 18:24:15 MarkDuplicates Marking 3000708 records as duplicates.
INFO 2021-03-14 18:24:15 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:24:16 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:24:29 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:24:29 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:24:29 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:24:29 MarkDuplicates Before output close freeMemory: 263547368; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:24:29 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:24:29 MarkDuplicates After output close freeMemory: 262955320; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:24:29 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.53 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/137.sorted.bam O=../qc-processing/bowtie/deduplicated/137.dedup.bam M=../qc-processing/bowtie/deduplicated/137.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:24:31 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/139.sorted.bam -O ../qc-processing/bowtie/deduplicated/139.dedup.bam -M ../qc-processing/bowtie/deduplicated/139.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:24:31.775 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:24:31 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/139.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/139.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/139.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:24:31 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:24:31 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:24:32 MarkDuplicates Start of doWork freeMemory: 72062600; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:24:32 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:24:32 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:24:35 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig35213:1,244
INFO 2021-03-14 18:24:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:37 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,809
INFO 2021-03-14 18:24:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:38 MarkDuplicates Read 2959909 records. 0 pairs never matched.
INFO 2021-03-14 18:24:39 MarkDuplicates After buildSortedReadEndLists freeMemory: 716064328; totalMemory: 1017118720; maxMemory: 4294967296
INFO 2021-03-14 18:24:39 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:24:39 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:24:39 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:24:39 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:24:40 MarkDuplicates After generateDuplicateIndexes freeMemory: 958898400; totalMemory: 2091909120; maxMemory: 4294967296
INFO 2021-03-14 18:24:40 MarkDuplicates Marking 894339 records as duplicates.
INFO 2021-03-14 18:24:40 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:24:40 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:24:46 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:24:46 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:24:46 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:24:46 MarkDuplicates Before output close freeMemory: 268296456; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:24:46 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:24:46 MarkDuplicates After output close freeMemory: 258759248; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:24:46 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.24 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/139.sorted.bam O=../qc-processing/bowtie/deduplicated/139.dedup.bam M=../qc-processing/bowtie/deduplicated/139.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:24:48 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/140.sorted.bam -O ../qc-processing/bowtie/deduplicated/140.dedup.bam -M ../qc-processing/bowtie/deduplicated/140.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:24:48.234 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:24:48 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/140.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/140.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/140.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:24:48 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:24:48 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:24:48 MarkDuplicates Start of doWork freeMemory: 72064224; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:24:48 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:24:48 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:24:51 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16788:4,765
INFO 2021-03-14 18:24:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:54 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 2s. Last read position: Contig33224:2,663
INFO 2021-03-14 18:24:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:56 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig44127:2,347
INFO 2021-03-14 18:24:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:58 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig63634:4,870
INFO 2021-03-14 18:24:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:24:59 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig86127:727
INFO 2021-03-14 18:24:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:25:01 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig138416:6,522
INFO 2021-03-14 18:25:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:25:02 MarkDuplicates Read 6554977 records. 0 pairs never matched.
INFO 2021-03-14 18:25:02 MarkDuplicates After buildSortedReadEndLists freeMemory: 1085657072; totalMemory: 1545601024; maxMemory: 4294967296
INFO 2021-03-14 18:25:02 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:25:03 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:25:03 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:25:04 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:25:04 MarkDuplicates After generateDuplicateIndexes freeMemory: 1487381824; totalMemory: 2620391424; maxMemory: 4294967296
INFO 2021-03-14 18:25:04 MarkDuplicates Marking 2530853 records as duplicates.
INFO 2021-03-14 18:25:04 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:25:05 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:25:21 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:25:21 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:25:21 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:25:21 MarkDuplicates Before output close freeMemory: 818914624; totalMemory: 2620391424; maxMemory: 4294967296
INFO 2021-03-14 18:25:21 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:25:21 MarkDuplicates After output close freeMemory: 258761072; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:25:21 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.55 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/140.sorted.bam O=../qc-processing/bowtie/deduplicated/140.dedup.bam M=../qc-processing/bowtie/deduplicated/140.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:25:24 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/141.sorted.bam -O ../qc-processing/bowtie/deduplicated/141.dedup.bam -M ../qc-processing/bowtie/deduplicated/141.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:25:24.232 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:25:24 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/141.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/141.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/141.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:25:24 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:25:24 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:25:24 MarkDuplicates Start of doWork freeMemory: 67870664; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:25:24 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:25:24 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:25:28 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 3s. Last read position: Contig38783:3,115
INFO 2021-03-14 18:25:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:25:30 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 2s. Last read position: Contig94656:10,928
INFO 2021-03-14 18:25:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:25:31 MarkDuplicates Read 2538247 records. 0 pairs never matched.
INFO 2021-03-14 18:25:31 MarkDuplicates After buildSortedReadEndLists freeMemory: 682934792; totalMemory: 968884224; maxMemory: 4294967296
INFO 2021-03-14 18:25:31 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:25:32 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:25:32 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:25:32 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:25:32 MarkDuplicates After generateDuplicateIndexes freeMemory: 910664400; totalMemory: 2043674624; maxMemory: 4294967296
INFO 2021-03-14 18:25:32 MarkDuplicates Marking 782861 records as duplicates.
INFO 2021-03-14 18:25:32 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:25:33 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:25:38 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:25:38 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:25:38 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:25:38 MarkDuplicates Before output close freeMemory: 264520856; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:25:38 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:25:38 MarkDuplicates After output close freeMemory: 262953896; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:25:38 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.24 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/141.sorted.bam O=../qc-processing/bowtie/deduplicated/141.dedup.bam M=../qc-processing/bowtie/deduplicated/141.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:25:40 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/156.sorted.bam -O ../qc-processing/bowtie/deduplicated/156.dedup.bam -M ../qc-processing/bowtie/deduplicated/156.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:25:40.346 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:25:40 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/156.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/156.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/156.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:25:40 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:25:40 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:25:40 MarkDuplicates Start of doWork freeMemory: 72062888; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:25:40 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:25:40 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:25:44 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig20210:18,604
INFO 2021-03-14 18:25:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:25:46 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 2s. Last read position: Contig36057:6,843
INFO 2021-03-14 18:25:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:25:48 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig52757:1,474
INFO 2021-03-14 18:25:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:25:50 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig71173:8,003
INFO 2021-03-14 18:25:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:25:52 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig124918:1,863
INFO 2021-03-14 18:25:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:25:53 MarkDuplicates Read 5611214 records. 0 pairs never matched.
INFO 2021-03-14 18:25:53 MarkDuplicates After buildSortedReadEndLists freeMemory: 1013168264; totalMemory: 1443889152; maxMemory: 4294967296
INFO 2021-03-14 18:25:53 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:25:53 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:25:53 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:25:54 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:25:54 MarkDuplicates After generateDuplicateIndexes freeMemory: 1385669720; totalMemory: 2518679552; maxMemory: 4294967296
INFO 2021-03-14 18:25:54 MarkDuplicates Marking 2613891 records as duplicates.
INFO 2021-03-14 18:25:54 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:25:55 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:26:05 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:26:05 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:26:05 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:26:06 MarkDuplicates Before output close freeMemory: 266095472; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:26:06 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:26:06 MarkDuplicates After output close freeMemory: 262954608; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:26:06 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.43 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/156.sorted.bam O=../qc-processing/bowtie/deduplicated/156.dedup.bam M=../qc-processing/bowtie/deduplicated/156.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:26:08 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/159.sorted.bam -O ../qc-processing/bowtie/deduplicated/159.dedup.bam -M ../qc-processing/bowtie/deduplicated/159.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:26:08.560 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:26:08 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/159.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/159.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/159.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:26:08 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:26:08 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:26:08 MarkDuplicates Start of doWork freeMemory: 72062472; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:26:08 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:26:08 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:26:12 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3018:8,594
INFO 2021-03-14 18:26:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:14 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig19438:13,623
INFO 2021-03-14 18:26:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:16 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig30555:37,097
INFO 2021-03-14 18:26:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:18 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig36767:905
INFO 2021-03-14 18:26:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:19 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig43042:1,725
INFO 2021-03-14 18:26:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:21 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig54850:2,425
INFO 2021-03-14 18:26:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:23 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig66247:2,701
INFO 2021-03-14 18:26:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:24 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig80374:5,873
INFO 2021-03-14 18:26:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:26 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig103041:521
INFO 2021-03-14 18:26:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:27 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig150704:1,077
INFO 2021-03-14 18:26:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:29 MarkDuplicates Read 10773184 records. 0 pairs never matched.
INFO 2021-03-14 18:26:29 MarkDuplicates After buildSortedReadEndLists freeMemory: 791832480; totalMemory: 1449132032; maxMemory: 4294967296
INFO 2021-03-14 18:26:29 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:26:29 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:26:29 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:26:31 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:26:31 MarkDuplicates After generateDuplicateIndexes freeMemory: 1390911680; totalMemory: 2523922432; maxMemory: 4294967296
INFO 2021-03-14 18:26:31 MarkDuplicates Marking 5421619 records as duplicates.
INFO 2021-03-14 18:26:31 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:26:32 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:26:50 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:26:50 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:26:50 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:26:50 MarkDuplicates Before output close freeMemory: 262353240; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 18:26:50 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:26:51 MarkDuplicates After output close freeMemory: 255613808; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 18:26:51 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.71 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/159.sorted.bam O=../qc-processing/bowtie/deduplicated/159.dedup.bam M=../qc-processing/bowtie/deduplicated/159.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:26:52 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/161.sorted.bam -O ../qc-processing/bowtie/deduplicated/161.dedup.bam -M ../qc-processing/bowtie/deduplicated/161.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:26:52.843 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:26:52 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/161.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/161.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/161.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:26:53 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:26:53 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:26:53 MarkDuplicates Start of doWork freeMemory: 67873536; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:26:53 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:26:53 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:26:56 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig8182:1,101
INFO 2021-03-14 18:26:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:26:58 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 2s. Last read position: Contig26402:9,052
INFO 2021-03-14 18:26:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:00 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig36682:230
INFO 2021-03-14 18:27:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:02 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig51337:2,870
INFO 2021-03-14 18:27:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:04 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68185:2,553
INFO 2021-03-14 18:27:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:06 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig90312:2,801
INFO 2021-03-14 18:27:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:07 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig137778:2,754
INFO 2021-03-14 18:27:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:09 MarkDuplicates Read 7701698 records. 0 pairs never matched.
INFO 2021-03-14 18:27:09 MarkDuplicates After buildSortedReadEndLists freeMemory: 1100531504; totalMemory: 1591738368; maxMemory: 4294967296
INFO 2021-03-14 18:27:09 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:27:09 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:27:09 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:27:10 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:27:11 MarkDuplicates After generateDuplicateIndexes freeMemory: 1533519224; totalMemory: 2666528768; maxMemory: 4294967296
INFO 2021-03-14 18:27:11 MarkDuplicates Marking 2615842 records as duplicates.
INFO 2021-03-14 18:27:11 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:27:11 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:27:25 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:27:25 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:27:25 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:27:26 MarkDuplicates Before output close freeMemory: 266714976; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:27:26 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:27:26 MarkDuplicates After output close freeMemory: 262955104; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:27:26 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.56 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/161.sorted.bam O=../qc-processing/bowtie/deduplicated/161.dedup.bam M=../qc-processing/bowtie/deduplicated/161.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:27:28 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/162.sorted.bam -O ../qc-processing/bowtie/deduplicated/162.dedup.bam -M ../qc-processing/bowtie/deduplicated/162.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:27:28.070 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:27:28 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/162.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/162.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/162.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:27:28 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:27:28 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:27:28 MarkDuplicates Start of doWork freeMemory: 72065416; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:27:28 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:27:28 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:27:32 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16595:34,145
INFO 2021-03-14 18:27:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:34 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 2s. Last read position: Contig30879:27,671
INFO 2021-03-14 18:27:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:36 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 2s. Last read position: Contig42436:7,996
INFO 2021-03-14 18:27:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:38 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig60239:5,847
INFO 2021-03-14 18:27:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:39 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig85010:16,644
INFO 2021-03-14 18:27:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:41 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig136321:1,929
INFO 2021-03-14 18:27:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:27:42 MarkDuplicates Read 6639584 records. 0 pairs never matched.
INFO 2021-03-14 18:27:42 MarkDuplicates After buildSortedReadEndLists freeMemory: 879405040; totalMemory: 1339031552; maxMemory: 4294967296
INFO 2021-03-14 18:27:42 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:27:43 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:27:43 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:27:43 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:27:44 MarkDuplicates After generateDuplicateIndexes freeMemory: 1280813424; totalMemory: 2413821952; maxMemory: 4294967296
INFO 2021-03-14 18:27:44 MarkDuplicates Marking 2507433 records as duplicates.
INFO 2021-03-14 18:27:44 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:27:44 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:27:55 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:27:55 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:27:55 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:27:55 MarkDuplicates Before output close freeMemory: 271407064; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 18:27:55 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:27:56 MarkDuplicates After output close freeMemory: 258761816; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:27:56 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.47 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/162.sorted.bam O=../qc-processing/bowtie/deduplicated/162.dedup.bam M=../qc-processing/bowtie/deduplicated/162.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:27:57 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/168.sorted.bam -O ../qc-processing/bowtie/deduplicated/168.dedup.bam -M ../qc-processing/bowtie/deduplicated/168.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:27:57.963 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:27:58 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/168.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/168.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/168.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:27:58 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:27:58 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:27:58 MarkDuplicates Start of doWork freeMemory: 72067144; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:27:58 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:27:58 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:28:01 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig32542:18,032
INFO 2021-03-14 18:28:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:03 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig57586:1,981
INFO 2021-03-14 18:28:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:04 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig105506:8,234
INFO 2021-03-14 18:28:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:05 MarkDuplicates Read 3509744 records. 0 pairs never matched.
INFO 2021-03-14 18:28:05 MarkDuplicates After buildSortedReadEndLists freeMemory: 797268464; totalMemory: 1132462080; maxMemory: 4294967296
INFO 2021-03-14 18:28:05 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:28:06 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:28:06 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:28:06 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:28:06 MarkDuplicates After generateDuplicateIndexes freeMemory: 1074242192; totalMemory: 2207252480; maxMemory: 4294967296
INFO 2021-03-14 18:28:06 MarkDuplicates Marking 1442402 records as duplicates.
INFO 2021-03-14 18:28:06 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:28:07 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:28:15 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:28:15 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:28:15 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:28:15 MarkDuplicates Before output close freeMemory: 262949616; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 18:28:15 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:28:16 MarkDuplicates After output close freeMemory: 262954480; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:28:16 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.30 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/168.sorted.bam O=../qc-processing/bowtie/deduplicated/168.dedup.bam M=../qc-processing/bowtie/deduplicated/168.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:28:18 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/169.sorted.bam -O ../qc-processing/bowtie/deduplicated/169.dedup.bam -M ../qc-processing/bowtie/deduplicated/169.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:28:18.302 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:28:18 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/169.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/169.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/169.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:28:18 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:28:18 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:28:18 MarkDuplicates Start of doWork freeMemory: 72062464; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:28:18 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:28:18 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:28:22 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig1699:19,616
INFO 2021-03-14 18:28:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:24 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig15298:857
INFO 2021-03-14 18:28:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:26 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig23200:4,720
INFO 2021-03-14 18:28:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:28 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig31405:9,632
INFO 2021-03-14 18:28:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:29 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig36057:7,071
INFO 2021-03-14 18:28:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:31 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig42435:21,752
INFO 2021-03-14 18:28:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:32 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig47503:8,697
INFO 2021-03-14 18:28:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:34 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig56693:11,741
INFO 2021-03-14 18:28:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:36 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig64789:2,789
INFO 2021-03-14 18:28:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:37 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig70334:9,786
INFO 2021-03-14 18:28:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:39 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig84562:1,585
INFO 2021-03-14 18:28:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:41 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,751
INFO 2021-03-14 18:28:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:43 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:23s. Time for last 1,000,000: 1s. Last read position: Contig130284:6,245
INFO 2021-03-14 18:28:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:44 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 1s. Last read position: Contig194722:1,063
INFO 2021-03-14 18:28:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:28:45 MarkDuplicates Read 14466237 records. 0 pairs never matched.
INFO 2021-03-14 18:28:45 MarkDuplicates After buildSortedReadEndLists freeMemory: 561202528; totalMemory: 1319108608; maxMemory: 4294967296
INFO 2021-03-14 18:28:45 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:28:46 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:28:46 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:28:47 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:28:48 MarkDuplicates After generateDuplicateIndexes freeMemory: 1260887504; totalMemory: 2393899008; maxMemory: 4294967296
INFO 2021-03-14 18:28:48 MarkDuplicates Marking 6189939 records as duplicates.
INFO 2021-03-14 18:28:48 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:28:48 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:29:14 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:29:14 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:29:14 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:29:15 MarkDuplicates Before output close freeMemory: 268520624; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:29:15 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:29:15 MarkDuplicates After output close freeMemory: 266099088; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 18:29:15 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.95 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/169.sorted.bam O=../qc-processing/bowtie/deduplicated/169.dedup.bam M=../qc-processing/bowtie/deduplicated/169.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:29:17 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/171.sorted.bam -O ../qc-processing/bowtie/deduplicated/171.dedup.bam -M ../qc-processing/bowtie/deduplicated/171.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:29:17.126 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:29:17 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/171.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/171.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/171.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:29:17 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:29:17 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:29:17 MarkDuplicates Start of doWork freeMemory: 64728408; totalMemory: 91226112; maxMemory: 4294967296
INFO 2021-03-14 18:29:17 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:29:17 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:29:21 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4035:804
INFO 2021-03-14 18:29:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:22 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig22257:13,195
INFO 2021-03-14 18:29:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:24 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig34479:1
INFO 2021-03-14 18:29:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:26 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig42072:19,471
INFO 2021-03-14 18:29:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:27 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig50647:500
INFO 2021-03-14 18:29:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:29 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig64427:9,249
INFO 2021-03-14 18:29:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:31 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig77522:6,732
INFO 2021-03-14 18:29:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:32 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig101302:2,438
INFO 2021-03-14 18:29:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:34 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig149841:1,610
INFO 2021-03-14 18:29:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:35 MarkDuplicates Read 9737759 records. 0 pairs never matched.
INFO 2021-03-14 18:29:35 MarkDuplicates After buildSortedReadEndLists freeMemory: 917614560; totalMemory: 1514143744; maxMemory: 4294967296
INFO 2021-03-14 18:29:35 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:29:36 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:29:36 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:29:37 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:29:37 MarkDuplicates After generateDuplicateIndexes freeMemory: 1455925752; totalMemory: 2588934144; maxMemory: 4294967296
INFO 2021-03-14 18:29:37 MarkDuplicates Marking 4340697 records as duplicates.
INFO 2021-03-14 18:29:37 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:29:38 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:29:52 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:29:52 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:29:52 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:29:53 MarkDuplicates Before output close freeMemory: 266096992; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:29:53 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:29:53 MarkDuplicates After output close freeMemory: 262956000; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:29:53 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.60 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/171.sorted.bam O=../qc-processing/bowtie/deduplicated/171.dedup.bam M=../qc-processing/bowtie/deduplicated/171.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:29:54 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/172.sorted.bam -O ../qc-processing/bowtie/deduplicated/172.dedup.bam -M ../qc-processing/bowtie/deduplicated/172.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:29:54.961 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:29:55 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/172.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/172.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/172.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:29:55 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:29:55 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:29:55 MarkDuplicates Start of doWork freeMemory: 72064144; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:29:55 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:29:55 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:29:58 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig17120:4,297
INFO 2021-03-14 18:29:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:29:59 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig35026:10,391
INFO 2021-03-14 18:29:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:01 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig46355:20,730
INFO 2021-03-14 18:30:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:03 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig67033:4,974
INFO 2021-03-14 18:30:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:04 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig95218:1,123
INFO 2021-03-14 18:30:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:06 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig170685:812
INFO 2021-03-14 18:30:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:06 MarkDuplicates Read 6309512 records. 0 pairs never matched.
INFO 2021-03-14 18:30:07 MarkDuplicates After buildSortedReadEndLists freeMemory: 1042239176; totalMemory: 1482686464; maxMemory: 4294967296
INFO 2021-03-14 18:30:07 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:30:07 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:30:07 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:30:08 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:30:08 MarkDuplicates After generateDuplicateIndexes freeMemory: 1424467408; totalMemory: 2557476864; maxMemory: 4294967296
INFO 2021-03-14 18:30:08 MarkDuplicates Marking 2397405 records as duplicates.
INFO 2021-03-14 18:30:08 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:30:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:30:18 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:30:18 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:30:18 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:30:18 MarkDuplicates Before output close freeMemory: 261879856; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 18:30:18 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:30:18 MarkDuplicates After output close freeMemory: 262955488; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:30:18 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.40 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/172.sorted.bam O=../qc-processing/bowtie/deduplicated/172.dedup.bam M=../qc-processing/bowtie/deduplicated/172.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:30:20 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/181.sorted.bam -O ../qc-processing/bowtie/deduplicated/181.dedup.bam -M ../qc-processing/bowtie/deduplicated/181.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:30:20.744 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:30:20 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/181.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/181.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/181.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:30:20 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:30:20 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:30:20 MarkDuplicates Start of doWork freeMemory: 75212624; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 18:30:20 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:30:20 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:30:24 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig31688:46,079
INFO 2021-03-14 18:30:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:25 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig57586:2,002
INFO 2021-03-14 18:30:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:27 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig129088:996
INFO 2021-03-14 18:30:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:28 MarkDuplicates Read 3374761 records. 0 pairs never matched.
INFO 2021-03-14 18:30:28 MarkDuplicates After buildSortedReadEndLists freeMemory: 786018224; totalMemory: 1118830592; maxMemory: 4294967296
INFO 2021-03-14 18:30:28 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:30:28 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:30:28 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:30:29 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:30:29 MarkDuplicates After generateDuplicateIndexes freeMemory: 1060613440; totalMemory: 2193620992; maxMemory: 4294967296
INFO 2021-03-14 18:30:29 MarkDuplicates Marking 1570013 records as duplicates.
INFO 2021-03-14 18:30:29 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:30:29 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:30:35 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:30:35 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:30:35 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:30:35 MarkDuplicates Before output close freeMemory: 266097968; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:30:35 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:30:35 MarkDuplicates After output close freeMemory: 266102832; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 18:30:35 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.24 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/181.sorted.bam O=../qc-processing/bowtie/deduplicated/181.dedup.bam M=../qc-processing/bowtie/deduplicated/181.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:30:37 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/183.sorted.bam -O ../qc-processing/bowtie/deduplicated/183.dedup.bam -M ../qc-processing/bowtie/deduplicated/183.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:30:37.089 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:30:37 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/183.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/183.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/183.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:30:37 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:30:37 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:30:37 MarkDuplicates Start of doWork freeMemory: 72063664; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:30:37 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:30:37 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:30:40 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2973:6,068
INFO 2021-03-14 18:30:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:42 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig18527:16,246
INFO 2021-03-14 18:30:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:44 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig26702:12,979
INFO 2021-03-14 18:30:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:46 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig35026:45,565
INFO 2021-03-14 18:30:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:47 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig41141:1,660
INFO 2021-03-14 18:30:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:49 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig51347:15,900
INFO 2021-03-14 18:30:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:51 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig61480:2,233
INFO 2021-03-14 18:30:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:53 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,891
INFO 2021-03-14 18:30:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:54 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig89720:19,337
INFO 2021-03-14 18:30:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:56 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig110929:7,647
INFO 2021-03-14 18:30:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:58 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig166726:3,649
INFO 2021-03-14 18:30:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:30:59 MarkDuplicates Read 11658116 records. 0 pairs never matched.
INFO 2021-03-14 18:30:59 MarkDuplicates After buildSortedReadEndLists freeMemory: 838149000; totalMemory: 1479540736; maxMemory: 4294967296
INFO 2021-03-14 18:30:59 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:31:00 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:31:00 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:31:01 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:31:01 MarkDuplicates After generateDuplicateIndexes freeMemory: 1421320080; totalMemory: 2554331136; maxMemory: 4294967296
INFO 2021-03-14 18:31:01 MarkDuplicates Marking 4120081 records as duplicates.
INFO 2021-03-14 18:31:01 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:31:01 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:31:22 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:31:22 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:31:22 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:31:22 MarkDuplicates Before output close freeMemory: 265153800; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:31:22 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:31:22 MarkDuplicates After output close freeMemory: 258759160; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:31:22 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.76 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/183.sorted.bam O=../qc-processing/bowtie/deduplicated/183.dedup.bam M=../qc-processing/bowtie/deduplicated/183.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:31:24 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/184.sorted.bam -O ../qc-processing/bowtie/deduplicated/184.dedup.bam -M ../qc-processing/bowtie/deduplicated/184.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:31:24.662 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:31:24 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/184.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/184.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/184.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:31:25 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:31:25 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:31:25 MarkDuplicates Start of doWork freeMemory: 72061096; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:31:25 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:31:25 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:31:28 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig25558:32,037
INFO 2021-03-14 18:31:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:31:30 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig44324:13,005
INFO 2021-03-14 18:31:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:31:31 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig69409:10,984
INFO 2021-03-14 18:31:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:31:33 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig137005:5,549
INFO 2021-03-14 18:31:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:31:34 MarkDuplicates Read 4368139 records. 0 pairs never matched.
INFO 2021-03-14 18:31:34 MarkDuplicates After buildSortedReadEndLists freeMemory: 847730504; totalMemory: 1205862400; maxMemory: 4294967296
INFO 2021-03-14 18:31:34 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:31:34 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:31:34 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:31:35 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:31:35 MarkDuplicates After generateDuplicateIndexes freeMemory: 1147639976; totalMemory: 2280652800; maxMemory: 4294967296
INFO 2021-03-14 18:31:35 MarkDuplicates Marking 1535740 records as duplicates.
INFO 2021-03-14 18:31:35 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:31:36 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:31:43 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:31:43 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:31:43 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:31:43 MarkDuplicates Before output close freeMemory: 262351576; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 18:31:43 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:31:43 MarkDuplicates After output close freeMemory: 255611392; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 18:31:43 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.32 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/184.sorted.bam O=../qc-processing/bowtie/deduplicated/184.dedup.bam M=../qc-processing/bowtie/deduplicated/184.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:31:45 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/185.sorted.bam -O ../qc-processing/bowtie/deduplicated/185.dedup.bam -M ../qc-processing/bowtie/deduplicated/185.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:31:45.612 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:31:45 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/185.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/185.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/185.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:31:45 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:31:45 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:31:45 MarkDuplicates Start of doWork freeMemory: 67870624; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:31:45 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:31:45 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:31:49 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig42436:8,000
INFO 2021-03-14 18:31:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:31:50 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig104701:4,807
INFO 2021-03-14 18:31:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:31:51 MarkDuplicates Read 2341517 records. 0 pairs never matched.
INFO 2021-03-14 18:31:51 MarkDuplicates After buildSortedReadEndLists freeMemory: 669158248; totalMemory: 951058432; maxMemory: 4294967296
INFO 2021-03-14 18:31:51 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:31:51 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:31:51 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:31:52 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:31:52 MarkDuplicates After generateDuplicateIndexes freeMemory: 892840672; totalMemory: 2025848832; maxMemory: 4294967296
INFO 2021-03-14 18:31:52 MarkDuplicates Marking 851956 records as duplicates.
INFO 2021-03-14 18:31:52 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:31:53 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:31:56 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:31:56 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:31:56 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:31:57 MarkDuplicates Before output close freeMemory: 262325008; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 18:31:57 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:31:57 MarkDuplicates After output close freeMemory: 258761944; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:31:57 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.19 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/185.sorted.bam O=../qc-processing/bowtie/deduplicated/185.dedup.bam M=../qc-processing/bowtie/deduplicated/185.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:31:58 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/291.sorted.bam -O ../qc-processing/bowtie/deduplicated/291.dedup.bam -M ../qc-processing/bowtie/deduplicated/291.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:31:58.775 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:31:58 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/291.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/291.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/291.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:31:58 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:31:58 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:31:58 MarkDuplicates Start of doWork freeMemory: 72062688; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:31:58 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:31:58 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:32:02 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4488:6,212
INFO 2021-03-14 18:32:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:03 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig20865:53,989
INFO 2021-03-14 18:32:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:05 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30614:21,755
INFO 2021-03-14 18:32:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:07 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35841:1,508
INFO 2021-03-14 18:32:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:09 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig46265:4,039
INFO 2021-03-14 18:32:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:10 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig56693:9,308
INFO 2021-03-14 18:32:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:12 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig68037:876
INFO 2021-03-14 18:32:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:14 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig73139:2,271
INFO 2021-03-14 18:32:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:15 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig96826:9,748
INFO 2021-03-14 18:32:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:17 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig116145:3,097
INFO 2021-03-14 18:32:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:19 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig192518:1,762
INFO 2021-03-14 18:32:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:19 MarkDuplicates Read 11370439 records. 0 pairs never matched.
INFO 2021-03-14 18:32:20 MarkDuplicates After buildSortedReadEndLists freeMemory: 1096868560; totalMemory: 1661992960; maxMemory: 4294967296
INFO 2021-03-14 18:32:20 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:32:20 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:32:20 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:32:21 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:32:22 MarkDuplicates After generateDuplicateIndexes freeMemory: 1603773280; totalMemory: 2736783360; maxMemory: 4294967296
INFO 2021-03-14 18:32:22 MarkDuplicates Marking 3163505 records as duplicates.
INFO 2021-03-14 18:32:22 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:32:22 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:32:42 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:32:42 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:32:42 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:32:42 MarkDuplicates Before output close freeMemory: 700206888; totalMemory: 2736783360; maxMemory: 4294967296
INFO 2021-03-14 18:32:42 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:32:42 MarkDuplicates After output close freeMemory: 262954496; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:32:42 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.73 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/291.sorted.bam O=../qc-processing/bowtie/deduplicated/291.dedup.bam M=../qc-processing/bowtie/deduplicated/291.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:32:44 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/292.sorted.bam -O ../qc-processing/bowtie/deduplicated/292.dedup.bam -M ../qc-processing/bowtie/deduplicated/292.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:32:44.365 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:32:44 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/292.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/292.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/292.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:32:44 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:32:44 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:32:44 MarkDuplicates Start of doWork freeMemory: 72061968; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:32:44 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:32:44 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:32:47 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig10923:1,513
INFO 2021-03-14 18:32:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:49 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig27088:5,109
INFO 2021-03-14 18:32:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:54 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 4s. Last read position: Contig37806:33,149
INFO 2021-03-14 18:32:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:56 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig54291:1,050
INFO 2021-03-14 18:32:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:57 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,809
INFO 2021-03-14 18:32:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:32:59 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig89862:1,748
INFO 2021-03-14 18:32:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:01 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig137778:4,144
INFO 2021-03-14 18:33:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:02 MarkDuplicates Read 7689844 records. 0 pairs never matched.
INFO 2021-03-14 18:33:02 MarkDuplicates After buildSortedReadEndLists freeMemory: 1082121376; totalMemory: 1541406720; maxMemory: 4294967296
INFO 2021-03-14 18:33:02 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:33:03 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:33:03 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:33:03 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:33:04 MarkDuplicates After generateDuplicateIndexes freeMemory: 1483185600; totalMemory: 2616197120; maxMemory: 4294967296
INFO 2021-03-14 18:33:04 MarkDuplicates Marking 2232065 records as duplicates.
INFO 2021-03-14 18:33:04 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:33:04 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:33:17 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:33:17 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:33:17 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:33:17 MarkDuplicates Before output close freeMemory: 264360216; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:33:17 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:33:17 MarkDuplicates After output close freeMemory: 255613152; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 18:33:17 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.55 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/292.sorted.bam O=../qc-processing/bowtie/deduplicated/292.dedup.bam M=../qc-processing/bowtie/deduplicated/292.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:33:19 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/293.sorted.bam -O ../qc-processing/bowtie/deduplicated/293.dedup.bam -M ../qc-processing/bowtie/deduplicated/293.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:33:19.226 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:33:19 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/293.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/293.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/293.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:33:19 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:33:19 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:33:19 MarkDuplicates Start of doWork freeMemory: 72061608; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:33:19 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:33:19 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:33:22 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15654:24,998
INFO 2021-03-14 18:33:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:24 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig27445:16,607
INFO 2021-03-14 18:33:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:25 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35655:10,905
INFO 2021-03-14 18:33:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:27 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig50479:5,876
INFO 2021-03-14 18:33:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:29 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig59409:16,403
INFO 2021-03-14 18:33:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:30 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig69234:14,487
INFO 2021-03-14 18:33:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:32 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig85279:269
INFO 2021-03-14 18:33:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:33 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig107599:1,003
INFO 2021-03-14 18:33:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:35 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig253642:35
INFO 2021-03-14 18:33:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:35 MarkDuplicates Read 9175577 records. 0 pairs never matched.
INFO 2021-03-14 18:33:35 MarkDuplicates After buildSortedReadEndLists freeMemory: 899014408; totalMemory: 1376780288; maxMemory: 4294967296
INFO 2021-03-14 18:33:35 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:33:36 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:33:36 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:33:37 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:33:37 MarkDuplicates After generateDuplicateIndexes freeMemory: 1318558800; totalMemory: 2451570688; maxMemory: 4294967296
INFO 2021-03-14 18:33:37 MarkDuplicates Marking 2446594 records as duplicates.
INFO 2021-03-14 18:33:37 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:33:37 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:33:52 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:33:52 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:33:52 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:33:52 MarkDuplicates Before output close freeMemory: 1232906080; totalMemory: 2451570688; maxMemory: 4294967296
INFO 2021-03-14 18:33:52 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:33:53 MarkDuplicates After output close freeMemory: 258758904; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:33:53 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.57 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/293.sorted.bam O=../qc-processing/bowtie/deduplicated/293.dedup.bam M=../qc-processing/bowtie/deduplicated/293.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:33:54 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/294.sorted.bam -O ../qc-processing/bowtie/deduplicated/294.dedup.bam -M ../qc-processing/bowtie/deduplicated/294.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:33:54.731 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:33:54 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/294.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/294.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/294.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:33:54 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:33:54 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:33:54 MarkDuplicates Start of doWork freeMemory: 67872448; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:33:54 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:33:54 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:33:58 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3757:12,847
INFO 2021-03-14 18:33:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:33:59 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig19330:89,455
INFO 2021-03-14 18:33:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:01 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig27897:3,122
INFO 2021-03-14 18:34:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:02 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35213:158
INFO 2021-03-14 18:34:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:04 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig42409:10,064
INFO 2021-03-14 18:34:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:06 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig54547:5,869
INFO 2021-03-14 18:34:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:07 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig65875:5,539
INFO 2021-03-14 18:34:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:09 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig73139:247
INFO 2021-03-14 18:34:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:10 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig95984:2,551
INFO 2021-03-14 18:34:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:12 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig130284:6,279
INFO 2021-03-14 18:34:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:14 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig253642:31
INFO 2021-03-14 18:34:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:14 MarkDuplicates Read 11184645 records. 0 pairs never matched.
INFO 2021-03-14 18:34:14 MarkDuplicates After buildSortedReadEndLists freeMemory: 946432888; totalMemory: 1539309568; maxMemory: 4294967296
INFO 2021-03-14 18:34:14 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:34:14 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:34:14 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:34:15 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:34:15 MarkDuplicates After generateDuplicateIndexes freeMemory: 1481090224; totalMemory: 2614099968; maxMemory: 4294967296
INFO 2021-03-14 18:34:15 MarkDuplicates Marking 3575239 records as duplicates.
INFO 2021-03-14 18:34:15 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:34:16 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:34:33 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:34:33 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:34:33 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:34:33 MarkDuplicates Before output close freeMemory: 262191128; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 18:34:33 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:34:33 MarkDuplicates After output close freeMemory: 258760352; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:34:33 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.65 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/294.sorted.bam O=../qc-processing/bowtie/deduplicated/294.dedup.bam M=../qc-processing/bowtie/deduplicated/294.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:34:35 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/295.sorted.bam -O ../qc-processing/bowtie/deduplicated/295.dedup.bam -M ../qc-processing/bowtie/deduplicated/295.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:34:35.519 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:34:35 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/295.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/295.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/295.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:34:35 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:34:35 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:34:35 MarkDuplicates Start of doWork freeMemory: 72064104; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:34:35 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:34:35 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:34:38 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2252:1,684
INFO 2021-03-14 18:34:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:40 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig15378:13,159
INFO 2021-03-14 18:34:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:42 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig21241:6,825
INFO 2021-03-14 18:34:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:43 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig27981:18,747
INFO 2021-03-14 18:34:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:45 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig33004:2,818
INFO 2021-03-14 18:34:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:47 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig38138:16,728
INFO 2021-03-14 18:34:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:48 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig46605:5,982
INFO 2021-03-14 18:34:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:50 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig56033:1,102
INFO 2021-03-14 18:34:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:52 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig58952:6,144
INFO 2021-03-14 18:34:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:53 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig68037:949
INFO 2021-03-14 18:34:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:55 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig69234:14,522
INFO 2021-03-14 18:34:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:57 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig77508:4,218
INFO 2021-03-14 18:34:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:34:59 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig91189:2,415
INFO 2021-03-14 18:34:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:00 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 1s. Last read position: Contig103849:5,674
INFO 2021-03-14 18:35:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:02 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:25s. Time for last 1,000,000: 1s. Last read position: Contig130284:6,254
INFO 2021-03-14 18:35:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:04 MarkDuplicates Read 16,000,000 records. Elapsed time: 00:00:27s. Time for last 1,000,000: 1s. Last read position: Contig198530:2,083
INFO 2021-03-14 18:35:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:05 MarkDuplicates Read 16480928 records. 0 pairs never matched.
INFO 2021-03-14 18:35:05 MarkDuplicates After buildSortedReadEndLists freeMemory: 717219280; totalMemory: 1456472064; maxMemory: 4294967296
INFO 2021-03-14 18:35:05 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:35:05 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:35:05 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:35:07 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:35:07 MarkDuplicates After generateDuplicateIndexes freeMemory: 1398252688; totalMemory: 2531262464; maxMemory: 4294967296
INFO 2021-03-14 18:35:07 MarkDuplicates Marking 4956794 records as duplicates.
INFO 2021-03-14 18:35:07 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:35:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:35:32 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:23s. Time for last 10,000,000: 23s. Last read position: Contig109675:6,392
INFO 2021-03-14 18:35:35 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:35:35 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:35:35 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:35:35 MarkDuplicates Before output close freeMemory: 2417635976; totalMemory: 2531262464; maxMemory: 4294967296
INFO 2021-03-14 18:35:35 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:35:35 MarkDuplicates After output close freeMemory: 262954968; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:35:35 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 1.01 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/295.sorted.bam O=../qc-processing/bowtie/deduplicated/295.dedup.bam M=../qc-processing/bowtie/deduplicated/295.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:35:37 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/296.sorted.bam -O ../qc-processing/bowtie/deduplicated/296.dedup.bam -M ../qc-processing/bowtie/deduplicated/296.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:35:37.723 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:35:37 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/296.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/296.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/296.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:35:38 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:35:38 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:35:38 MarkDuplicates Start of doWork freeMemory: 75209096; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 18:35:38 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:35:38 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:35:40 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig1476:13,901
INFO 2021-03-14 18:35:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:42 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig5738:10,717
INFO 2021-03-14 18:35:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:44 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig18358:12,982
INFO 2021-03-14 18:35:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:45 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig23690:10,858
INFO 2021-03-14 18:35:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:47 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig28548:12,549
INFO 2021-03-14 18:35:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:48 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig33649:6,790
INFO 2021-03-14 18:35:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:50 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig36557:11,338
INFO 2021-03-14 18:35:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:51 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig42187:43,039
INFO 2021-03-14 18:35:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:53 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig49451:13,755
INFO 2021-03-14 18:35:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:55 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,470
INFO 2021-03-14 18:35:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:56 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig63672:13,897
INFO 2021-03-14 18:35:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:58 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig69318:4,434
INFO 2021-03-14 18:35:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:35:59 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig78973:3,817
INFO 2021-03-14 18:35:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:01 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig94256:1,450
INFO 2021-03-14 18:36:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:02 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 1s. Last read position: Contig110952:843
INFO 2021-03-14 18:36:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:04 MarkDuplicates Read 16,000,000 records. Elapsed time: 00:00:25s. Time for last 1,000,000: 1s. Last read position: Contig137778:6,413
INFO 2021-03-14 18:36:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:06 MarkDuplicates Read 17,000,000 records. Elapsed time: 00:00:27s. Time for last 1,000,000: 1s. Last read position: Contig194722:1,063
INFO 2021-03-14 18:36:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:07 MarkDuplicates Read 17619461 records. 0 pairs never matched.
INFO 2021-03-14 18:36:07 MarkDuplicates After buildSortedReadEndLists freeMemory: 874315864; totalMemory: 1704984576; maxMemory: 4294967296
INFO 2021-03-14 18:36:07 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:36:07 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:36:07 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:36:09 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:36:09 MarkDuplicates After generateDuplicateIndexes freeMemory: 1646764872; totalMemory: 2779774976; maxMemory: 4294967296
INFO 2021-03-14 18:36:09 MarkDuplicates Marking 5650562 records as duplicates.
INFO 2021-03-14 18:36:09 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:36:10 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:36:31 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:21s. Time for last 10,000,000: 21s. Last read position: Contig108753:1,020
INFO 2021-03-14 18:36:35 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:36:35 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:36:35 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:36:35 MarkDuplicates Before output close freeMemory: 262950280; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 18:36:36 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:36:36 MarkDuplicates After output close freeMemory: 262955144; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:36:36 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.97 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/296.sorted.bam O=../qc-processing/bowtie/deduplicated/296.dedup.bam M=../qc-processing/bowtie/deduplicated/296.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:36:37 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/298.sorted.bam -O ../qc-processing/bowtie/deduplicated/298.dedup.bam -M ../qc-processing/bowtie/deduplicated/298.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:36:37.965 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:36:38 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/298.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/298.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/298.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:36:38 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:36:38 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:36:38 MarkDuplicates Start of doWork freeMemory: 67868808; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:36:38 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:36:38 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:36:41 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5475:1,639
INFO 2021-03-14 18:36:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:42 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig22371:1,107
INFO 2021-03-14 18:36:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:44 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig31099:15,415
INFO 2021-03-14 18:36:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:46 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig39551:6,656
INFO 2021-03-14 18:36:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:47 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig52580:8,092
INFO 2021-03-14 18:36:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:49 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig62261:3,644
INFO 2021-03-14 18:36:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:50 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig70638:19,186
INFO 2021-03-14 18:36:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:52 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig91176:138
INFO 2021-03-14 18:36:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:53 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig125409:134
INFO 2021-03-14 18:36:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:55 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig532034:3,473
INFO 2021-03-14 18:36:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:36:55 MarkDuplicates Read 10080774 records. 0 pairs never matched.
INFO 2021-03-14 18:36:55 MarkDuplicates After buildSortedReadEndLists freeMemory: 1027160808; totalMemory: 1570766848; maxMemory: 4294967296
INFO 2021-03-14 18:36:55 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:36:56 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:36:56 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:36:57 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:36:57 MarkDuplicates After generateDuplicateIndexes freeMemory: 1512547032; totalMemory: 2645557248; maxMemory: 4294967296
INFO 2021-03-14 18:36:57 MarkDuplicates Marking 2927649 records as duplicates.
INFO 2021-03-14 18:36:57 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:36:57 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:37:13 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:37:13 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:37:13 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:37:14 MarkDuplicates Before output close freeMemory: 257273256; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 18:37:14 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:37:14 MarkDuplicates After output close freeMemory: 258760480; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:37:14 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.60 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/298.sorted.bam O=../qc-processing/bowtie/deduplicated/298.dedup.bam M=../qc-processing/bowtie/deduplicated/298.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:37:15 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/299.sorted.bam -O ../qc-processing/bowtie/deduplicated/299.dedup.bam -M ../qc-processing/bowtie/deduplicated/299.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:37:15.948 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:37:16 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/299.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/299.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/299.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:37:16 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:37:16 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:37:16 MarkDuplicates Start of doWork freeMemory: 75209704; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 18:37:16 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:37:16 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:37:19 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2333:3,238
INFO 2021-03-14 18:37:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:20 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig15915:4,117
INFO 2021-03-14 18:37:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:22 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig22606:5,619
INFO 2021-03-14 18:37:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:23 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig29062:15,624
INFO 2021-03-14 18:37:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:25 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig34398:2,822
INFO 2021-03-14 18:37:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:26 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig40465:1,727
INFO 2021-03-14 18:37:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:28 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig48780:13,700
INFO 2021-03-14 18:37:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:30 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig56693:9,310
INFO 2021-03-14 18:37:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:31 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig64157:772
INFO 2021-03-14 18:37:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:33 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig69234:14,811
INFO 2021-03-14 18:37:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:34 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig80308:10,132
INFO 2021-03-14 18:37:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:36 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,706
INFO 2021-03-14 18:37:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:37 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig118978:2,609
INFO 2021-03-14 18:37:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:39 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig167362:627
INFO 2021-03-14 18:37:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:37:40 MarkDuplicates Read 14788657 records. 0 pairs never matched.
INFO 2021-03-14 18:37:41 MarkDuplicates After buildSortedReadEndLists freeMemory: 679525208; totalMemory: 1376780288; maxMemory: 4294967296
INFO 2021-03-14 18:37:41 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:37:41 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:37:41 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:37:42 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:37:42 MarkDuplicates After generateDuplicateIndexes freeMemory: 1318560656; totalMemory: 2451570688; maxMemory: 4294967296
INFO 2021-03-14 18:37:42 MarkDuplicates Marking 4313552 records as duplicates.
INFO 2021-03-14 18:37:42 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:37:43 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:38:05 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:21s. Time for last 10,000,000: 21s. Last read position: Contig182947:809
INFO 2021-03-14 18:38:06 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:38:06 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:38:06 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:38:06 MarkDuplicates Before output close freeMemory: 258389408; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 18:38:06 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:38:06 MarkDuplicates After output close freeMemory: 255613992; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 18:38:06 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.84 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/299.sorted.bam O=../qc-processing/bowtie/deduplicated/299.dedup.bam M=../qc-processing/bowtie/deduplicated/299.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:38:08 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/301.sorted.bam -O ../qc-processing/bowtie/deduplicated/301.dedup.bam -M ../qc-processing/bowtie/deduplicated/301.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:38:08.179 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:38:08 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/301.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/301.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/301.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:38:08 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:38:08 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:38:08 MarkDuplicates Start of doWork freeMemory: 67869032; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:38:08 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:38:08 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:38:12 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3692:6,055
INFO 2021-03-14 18:38:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:13 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig20021:7,076
INFO 2021-03-14 18:38:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:15 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig28866:13,366
INFO 2021-03-14 18:38:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:17 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35213:2,558
INFO 2021-03-14 18:38:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:18 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig44579:3,367
INFO 2021-03-14 18:38:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:20 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,489
INFO 2021-03-14 18:38:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:21 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig67291:5,964
INFO 2021-03-14 18:38:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:23 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig73139:79
INFO 2021-03-14 18:38:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:24 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig95735:1,519
INFO 2021-03-14 18:38:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:26 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig126307:10,748
INFO 2021-03-14 18:38:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:28 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig276216:4,928
INFO 2021-03-14 18:38:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:28 MarkDuplicates Read 11189722 records. 0 pairs never matched.
INFO 2021-03-14 18:38:28 MarkDuplicates After buildSortedReadEndLists freeMemory: 1145499856; totalMemory: 1724907520; maxMemory: 4294967296
INFO 2021-03-14 18:38:28 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:38:29 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:38:29 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:38:29 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:38:30 MarkDuplicates After generateDuplicateIndexes freeMemory: 1666688016; totalMemory: 2799697920; maxMemory: 4294967296
INFO 2021-03-14 18:38:30 MarkDuplicates Marking 3251158 records as duplicates.
INFO 2021-03-14 18:38:30 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:38:30 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:38:48 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:38:48 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:38:48 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:38:48 MarkDuplicates Before output close freeMemory: 1609707920; totalMemory: 2799697920; maxMemory: 4294967296
INFO 2021-03-14 18:38:48 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:38:48 MarkDuplicates After output close freeMemory: 262954752; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:38:48 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.68 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/301.sorted.bam O=../qc-processing/bowtie/deduplicated/301.dedup.bam M=../qc-processing/bowtie/deduplicated/301.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:38:50 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/302.sorted.bam -O ../qc-processing/bowtie/deduplicated/302.dedup.bam -M ../qc-processing/bowtie/deduplicated/302.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:38:50.737 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:38:50 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/302.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/302.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/302.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:38:50 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:38:50 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:38:50 MarkDuplicates Start of doWork freeMemory: 75209040; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 18:38:50 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:38:50 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:38:53 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig1705:12,981
INFO 2021-03-14 18:38:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:55 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig7126:1,579
INFO 2021-03-14 18:38:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:57 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig19570:5,110
INFO 2021-03-14 18:38:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:38:58 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig25482:7,996
INFO 2021-03-14 18:38:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:00 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig31246:2,376
INFO 2021-03-14 18:39:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:01 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig35290:25,932
INFO 2021-03-14 18:39:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:03 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig41518:3,123
INFO 2021-03-14 18:39:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:05 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig49076:2,430
INFO 2021-03-14 18:39:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:06 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig56693:12,118
INFO 2021-03-14 18:39:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:08 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig65858:13,810
INFO 2021-03-14 18:39:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:09 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig72420:2,499
INFO 2021-03-14 18:39:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:11 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig85537:4,684
INFO 2021-03-14 18:39:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:12 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig104309:3,902
INFO 2021-03-14 18:39:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:14 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig134879:2,027
INFO 2021-03-14 18:39:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:16 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 1s. Last read position: Contig189043:2,137
INFO 2021-03-14 18:39:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:17 MarkDuplicates Read 15628981 records. 0 pairs never matched.
INFO 2021-03-14 18:39:17 MarkDuplicates After buildSortedReadEndLists freeMemory: 791553144; totalMemory: 1536163840; maxMemory: 4294967296
INFO 2021-03-14 18:39:17 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:39:17 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:39:17 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:39:18 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:39:19 MarkDuplicates After generateDuplicateIndexes freeMemory: 1477944792; totalMemory: 2610954240; maxMemory: 4294967296
INFO 2021-03-14 18:39:19 MarkDuplicates Marking 4512368 records as duplicates.
INFO 2021-03-14 18:39:19 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:39:19 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:39:41 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:21s. Time for last 10,000,000: 21s. Last read position: Contig137778:6,941
INFO 2021-03-14 18:39:44 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:39:44 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:39:44 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:39:44 MarkDuplicates Before output close freeMemory: 266785776; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:39:44 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:39:44 MarkDuplicates After output close freeMemory: 258761176; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:39:44 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.90 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/302.sorted.bam O=../qc-processing/bowtie/deduplicated/302.dedup.bam M=../qc-processing/bowtie/deduplicated/302.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:39:46 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/303.sorted.bam -O ../qc-processing/bowtie/deduplicated/303.dedup.bam -M ../qc-processing/bowtie/deduplicated/303.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:39:46.604 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:39:46 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/303.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/303.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/303.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:39:46 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:39:46 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:39:46 MarkDuplicates Start of doWork freeMemory: 67869912; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:39:46 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:39:46 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:39:49 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3667:3,773
INFO 2021-03-14 18:39:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:51 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig19570:3,030
INFO 2021-03-14 18:39:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:53 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig27893:7,742
INFO 2021-03-14 18:39:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:54 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig34928:5,187
INFO 2021-03-14 18:39:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:56 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig42678:1,093
INFO 2021-03-14 18:39:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:58 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig54567:4,674
INFO 2021-03-14 18:39:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:39:59 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig64598:10,514
INFO 2021-03-14 18:39:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:01 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig73139:296
INFO 2021-03-14 18:40:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:02 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig97441:30
INFO 2021-03-14 18:40:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:04 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig132450:8,963
INFO 2021-03-14 18:40:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:06 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig341658:662
INFO 2021-03-14 18:40:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:06 MarkDuplicates Read 11158976 records. 0 pairs never matched.
INFO 2021-03-14 18:40:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 1105603600; totalMemory: 1716518912; maxMemory: 4294967296
INFO 2021-03-14 18:40:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:40:07 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:40:07 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:40:07 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:40:08 MarkDuplicates After generateDuplicateIndexes freeMemory: 1658300312; totalMemory: 2791309312; maxMemory: 4294967296
INFO 2021-03-14 18:40:08 MarkDuplicates Marking 3496227 records as duplicates.
INFO 2021-03-14 18:40:08 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:40:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:40:26 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:40:26 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:40:26 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:40:26 MarkDuplicates Before output close freeMemory: 267477520; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:40:26 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:40:26 MarkDuplicates After output close freeMemory: 262955632; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:40:26 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.67 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/303.sorted.bam O=../qc-processing/bowtie/deduplicated/303.dedup.bam M=../qc-processing/bowtie/deduplicated/303.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:40:28 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/304.sorted.bam -O ../qc-processing/bowtie/deduplicated/304.dedup.bam -M ../qc-processing/bowtie/deduplicated/304.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:40:28.467 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:40:28 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/304.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/304.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/304.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:40:28 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:40:28 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:40:28 MarkDuplicates Start of doWork freeMemory: 67872272; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:40:28 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:40:28 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:40:31 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3005:21,657
INFO 2021-03-14 18:40:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:33 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig17943:1,943
INFO 2021-03-14 18:40:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:35 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig25971:16,015
INFO 2021-03-14 18:40:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:36 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig32753:1,496
INFO 2021-03-14 18:40:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:38 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig39210:1,639
INFO 2021-03-14 18:40:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:39 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig48706:8,275
INFO 2021-03-14 18:40:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:41 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig57512:454
INFO 2021-03-14 18:40:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:42 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,780
INFO 2021-03-14 18:40:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:44 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig74622:3,659
INFO 2021-03-14 18:40:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:45 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig96933:2,162
INFO 2021-03-14 18:40:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:47 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig122955:4,964
INFO 2021-03-14 18:40:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:49 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig194710:698
INFO 2021-03-14 18:40:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:40:49 MarkDuplicates Read 12431900 records. 0 pairs never matched.
INFO 2021-03-14 18:40:49 MarkDuplicates After buildSortedReadEndLists freeMemory: 839092400; totalMemory: 1485832192; maxMemory: 4294967296
INFO 2021-03-14 18:40:49 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:40:50 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:40:50 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:40:51 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:40:51 MarkDuplicates After generateDuplicateIndexes freeMemory: 1427612944; totalMemory: 2560622592; maxMemory: 4294967296
INFO 2021-03-14 18:40:51 MarkDuplicates Marking 3899163 records as duplicates.
INFO 2021-03-14 18:40:51 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:40:52 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:41:10 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:41:10 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:41:10 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:41:11 MarkDuplicates Before output close freeMemory: 271266616; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 18:41:11 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:41:11 MarkDuplicates After output close freeMemory: 262955216; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:41:11 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.71 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/304.sorted.bam O=../qc-processing/bowtie/deduplicated/304.dedup.bam M=../qc-processing/bowtie/deduplicated/304.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:41:12 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/305.sorted.bam -O ../qc-processing/bowtie/deduplicated/305.dedup.bam -M ../qc-processing/bowtie/deduplicated/305.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:41:13.017 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:41:13 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/305.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/305.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/305.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:41:13 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:41:13 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:41:13 MarkDuplicates Start of doWork freeMemory: 75207992; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 18:41:13 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:41:13 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:41:16 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5042:6,316
INFO 2021-03-14 18:41:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:17 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig22036:959
INFO 2021-03-14 18:41:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:19 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,992
INFO 2021-03-14 18:41:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:21 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig39579:29,793
INFO 2021-03-14 18:41:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:22 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig52771:6,422
INFO 2021-03-14 18:41:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:24 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig63317:18,483
INFO 2021-03-14 18:41:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:25 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig70819:7,354
INFO 2021-03-14 18:41:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:27 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig93510:1
INFO 2021-03-14 18:41:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:29 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig123544:583
INFO 2021-03-14 18:41:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:31 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig391966:1,005
INFO 2021-03-14 18:41:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:31 MarkDuplicates Read 10141601 records. 0 pairs never matched.
INFO 2021-03-14 18:41:31 MarkDuplicates After buildSortedReadEndLists freeMemory: 966411328; totalMemory: 1470103552; maxMemory: 4294967296
INFO 2021-03-14 18:41:31 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:41:31 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:41:31 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:41:32 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:41:32 MarkDuplicates After generateDuplicateIndexes freeMemory: 1411883200; totalMemory: 2544893952; maxMemory: 4294967296
INFO 2021-03-14 18:41:32 MarkDuplicates Marking 2355711 records as duplicates.
INFO 2021-03-14 18:41:32 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:41:33 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:41:51 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:41:51 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:41:51 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:41:51 MarkDuplicates Before output close freeMemory: 2433219384; totalMemory: 2544893952; maxMemory: 4294967296
INFO 2021-03-14 18:41:51 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:41:51 MarkDuplicates After output close freeMemory: 262953584; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:41:51 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.64 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/305.sorted.bam O=../qc-processing/bowtie/deduplicated/305.dedup.bam M=../qc-processing/bowtie/deduplicated/305.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:41:53 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/306.sorted.bam -O ../qc-processing/bowtie/deduplicated/306.dedup.bam -M ../qc-processing/bowtie/deduplicated/306.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:41:53.289 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:41:53 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/306.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/306.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/306.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:41:53 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:41:53 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:41:53 MarkDuplicates Start of doWork freeMemory: 72063768; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:41:53 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:41:53 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:41:56 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4151:778
INFO 2021-03-14 18:41:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:58 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig20021:6,825
INFO 2021-03-14 18:41:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:41:59 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig29072:37,233
INFO 2021-03-14 18:41:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:01 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35213:878
INFO 2021-03-14 18:42:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:03 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig44127:2,209
INFO 2021-03-14 18:42:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:04 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig56521:5,240
INFO 2021-03-14 18:42:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:06 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig66871:14,241
INFO 2021-03-14 18:42:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:08 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig73139:255
INFO 2021-03-14 18:42:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:09 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig94817:3,286
INFO 2021-03-14 18:42:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:11 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig117949:512
INFO 2021-03-14 18:42:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:13 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig194722:1,063
INFO 2021-03-14 18:42:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:13 MarkDuplicates Read 11374201 records. 0 pairs never matched.
INFO 2021-03-14 18:42:13 MarkDuplicates After buildSortedReadEndLists freeMemory: 889362312; totalMemory: 1482686464; maxMemory: 4294967296
INFO 2021-03-14 18:42:13 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:42:14 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:42:14 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:42:15 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:42:15 MarkDuplicates After generateDuplicateIndexes freeMemory: 1424467488; totalMemory: 2557476864; maxMemory: 4294967296
INFO 2021-03-14 18:42:15 MarkDuplicates Marking 3524830 records as duplicates.
INFO 2021-03-14 18:42:15 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:42:16 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:42:34 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:42:34 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:42:34 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:42:34 MarkDuplicates Before output close freeMemory: 2443617840; totalMemory: 2557476864; maxMemory: 4294967296
INFO 2021-03-14 18:42:34 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:42:34 MarkDuplicates After output close freeMemory: 266101048; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 18:42:34 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.69 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/306.sorted.bam O=../qc-processing/bowtie/deduplicated/306.dedup.bam M=../qc-processing/bowtie/deduplicated/306.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:42:36 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/307.sorted.bam -O ../qc-processing/bowtie/deduplicated/307.dedup.bam -M ../qc-processing/bowtie/deduplicated/307.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:42:36.423 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:42:36 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/307.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/307.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/307.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:42:36 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:42:36 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:42:36 MarkDuplicates Start of doWork freeMemory: 75210616; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 18:42:36 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:42:36 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:42:39 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4035:809
INFO 2021-03-14 18:42:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:41 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig20103:4,160
INFO 2021-03-14 18:42:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:42 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig29276:2,601
INFO 2021-03-14 18:42:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:44 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35461:3,280
INFO 2021-03-14 18:42:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:46 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig45593:3,499
INFO 2021-03-14 18:42:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:47 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig57206:190
INFO 2021-03-14 18:42:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:49 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,809
INFO 2021-03-14 18:42:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:51 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig84270:1,150
INFO 2021-03-14 18:42:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:52 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig105337:8,560
INFO 2021-03-14 18:42:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:54 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig154392:6,495
INFO 2021-03-14 18:42:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:42:55 MarkDuplicates Read 10677350 records. 0 pairs never matched.
INFO 2021-03-14 18:42:55 MarkDuplicates After buildSortedReadEndLists freeMemory: 1229584120; totalMemory: 1795162112; maxMemory: 4294967296
INFO 2021-03-14 18:42:55 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:42:56 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:42:56 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:42:57 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:42:57 MarkDuplicates After generateDuplicateIndexes freeMemory: 1736943024; totalMemory: 2869952512; maxMemory: 4294967296
INFO 2021-03-14 18:42:57 MarkDuplicates Marking 3068097 records as duplicates.
INFO 2021-03-14 18:42:57 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:42:57 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:43:15 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:43:15 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:43:15 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:43:15 MarkDuplicates Before output close freeMemory: 263501632; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:43:15 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:43:15 MarkDuplicates After output close freeMemory: 258761176; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:43:15 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.65 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/307.sorted.bam O=../qc-processing/bowtie/deduplicated/307.dedup.bam M=../qc-processing/bowtie/deduplicated/307.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:43:17 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/308.sorted.bam -O ../qc-processing/bowtie/deduplicated/308.dedup.bam -M ../qc-processing/bowtie/deduplicated/308.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:43:17.340 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:43:17 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/308.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/308.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/308.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:43:17 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:43:17 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:43:17 MarkDuplicates Start of doWork freeMemory: 72061112; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:43:17 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:43:17 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:43:20 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3005:21,600
INFO 2021-03-14 18:43:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:22 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig17443:17,549
INFO 2021-03-14 18:43:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:23 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig25496:21,631
INFO 2021-03-14 18:43:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:25 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig32730:7,388
INFO 2021-03-14 18:43:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:26 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig37686:4,130
INFO 2021-03-14 18:43:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:28 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig46700:4,617
INFO 2021-03-14 18:43:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:30 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,680
INFO 2021-03-14 18:43:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:31 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig64644:275
INFO 2021-03-14 18:43:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:33 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig71412:29,764
INFO 2021-03-14 18:43:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:34 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,944
INFO 2021-03-14 18:43:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:36 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig107599:7,284
INFO 2021-03-14 18:43:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:38 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig147638:757
INFO 2021-03-14 18:43:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:43:39 MarkDuplicates Read 12964605 records. 0 pairs never matched.
INFO 2021-03-14 18:43:39 MarkDuplicates After buildSortedReadEndLists freeMemory: 829658464; totalMemory: 1465909248; maxMemory: 4294967296
INFO 2021-03-14 18:43:39 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:43:40 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:43:40 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:43:41 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:43:41 MarkDuplicates After generateDuplicateIndexes freeMemory: 1407687672; totalMemory: 2540699648; maxMemory: 4294967296
INFO 2021-03-14 18:43:41 MarkDuplicates Marking 3980225 records as duplicates.
INFO 2021-03-14 18:43:41 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:43:42 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:44:02 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:44:02 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:44:02 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:44:02 MarkDuplicates Before output close freeMemory: 267981816; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:44:02 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:44:02 MarkDuplicates After output close freeMemory: 262950416; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:44:02 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.75 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/308.sorted.bam O=../qc-processing/bowtie/deduplicated/308.dedup.bam M=../qc-processing/bowtie/deduplicated/308.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:44:04 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/309.sorted.bam -O ../qc-processing/bowtie/deduplicated/309.dedup.bam -M ../qc-processing/bowtie/deduplicated/309.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:44:04.269 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:44:04 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/309.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/309.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/309.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:44:04 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:44:04 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:44:04 MarkDuplicates Start of doWork freeMemory: 72063016; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:44:04 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:44:04 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:44:07 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2750:11,456
INFO 2021-03-14 18:44:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:09 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig16904:4,783
INFO 2021-03-14 18:44:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:11 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig24126:14,157
INFO 2021-03-14 18:44:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:12 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,655
INFO 2021-03-14 18:44:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:14 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig35613:10,283
INFO 2021-03-14 18:44:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:16 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig42980:24,005
INFO 2021-03-14 18:44:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:17 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig52971:1,838
INFO 2021-03-14 18:44:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:19 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig59523:8,321
INFO 2021-03-14 18:44:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:21 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,821
INFO 2021-03-14 18:44:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:22 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig74383:6,496
INFO 2021-03-14 18:44:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:24 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig90311:478
INFO 2021-03-14 18:44:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:25 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig111123:11,150
INFO 2021-03-14 18:44:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:27 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig148303:745
INFO 2021-03-14 18:44:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:44:29 MarkDuplicates Read 13993476 records. 0 pairs never matched.
INFO 2021-03-14 18:44:29 MarkDuplicates After buildSortedReadEndLists freeMemory: 906439848; totalMemory: 1569718272; maxMemory: 4294967296
INFO 2021-03-14 18:44:29 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:44:29 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:44:29 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:44:30 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:44:31 MarkDuplicates After generateDuplicateIndexes freeMemory: 1511499216; totalMemory: 2644508672; maxMemory: 4294967296
INFO 2021-03-14 18:44:31 MarkDuplicates Marking 4005909 records as duplicates.
INFO 2021-03-14 18:44:31 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:44:31 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:44:54 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:44:54 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:44:54 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:44:54 MarkDuplicates Before output close freeMemory: 271011368; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 18:44:54 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:44:54 MarkDuplicates After output close freeMemory: 262954976; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:44:54 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.84 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/309.sorted.bam O=../qc-processing/bowtie/deduplicated/309.dedup.bam M=../qc-processing/bowtie/deduplicated/309.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:44:56 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/311.sorted.bam -O ../qc-processing/bowtie/deduplicated/311.dedup.bam -M ../qc-processing/bowtie/deduplicated/311.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:44:56.704 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:44:56 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/311.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/311.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/311.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:44:56 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:44:56 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:44:56 MarkDuplicates Start of doWork freeMemory: 72063200; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:44:56 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:44:56 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:44:59 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3874:7,263
INFO 2021-03-14 18:44:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:01 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig20004:12,075
INFO 2021-03-14 18:45:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:03 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig28601:21,958
INFO 2021-03-14 18:45:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:04 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35213:870
INFO 2021-03-14 18:45:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:06 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig43353:18,683
INFO 2021-03-14 18:45:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:08 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig56033:1,092
INFO 2021-03-14 18:45:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:09 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig66007:6,443
INFO 2021-03-14 18:45:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:11 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig70638:19,186
INFO 2021-03-14 18:45:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:12 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig87755:1,513
INFO 2021-03-14 18:45:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:14 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig111970:7,422
INFO 2021-03-14 18:45:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:16 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig175154:3,901
INFO 2021-03-14 18:45:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:17 MarkDuplicates Read 11533234 records. 0 pairs never matched.
INFO 2021-03-14 18:45:17 MarkDuplicates After buildSortedReadEndLists freeMemory: 1001635824; totalMemory: 1576009728; maxMemory: 4294967296
INFO 2021-03-14 18:45:17 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:45:17 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:45:17 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:45:18 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:45:18 MarkDuplicates After generateDuplicateIndexes freeMemory: 1517791112; totalMemory: 2650800128; maxMemory: 4294967296
INFO 2021-03-14 18:45:18 MarkDuplicates Marking 3094631 records as duplicates.
INFO 2021-03-14 18:45:18 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:45:19 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:45:37 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:45:37 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:45:37 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:45:38 MarkDuplicates Before output close freeMemory: 266926408; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:45:38 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:45:38 MarkDuplicates After output close freeMemory: 262955568; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:45:38 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.69 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/311.sorted.bam O=../qc-processing/bowtie/deduplicated/311.dedup.bam M=../qc-processing/bowtie/deduplicated/311.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:45:40 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/312.sorted.bam -O ../qc-processing/bowtie/deduplicated/312.dedup.bam -M ../qc-processing/bowtie/deduplicated/312.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:45:40.294 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:45:40 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/312.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/312.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/312.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:45:40 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:45:40 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:45:40 MarkDuplicates Start of doWork freeMemory: 72063536; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:45:40 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:45:40 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:45:43 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3036:14,200
INFO 2021-03-14 18:45:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:45 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig18125:18,538
INFO 2021-03-14 18:45:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:46 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig26549:5,947
INFO 2021-03-14 18:45:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:48 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig33765:4,506
INFO 2021-03-14 18:45:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:50 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig41930:235
INFO 2021-03-14 18:45:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:51 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig53025:2,316
INFO 2021-03-14 18:45:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:53 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig64723:1,565
INFO 2021-03-14 18:45:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:54 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig74351:646
INFO 2021-03-14 18:45:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:56 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig97360:708
INFO 2021-03-14 18:45:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:58 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig123544:573
INFO 2021-03-14 18:45:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:45:59 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig196361:896
INFO 2021-03-14 18:45:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:00 MarkDuplicates Read 11408430 records. 0 pairs never matched.
INFO 2021-03-14 18:46:00 MarkDuplicates After buildSortedReadEndLists freeMemory: 1172340304; totalMemory: 1737490432; maxMemory: 4294967296
INFO 2021-03-14 18:46:00 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:46:00 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:46:00 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:46:01 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:46:02 MarkDuplicates After generateDuplicateIndexes freeMemory: 1679271008; totalMemory: 2812280832; maxMemory: 4294967296
INFO 2021-03-14 18:46:02 MarkDuplicates Marking 2991794 records as duplicates.
INFO 2021-03-14 18:46:02 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:46:02 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:46:21 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:46:21 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:46:21 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:46:21 MarkDuplicates Before output close freeMemory: 263603720; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:46:21 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:46:21 MarkDuplicates After output close freeMemory: 262954816; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:46:21 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.69 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/312.sorted.bam O=../qc-processing/bowtie/deduplicated/312.dedup.bam M=../qc-processing/bowtie/deduplicated/312.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:46:23 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/313.sorted.bam -O ../qc-processing/bowtie/deduplicated/313.dedup.bam -M ../qc-processing/bowtie/deduplicated/313.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:46:23.464 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:46:23 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/313.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/313.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/313.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:46:23 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:46:23 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:46:23 MarkDuplicates Start of doWork freeMemory: 75207864; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 18:46:23 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:46:23 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:46:27 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4608:13,623
INFO 2021-03-14 18:46:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:28 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig20707:12,497
INFO 2021-03-14 18:46:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:30 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig29976:25,827
INFO 2021-03-14 18:46:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:31 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig36186:33,329
INFO 2021-03-14 18:46:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:33 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig45969:18,573
INFO 2021-03-14 18:46:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:35 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig56693:11,961
INFO 2021-03-14 18:46:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:36 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig68253:1,151
INFO 2021-03-14 18:46:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:38 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig74927:534
INFO 2021-03-14 18:46:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:40 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig98616:5,019
INFO 2021-03-14 18:46:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:41 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig133838:606
INFO 2021-03-14 18:46:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:43 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig519002:3,189
INFO 2021-03-14 18:46:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:46:43 MarkDuplicates Read 11112238 records. 0 pairs never matched.
INFO 2021-03-14 18:46:43 MarkDuplicates After buildSortedReadEndLists freeMemory: 771321592; totalMemory: 1342177280; maxMemory: 4294967296
INFO 2021-03-14 18:46:43 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:46:44 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:46:44 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:46:45 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:46:45 MarkDuplicates After generateDuplicateIndexes freeMemory: 1283956728; totalMemory: 2416967680; maxMemory: 4294967296
INFO 2021-03-14 18:46:45 MarkDuplicates Marking 3249319 records as duplicates.
INFO 2021-03-14 18:46:45 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:46:46 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:47:03 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:47:03 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:47:03 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:47:04 MarkDuplicates Before output close freeMemory: 266811864; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:47:04 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:47:04 MarkDuplicates After output close freeMemory: 258759384; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:47:04 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.68 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/313.sorted.bam O=../qc-processing/bowtie/deduplicated/313.dedup.bam M=../qc-processing/bowtie/deduplicated/313.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:47:05 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/314.sorted.bam -O ../qc-processing/bowtie/deduplicated/314.dedup.bam -M ../qc-processing/bowtie/deduplicated/314.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:47:06.000 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:47:06 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/314.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/314.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/314.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:47:06 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:47:06 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:47:06 MarkDuplicates Start of doWork freeMemory: 72060736; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:47:06 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:47:06 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:47:09 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16563:14,119
INFO 2021-03-14 18:47:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:11 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig29213:12,328
INFO 2021-03-14 18:47:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:12 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig38003:16,521
INFO 2021-03-14 18:47:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:14 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56033:1,106
INFO 2021-03-14 18:47:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:16 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig66297:3,136
INFO 2021-03-14 18:47:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:17 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,925
INFO 2021-03-14 18:47:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:19 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig89106:10,274
INFO 2021-03-14 18:47:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:21 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig112073:1,953
INFO 2021-03-14 18:47:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:22 MarkDuplicates Read 8972206 records. 0 pairs never matched.
INFO 2021-03-14 18:47:23 MarkDuplicates After buildSortedReadEndLists freeMemory: 1061879600; totalMemory: 1509949440; maxMemory: 4294967296
INFO 2021-03-14 18:47:23 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:47:23 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:47:23 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:47:24 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:47:24 MarkDuplicates After generateDuplicateIndexes freeMemory: 1451727880; totalMemory: 2584739840; maxMemory: 4294967296
INFO 2021-03-14 18:47:24 MarkDuplicates Marking 2392794 records as duplicates.
INFO 2021-03-14 18:47:24 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:47:24 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:47:40 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:47:40 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:47:40 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:47:40 MarkDuplicates Before output close freeMemory: 271142088; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 18:47:40 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:47:40 MarkDuplicates After output close freeMemory: 258758784; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:47:40 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.58 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/314.sorted.bam O=../qc-processing/bowtie/deduplicated/314.dedup.bam M=../qc-processing/bowtie/deduplicated/314.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:47:42 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/315.sorted.bam -O ../qc-processing/bowtie/deduplicated/315.dedup.bam -M ../qc-processing/bowtie/deduplicated/315.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:47:42.593 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:47:42 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/315.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/315.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/315.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:47:42 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:47:42 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:47:42 MarkDuplicates Start of doWork freeMemory: 75210184; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 18:47:42 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:47:42 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:47:46 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2688:17,828
INFO 2021-03-14 18:47:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:47 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig16435:2,100
INFO 2021-03-14 18:47:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:49 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig23843:36,756
INFO 2021-03-14 18:47:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:50 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig30569:261
INFO 2021-03-14 18:47:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:52 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig35403:41,139
INFO 2021-03-14 18:47:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:54 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig42766:13,552
INFO 2021-03-14 18:47:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:55 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig52605:3,506
INFO 2021-03-14 18:47:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:57 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig59490:14,284
INFO 2021-03-14 18:47:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:47:58 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig69234:9,229
INFO 2021-03-14 18:47:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:00 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig78287:14,880
INFO 2021-03-14 18:48:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:01 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,705
INFO 2021-03-14 18:48:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:03 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig124113:2,808
INFO 2021-03-14 18:48:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:04 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig182192:5,547
INFO 2021-03-14 18:48:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:05 MarkDuplicates Read 13585458 records. 0 pairs never matched.
INFO 2021-03-14 18:48:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 870717000; totalMemory: 1534066688; maxMemory: 4294967296
INFO 2021-03-14 18:48:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:48:07 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:48:07 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:48:08 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:48:08 MarkDuplicates After generateDuplicateIndexes freeMemory: 1475847368; totalMemory: 2608857088; maxMemory: 4294967296
INFO 2021-03-14 18:48:08 MarkDuplicates Marking 4162565 records as duplicates.
INFO 2021-03-14 18:48:08 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:48:10 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:48:30 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:48:30 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:48:30 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:48:31 MarkDuplicates Before output close freeMemory: 271053512; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 18:48:31 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:48:31 MarkDuplicates After output close freeMemory: 262954792; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:48:31 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.81 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/315.sorted.bam O=../qc-processing/bowtie/deduplicated/315.dedup.bam M=../qc-processing/bowtie/deduplicated/315.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:48:32 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/316.sorted.bam -O ../qc-processing/bowtie/deduplicated/316.dedup.bam -M ../qc-processing/bowtie/deduplicated/316.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:48:32.922 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:48:33 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/316.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/316.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/316.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:48:33 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:48:33 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:48:33 MarkDuplicates Start of doWork freeMemory: 67871640; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:48:33 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:48:33 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:48:36 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig1927:15,652
INFO 2021-03-14 18:48:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:37 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig8637:2,376
INFO 2021-03-14 18:48:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:39 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig20015:4,524
INFO 2021-03-14 18:48:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:41 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig26025:641
INFO 2021-03-14 18:48:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:42 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig30879:28,594
INFO 2021-03-14 18:48:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:44 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig35776:12,482
INFO 2021-03-14 18:48:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:45 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig42152:3,532
INFO 2021-03-14 18:48:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:47 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig50033:24,901
INFO 2021-03-14 18:48:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:48 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig56693:12,060
INFO 2021-03-14 18:48:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:50 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig64488:2,628
INFO 2021-03-14 18:48:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:51 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig69263:4,710
INFO 2021-03-14 18:48:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:53 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig78322:61
INFO 2021-03-14 18:48:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:55 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig91287:23
INFO 2021-03-14 18:48:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:56 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig107387:6,269
INFO 2021-03-14 18:48:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:58 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 1s. Last read position: Contig137778:3,201
INFO 2021-03-14 18:48:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:48:59 MarkDuplicates Read 16,000,000 records. Elapsed time: 00:00:25s. Time for last 1,000,000: 1s. Last read position: Contig206377:598
INFO 2021-03-14 18:48:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:00 MarkDuplicates Read 16487174 records. 0 pairs never matched.
INFO 2021-03-14 18:49:00 MarkDuplicates After buildSortedReadEndLists freeMemory: 925568544; totalMemory: 1663041536; maxMemory: 4294967296
INFO 2021-03-14 18:49:00 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:49:01 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:49:01 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:49:02 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:49:02 MarkDuplicates After generateDuplicateIndexes freeMemory: 1604821976; totalMemory: 2737831936; maxMemory: 4294967296
INFO 2021-03-14 18:49:02 MarkDuplicates Marking 4746427 records as duplicates.
INFO 2021-03-14 18:49:02 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:49:03 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:49:24 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:21s. Time for last 10,000,000: 21s. Last read position: Contig111110:738
INFO 2021-03-14 18:49:28 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:49:28 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:49:28 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:49:28 MarkDuplicates Before output close freeMemory: 267230272; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:49:28 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:49:28 MarkDuplicates After output close freeMemory: 262954264; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:49:28 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.93 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/316.sorted.bam O=../qc-processing/bowtie/deduplicated/316.dedup.bam M=../qc-processing/bowtie/deduplicated/316.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:49:30 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/317.sorted.bam -O ../qc-processing/bowtie/deduplicated/317.dedup.bam -M ../qc-processing/bowtie/deduplicated/317.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:49:30.592 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:49:30 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/317.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/317.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/317.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:49:30 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:49:30 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:49:30 MarkDuplicates Start of doWork freeMemory: 72062560; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:49:30 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:49:30 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:49:33 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5612:4,135
INFO 2021-03-14 18:49:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:35 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig22309:2,660
INFO 2021-03-14 18:49:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:37 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig31395:31,151
INFO 2021-03-14 18:49:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:38 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig40789:87
INFO 2021-03-14 18:49:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:40 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig54239:529
INFO 2021-03-14 18:49:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:41 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig66209:9,419
INFO 2021-03-14 18:49:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:43 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig77346:3,884
INFO 2021-03-14 18:49:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:45 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,264
INFO 2021-03-14 18:49:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:46 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig149343:2,185
INFO 2021-03-14 18:49:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:49:47 MarkDuplicates Read 9672371 records. 0 pairs never matched.
INFO 2021-03-14 18:49:48 MarkDuplicates After buildSortedReadEndLists freeMemory: 914813424; totalMemory: 1423966208; maxMemory: 4294967296
INFO 2021-03-14 18:49:48 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:49:48 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:49:48 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:49:49 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:49:49 MarkDuplicates After generateDuplicateIndexes freeMemory: 1365745632; totalMemory: 2498756608; maxMemory: 4294967296
INFO 2021-03-14 18:49:49 MarkDuplicates Marking 2376458 records as duplicates.
INFO 2021-03-14 18:49:49 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:49:50 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:50:07 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:50:07 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:50:07 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:50:07 MarkDuplicates Before output close freeMemory: 2387710192; totalMemory: 2498756608; maxMemory: 4294967296
INFO 2021-03-14 18:50:07 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:50:07 MarkDuplicates After output close freeMemory: 262954104; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:50:07 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.62 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/317.sorted.bam O=../qc-processing/bowtie/deduplicated/317.dedup.bam M=../qc-processing/bowtie/deduplicated/317.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:50:09 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/318.sorted.bam -O ../qc-processing/bowtie/deduplicated/318.dedup.bam -M ../qc-processing/bowtie/deduplicated/318.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:50:09.567 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:50:09 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/318.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/318.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/318.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:50:09 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:50:09 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:50:09 MarkDuplicates Start of doWork freeMemory: 72062096; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:50:09 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:50:09 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:50:13 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 3s. Last read position: Contig4689:1
INFO 2021-03-14 18:50:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:15 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig21387:3,165
INFO 2021-03-14 18:50:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:17 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,667
INFO 2021-03-14 18:50:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:18 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig38252:19,888
INFO 2021-03-14 18:50:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:20 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig51001:3,157
INFO 2021-03-14 18:50:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:22 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig61804:4,494
INFO 2021-03-14 18:50:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:23 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig73139:80
INFO 2021-03-14 18:50:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:25 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig98642:9,877
INFO 2021-03-14 18:50:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:26 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig137778:4,200
INFO 2021-03-14 18:50:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:28 MarkDuplicates Read 9912254 records. 0 pairs never matched.
INFO 2021-03-14 18:50:28 MarkDuplicates After buildSortedReadEndLists freeMemory: 1191886000; totalMemory: 1729101824; maxMemory: 4294967296
INFO 2021-03-14 18:50:28 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:50:29 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:50:29 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:50:30 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:50:30 MarkDuplicates After generateDuplicateIndexes freeMemory: 1670881864; totalMemory: 2803892224; maxMemory: 4294967296
INFO 2021-03-14 18:50:30 MarkDuplicates Marking 2849552 records as duplicates.
INFO 2021-03-14 18:50:30 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:50:31 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:50:47 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:50:47 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:50:47 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:50:47 MarkDuplicates Before output close freeMemory: 265477248; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:50:47 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:50:47 MarkDuplicates After output close freeMemory: 262954248; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:50:47 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.63 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/318.sorted.bam O=../qc-processing/bowtie/deduplicated/318.dedup.bam M=../qc-processing/bowtie/deduplicated/318.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:50:49 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/319.sorted.bam -O ../qc-processing/bowtie/deduplicated/319.dedup.bam -M ../qc-processing/bowtie/deduplicated/319.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:50:49.071 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:50:49 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/319.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/319.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/319.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:50:49 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:50:49 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:50:49 MarkDuplicates Start of doWork freeMemory: 72063952; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:50:49 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:50:49 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:50:52 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3671:5,293
INFO 2021-03-14 18:50:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:53 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig19437:7,265
INFO 2021-03-14 18:50:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:55 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig28248:872
INFO 2021-03-14 18:50:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:57 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35213:23
INFO 2021-03-14 18:50:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:50:58 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig43565:1,872
INFO 2021-03-14 18:50:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:00 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig56033:1,103
INFO 2021-03-14 18:51:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:01 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig67512:4,131
INFO 2021-03-14 18:51:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:03 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig75123:6,167
INFO 2021-03-14 18:51:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:05 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig98472:2,165
INFO 2021-03-14 18:51:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:06 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig119450:2,337
INFO 2021-03-14 18:51:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:08 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig192657:2,815
INFO 2021-03-14 18:51:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:08 MarkDuplicates Read 11422505 records. 0 pairs never matched.
INFO 2021-03-14 18:51:09 MarkDuplicates After buildSortedReadEndLists freeMemory: 887889128; totalMemory: 1473249280; maxMemory: 4294967296
INFO 2021-03-14 18:51:09 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:51:09 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:51:09 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:51:10 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:51:10 MarkDuplicates After generateDuplicateIndexes freeMemory: 1415030608; totalMemory: 2548039680; maxMemory: 4294967296
INFO 2021-03-14 18:51:10 MarkDuplicates Marking 3882970 records as duplicates.
INFO 2021-03-14 18:51:10 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:51:11 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:51:29 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:51:29 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:51:29 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:51:29 MarkDuplicates Before output close freeMemory: 271494936; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 18:51:29 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:51:29 MarkDuplicates After output close freeMemory: 258761080; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:51:29 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.67 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/319.sorted.bam O=../qc-processing/bowtie/deduplicated/319.dedup.bam M=../qc-processing/bowtie/deduplicated/319.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:51:31 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/321.sorted.bam -O ../qc-processing/bowtie/deduplicated/321.dedup.bam -M ../qc-processing/bowtie/deduplicated/321.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:51:31.107 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:51:31 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/321.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/321.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/321.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:51:31 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:51:31 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:51:31 MarkDuplicates Start of doWork freeMemory: 72064064; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:51:31 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:51:31 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:51:34 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5118:1,555
INFO 2021-03-14 18:51:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:36 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig22087:36,503
INFO 2021-03-14 18:51:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:37 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,922
INFO 2021-03-14 18:51:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:39 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig39730:51
INFO 2021-03-14 18:51:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:41 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig52672:3,609
INFO 2021-03-14 18:51:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:42 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig63400:5,899
INFO 2021-03-14 18:51:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:44 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig71398:5,251
INFO 2021-03-14 18:51:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:45 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig95550:392
INFO 2021-03-14 18:51:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:47 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig127749:5,129
INFO 2021-03-14 18:51:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:49 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig519002:3,466
INFO 2021-03-14 18:51:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:51:49 MarkDuplicates Read 10083385 records. 0 pairs never matched.
INFO 2021-03-14 18:51:49 MarkDuplicates After buildSortedReadEndLists freeMemory: 959952224; totalMemory: 1482686464; maxMemory: 4294967296
INFO 2021-03-14 18:51:49 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:51:49 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:51:49 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:51:50 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:51:50 MarkDuplicates After generateDuplicateIndexes freeMemory: 1424467744; totalMemory: 2557476864; maxMemory: 4294967296
INFO 2021-03-14 18:51:50 MarkDuplicates Marking 2547479 records as duplicates.
INFO 2021-03-14 18:51:50 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:51:51 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:52:08 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:52:08 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:52:08 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:52:08 MarkDuplicates Before output close freeMemory: 267501160; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:52:08 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:52:08 MarkDuplicates After output close freeMemory: 266101096; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 18:52:08 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.63 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/321.sorted.bam O=../qc-processing/bowtie/deduplicated/321.dedup.bam M=../qc-processing/bowtie/deduplicated/321.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:52:10 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/322.sorted.bam -O ../qc-processing/bowtie/deduplicated/322.dedup.bam -M ../qc-processing/bowtie/deduplicated/322.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:52:10.655 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:52:10 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/322.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/322.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/322.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:52:10 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:52:10 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:52:10 MarkDuplicates Start of doWork freeMemory: 72065120; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:52:10 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:52:10 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:52:13 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3311:9,966
INFO 2021-03-14 18:52:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:15 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig19099:1,403
INFO 2021-03-14 18:52:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:17 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig27068:7,330
INFO 2021-03-14 18:52:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:18 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig34149:10,404
INFO 2021-03-14 18:52:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:20 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig41787:13,428
INFO 2021-03-14 18:52:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:23 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig61794:5,520
INFO 2021-03-14 18:52:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:25 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,581
INFO 2021-03-14 18:52:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:26 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig85486:2,331
INFO 2021-03-14 18:52:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:28 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig107823:19,404
INFO 2021-03-14 18:52:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:29 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig151503:1,505
INFO 2021-03-14 18:52:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:31 MarkDuplicates Read 11786078 records. 0 pairs never matched.
INFO 2021-03-14 18:52:31 MarkDuplicates After buildSortedReadEndLists freeMemory: 1029882048; totalMemory: 1627389952; maxMemory: 4294967296
INFO 2021-03-14 18:52:31 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:52:31 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:52:31 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:52:32 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:52:33 MarkDuplicates After generateDuplicateIndexes freeMemory: 1569171152; totalMemory: 2702180352; maxMemory: 4294967296
INFO 2021-03-14 18:52:33 MarkDuplicates Marking 3431166 records as duplicates.
INFO 2021-03-14 18:52:33 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:52:33 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:52:52 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:52:52 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:52:52 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:52:52 MarkDuplicates Before output close freeMemory: 261069632; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 18:52:52 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:52:52 MarkDuplicates After output close freeMemory: 258761104; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:52:52 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.70 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/322.sorted.bam O=../qc-processing/bowtie/deduplicated/322.dedup.bam M=../qc-processing/bowtie/deduplicated/322.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:52:54 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/323.sorted.bam -O ../qc-processing/bowtie/deduplicated/323.dedup.bam -M ../qc-processing/bowtie/deduplicated/323.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:52:54.494 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:52:54 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/323.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/323.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/323.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:52:54 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:52:54 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:52:54 MarkDuplicates Start of doWork freeMemory: 72063072; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:52:54 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:52:54 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:52:57 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2021:3,967
INFO 2021-03-14 18:52:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:52:59 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig9322:683
INFO 2021-03-14 18:52:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:01 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig20662:23,925
INFO 2021-03-14 18:53:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:02 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig27036:6,835
INFO 2021-03-14 18:53:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:04 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig32909:11,883
INFO 2021-03-14 18:53:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:05 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig37595:11,517
INFO 2021-03-14 18:53:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:07 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig44916:13,929
INFO 2021-03-14 18:53:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:08 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig53739:11,272
INFO 2021-03-14 18:53:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:10 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig61889:466
INFO 2021-03-14 18:53:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:11 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig70595:996
INFO 2021-03-14 18:53:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:13 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig83174:2,272
INFO 2021-03-14 18:53:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:14 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,208
INFO 2021-03-14 18:53:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:16 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig121487:5,946
INFO 2021-03-14 18:53:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:17 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig165237:1,379
INFO 2021-03-14 18:53:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:19 MarkDuplicates Read 14887252 records. 0 pairs never matched.
INFO 2021-03-14 18:53:19 MarkDuplicates After buildSortedReadEndLists freeMemory: 1013060728; totalMemory: 1742733312; maxMemory: 4294967296
INFO 2021-03-14 18:53:19 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:53:19 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:53:19 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:53:20 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:53:21 MarkDuplicates After generateDuplicateIndexes freeMemory: 1684512528; totalMemory: 2817523712; maxMemory: 4294967296
INFO 2021-03-14 18:53:21 MarkDuplicates Marking 4633463 records as duplicates.
INFO 2021-03-14 18:53:21 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:53:21 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:53:43 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:21s. Time for last 10,000,000: 21s. Last read position: Contig213225:1,154
INFO 2021-03-14 18:53:43 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:53:43 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:53:43 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:53:44 MarkDuplicates Before output close freeMemory: 264389488; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:53:44 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:53:44 MarkDuplicates After output close freeMemory: 258759096; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:53:44 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.83 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/323.sorted.bam O=../qc-processing/bowtie/deduplicated/323.dedup.bam M=../qc-processing/bowtie/deduplicated/323.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:53:45 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/324.sorted.bam -O ../qc-processing/bowtie/deduplicated/324.dedup.bam -M ../qc-processing/bowtie/deduplicated/324.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:53:45.972 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:53:46 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/324.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/324.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/324.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:53:46 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:53:46 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:53:46 MarkDuplicates Start of doWork freeMemory: 72064800; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:53:46 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:53:46 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:53:49 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2173:15,090
INFO 2021-03-14 18:53:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:50 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig16513:4,654
INFO 2021-03-14 18:53:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:52 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig24235:20,153
INFO 2021-03-14 18:53:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:53 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig31018:21,050
INFO 2021-03-14 18:53:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:55 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig35434:16,987
INFO 2021-03-14 18:53:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:56 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig42783:29,531
INFO 2021-03-14 18:53:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:53:58 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig53025:1,828
INFO 2021-03-14 18:53:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:00 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig62354:4,275
INFO 2021-03-14 18:54:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:01 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,891
INFO 2021-03-14 18:54:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:03 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,669
INFO 2021-03-14 18:54:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:04 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig106749:435
INFO 2021-03-14 18:54:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:06 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig141072:9,286
INFO 2021-03-14 18:54:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:08 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig627027:5,789
INFO 2021-03-14 18:54:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:08 MarkDuplicates Read 13050914 records. 0 pairs never matched.
INFO 2021-03-14 18:54:08 MarkDuplicates After buildSortedReadEndLists freeMemory: 890362904; totalMemory: 1548746752; maxMemory: 4294967296
INFO 2021-03-14 18:54:08 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:54:08 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:54:08 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:54:09 MarkDuplicates After generateDuplicateIndexes freeMemory: 1490527208; totalMemory: 2623537152; maxMemory: 4294967296
INFO 2021-03-14 18:54:09 MarkDuplicates Marking 4287054 records as duplicates.
INFO 2021-03-14 18:54:09 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:54:10 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:54:29 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:54:29 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:54:29 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:54:30 MarkDuplicates Before output close freeMemory: 262950384; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 18:54:30 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:54:30 MarkDuplicates After output close freeMemory: 258760584; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:54:30 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.74 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/324.sorted.bam O=../qc-processing/bowtie/deduplicated/324.dedup.bam M=../qc-processing/bowtie/deduplicated/324.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:54:32 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/325.sorted.bam -O ../qc-processing/bowtie/deduplicated/325.dedup.bam -M ../qc-processing/bowtie/deduplicated/325.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:54:32.667 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:54:32 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/325.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/325.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/325.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:54:32 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:54:32 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:54:32 MarkDuplicates Start of doWork freeMemory: 72063672; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:54:32 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:54:32 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:54:36 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2912:405
INFO 2021-03-14 18:54:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:37 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig17606:26,803
INFO 2021-03-14 18:54:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:39 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig25418:13,617
INFO 2021-03-14 18:54:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:41 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig32391:18,330
INFO 2021-03-14 18:54:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:42 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig38258:14,976
INFO 2021-03-14 18:54:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:44 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig47690:2,406
INFO 2021-03-14 18:54:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:46 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig56693:12,114
INFO 2021-03-14 18:54:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:47 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig68037:874
INFO 2021-03-14 18:54:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:49 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig74190:10,817
INFO 2021-03-14 18:54:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:50 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig94856:223
INFO 2021-03-14 18:54:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:52 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig119922:1,495
INFO 2021-03-14 18:54:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:54 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig181070:169
INFO 2021-03-14 18:54:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:54:55 MarkDuplicates Read 12567210 records. 0 pairs never matched.
INFO 2021-03-14 18:54:55 MarkDuplicates After buildSortedReadEndLists freeMemory: 1035481448; totalMemory: 1653604352; maxMemory: 4294967296
INFO 2021-03-14 18:54:55 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:54:55 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:54:55 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:54:56 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:54:57 MarkDuplicates After generateDuplicateIndexes freeMemory: 1595385448; totalMemory: 2728394752; maxMemory: 4294967296
INFO 2021-03-14 18:54:57 MarkDuplicates Marking 3361738 records as duplicates.
INFO 2021-03-14 18:54:57 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:54:57 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:55:19 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:55:19 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:55:19 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:55:19 MarkDuplicates Before output close freeMemory: 267198016; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 18:55:19 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:55:19 MarkDuplicates After output close freeMemory: 258761208; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:55:19 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.78 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/325.sorted.bam O=../qc-processing/bowtie/deduplicated/325.dedup.bam M=../qc-processing/bowtie/deduplicated/325.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:55:21 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/326.sorted.bam -O ../qc-processing/bowtie/deduplicated/326.dedup.bam -M ../qc-processing/bowtie/deduplicated/326.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:55:21.164 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:55:21 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/326.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/326.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/326.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:55:21 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:55:21 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:55:21 MarkDuplicates Start of doWork freeMemory: 72062192; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:55:21 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:55:21 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:55:24 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2653:9,933
INFO 2021-03-14 18:55:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:26 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig16586:1,455
INFO 2021-03-14 18:55:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:27 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig24238:9,655
INFO 2021-03-14 18:55:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:29 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig31058:33,223
INFO 2021-03-14 18:55:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:30 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig35720:2,419
INFO 2021-03-14 18:55:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:32 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig42760:2,704
INFO 2021-03-14 18:55:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:34 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig52413:7,853
INFO 2021-03-14 18:55:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:35 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig61298:3,832
INFO 2021-03-14 18:55:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:37 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig69234:14,768
INFO 2021-03-14 18:55:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:38 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig82015:2,365
INFO 2021-03-14 18:55:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:40 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,590
INFO 2021-03-14 18:55:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:42 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig137778:2,754
INFO 2021-03-14 18:55:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:43 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig217221:376
INFO 2021-03-14 18:55:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:55:44 MarkDuplicates Read 13259245 records. 0 pairs never matched.
INFO 2021-03-14 18:55:44 MarkDuplicates After buildSortedReadEndLists freeMemory: 737989656; totalMemory: 1427111936; maxMemory: 4294967296
INFO 2021-03-14 18:55:44 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:55:44 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:55:44 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:55:45 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:55:46 MarkDuplicates After generateDuplicateIndexes freeMemory: 1368891264; totalMemory: 2501902336; maxMemory: 4294967296
INFO 2021-03-14 18:55:46 MarkDuplicates Marking 4458033 records as duplicates.
INFO 2021-03-14 18:55:46 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:55:47 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:56:07 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:56:07 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:56:07 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:56:07 MarkDuplicates Before output close freeMemory: 264443040; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:56:07 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:56:07 MarkDuplicates After output close freeMemory: 262953792; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:56:07 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.77 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/326.sorted.bam O=../qc-processing/bowtie/deduplicated/326.dedup.bam M=../qc-processing/bowtie/deduplicated/326.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:56:09 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/327.sorted.bam -O ../qc-processing/bowtie/deduplicated/327.dedup.bam -M ../qc-processing/bowtie/deduplicated/327.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:56:09.083 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:56:09 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/327.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/327.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/327.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:56:09 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:56:09 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:56:09 MarkDuplicates Start of doWork freeMemory: 72062776; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:56:09 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:56:09 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:56:12 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5564:8,770
INFO 2021-03-14 18:56:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:14 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig22725:1,030
INFO 2021-03-14 18:56:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:16 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig32287:12,225
INFO 2021-03-14 18:56:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:17 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig41456:10,446
INFO 2021-03-14 18:56:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:19 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig55012:3,317
INFO 2021-03-14 18:56:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:20 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig66247:2,698
INFO 2021-03-14 18:56:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:22 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig77150:2,677
INFO 2021-03-14 18:56:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:24 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,239
INFO 2021-03-14 18:56:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:25 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig149741:49
INFO 2021-03-14 18:56:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:26 MarkDuplicates Read 9679107 records. 0 pairs never matched.
INFO 2021-03-14 18:56:27 MarkDuplicates After buildSortedReadEndLists freeMemory: 968185048; totalMemory: 1487929344; maxMemory: 4294967296
INFO 2021-03-14 18:56:27 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:56:27 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:56:27 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:56:28 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:56:28 MarkDuplicates After generateDuplicateIndexes freeMemory: 1429709536; totalMemory: 2562719744; maxMemory: 4294967296
INFO 2021-03-14 18:56:28 MarkDuplicates Marking 2740089 records as duplicates.
INFO 2021-03-14 18:56:28 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:56:29 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:56:44 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:56:44 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:56:44 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:56:45 MarkDuplicates Before output close freeMemory: 264067928; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:56:45 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:56:45 MarkDuplicates After output close freeMemory: 258760528; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:56:45 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.60 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/327.sorted.bam O=../qc-processing/bowtie/deduplicated/327.dedup.bam M=../qc-processing/bowtie/deduplicated/327.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:56:47 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/328.sorted.bam -O ../qc-processing/bowtie/deduplicated/328.dedup.bam -M ../qc-processing/bowtie/deduplicated/328.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:56:47.119 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:56:47 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/328.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/328.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/328.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:56:47 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:56:47 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:56:47 MarkDuplicates Start of doWork freeMemory: 75209824; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 18:56:47 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:56:47 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:56:50 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig1267:12,188
INFO 2021-03-14 18:56:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:51 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig4405:11,675
INFO 2021-03-14 18:56:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:53 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig16152:3,109
INFO 2021-03-14 18:56:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:54 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig20906:5,346
INFO 2021-03-14 18:56:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:56 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig25774:8,446
INFO 2021-03-14 18:56:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:57 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig30068:6,481
INFO 2021-03-14 18:56:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:56:59 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig34228:3,120
INFO 2021-03-14 18:56:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:00 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig38213:13,322
INFO 2021-03-14 18:57:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:02 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig43678:16,844
INFO 2021-03-14 18:57:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:03 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig50282:20,155
INFO 2021-03-14 18:57:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:05 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,683
INFO 2021-03-14 18:57:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:06 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig62463:2,733
INFO 2021-03-14 18:57:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:08 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig69234:14,487
INFO 2021-03-14 18:57:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:10 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig77337:13,648
INFO 2021-03-14 18:57:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:11 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:23s. Time for last 1,000,000: 1s. Last read position: Contig88912:141
INFO 2021-03-14 18:57:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:13 MarkDuplicates Read 16,000,000 records. Elapsed time: 00:00:25s. Time for last 1,000,000: 1s. Last read position: Contig103886:900
INFO 2021-03-14 18:57:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:14 MarkDuplicates Read 17,000,000 records. Elapsed time: 00:00:26s. Time for last 1,000,000: 1s. Last read position: Contig124017:8,941
INFO 2021-03-14 18:57:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:16 MarkDuplicates Read 18,000,000 records. Elapsed time: 00:00:28s. Time for last 1,000,000: 1s. Last read position: Contig150923:994
INFO 2021-03-14 18:57:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:18 MarkDuplicates Read 19,000,000 records. Elapsed time: 00:00:30s. Time for last 1,000,000: 1s. Last read position: Contig218418:1,227
INFO 2021-03-14 18:57:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:18 MarkDuplicates Read 19386217 records. 0 pairs never matched.
INFO 2021-03-14 18:57:18 MarkDuplicates After buildSortedReadEndLists freeMemory: 673274584; totalMemory: 1543503872; maxMemory: 4294967296
INFO 2021-03-14 18:57:18 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:57:19 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:57:19 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:57:21 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:57:21 MarkDuplicates After generateDuplicateIndexes freeMemory: 1485284472; totalMemory: 2618294272; maxMemory: 4294967296
INFO 2021-03-14 18:57:21 MarkDuplicates Marking 5729855 records as duplicates.
INFO 2021-03-14 18:57:21 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:57:22 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:57:43 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:20s. Time for last 10,000,000: 20s. Last read position: Contig82647:5,663
INFO 2021-03-14 18:57:50 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:57:50 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:57:50 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:57:50 MarkDuplicates Before output close freeMemory: 274042944; totalMemory: 384827392; maxMemory: 4294967296
INFO 2021-03-14 18:57:50 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:57:50 MarkDuplicates After output close freeMemory: 258760392; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 18:57:50 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 1.06 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/328.sorted.bam O=../qc-processing/bowtie/deduplicated/328.dedup.bam M=../qc-processing/bowtie/deduplicated/328.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:57:52 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/329.sorted.bam -O ../qc-processing/bowtie/deduplicated/329.dedup.bam -M ../qc-processing/bowtie/deduplicated/329.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:57:52.723 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:57:52 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/329.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/329.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/329.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:57:53 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:57:53 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:57:53 MarkDuplicates Start of doWork freeMemory: 72065272; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:57:53 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:57:53 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:57:55 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3002:39,955
INFO 2021-03-14 18:57:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:57 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig17443:32,129
INFO 2021-03-14 18:57:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:57:59 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig24722:16,736
INFO 2021-03-14 18:57:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:01 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,919
INFO 2021-03-14 18:58:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:02 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig37004:13,707
INFO 2021-03-14 18:58:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:04 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig46265:4,039
INFO 2021-03-14 18:58:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:05 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig56423:22,387
INFO 2021-03-14 18:58:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:07 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig65290:1
INFO 2021-03-14 18:58:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:09 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,637
INFO 2021-03-14 18:58:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:10 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig84222:2,290
INFO 2021-03-14 18:58:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:12 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig101096:4,576
INFO 2021-03-14 18:58:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:14 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig123987:367
INFO 2021-03-14 18:58:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:15 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig194073:1,437
INFO 2021-03-14 18:58:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:16 MarkDuplicates Read 13466533 records. 0 pairs never matched.
INFO 2021-03-14 18:58:16 MarkDuplicates After buildSortedReadEndLists freeMemory: 917677528; totalMemory: 1561329664; maxMemory: 4294967296
INFO 2021-03-14 18:58:16 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:58:17 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:58:17 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:58:18 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:58:18 MarkDuplicates After generateDuplicateIndexes freeMemory: 1503111264; totalMemory: 2636120064; maxMemory: 4294967296
INFO 2021-03-14 18:58:18 MarkDuplicates Marking 3925434 records as duplicates.
INFO 2021-03-14 18:58:18 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:58:19 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:58:40 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:58:40 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:58:40 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:58:40 MarkDuplicates Before output close freeMemory: 271405896; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 18:58:41 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:58:41 MarkDuplicates After output close freeMemory: 262955528; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:58:41 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.81 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/329.sorted.bam O=../qc-processing/bowtie/deduplicated/329.dedup.bam M=../qc-processing/bowtie/deduplicated/329.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:58:42 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/331.sorted.bam -O ../qc-processing/bowtie/deduplicated/331.dedup.bam -M ../qc-processing/bowtie/deduplicated/331.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:58:42.845 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:58:42 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/331.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/331.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/331.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:58:43 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:58:43 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:58:43 MarkDuplicates Start of doWork freeMemory: 72064368; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 18:58:43 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:58:43 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:58:46 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig13791:2,742
INFO 2021-03-14 18:58:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:47 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig26736:13,010
INFO 2021-03-14 18:58:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:49 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig33631:69,518
INFO 2021-03-14 18:58:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:50 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig44024:2,507
INFO 2021-03-14 18:58:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:52 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,466
INFO 2021-03-14 18:58:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:54 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig64644:275
INFO 2021-03-14 18:58:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:55 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig69234:11,734
INFO 2021-03-14 18:58:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:57 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig70638:19,290
INFO 2021-03-14 18:58:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:58:58 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig85486:2,602
INFO 2021-03-14 18:58:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:00 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,395
INFO 2021-03-14 18:59:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:02 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig137005:2,570
INFO 2021-03-14 18:59:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:03 MarkDuplicates Read 11773130 records. 0 pairs never matched.
INFO 2021-03-14 18:59:03 MarkDuplicates After buildSortedReadEndLists freeMemory: 994592904; totalMemory: 1516240896; maxMemory: 4294967296
INFO 2021-03-14 18:59:03 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:59:03 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:59:03 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:59:04 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:59:04 MarkDuplicates After generateDuplicateIndexes freeMemory: 1458022088; totalMemory: 2591031296; maxMemory: 4294967296
INFO 2021-03-14 18:59:04 MarkDuplicates Marking 3329870 records as duplicates.
INFO 2021-03-14 18:59:04 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:59:05 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:59:24 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:59:24 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:59:24 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:59:25 MarkDuplicates Before output close freeMemory: 264886040; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:59:25 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:59:25 MarkDuplicates After output close freeMemory: 266101536; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 18:59:25 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.71 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/331.sorted.bam O=../qc-processing/bowtie/deduplicated/331.dedup.bam M=../qc-processing/bowtie/deduplicated/331.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:59:26 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/332.sorted.bam -O ../qc-processing/bowtie/deduplicated/332.dedup.bam -M ../qc-processing/bowtie/deduplicated/332.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:59:26.925 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:59:27 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/332.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/332.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/332.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:59:27 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:59:27 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:59:27 MarkDuplicates Start of doWork freeMemory: 67873840; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:59:27 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:59:27 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 18:59:30 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig9826:4,742
INFO 2021-03-14 18:59:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:31 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig27063:3,916
INFO 2021-03-14 18:59:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:33 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig36412:566
INFO 2021-03-14 18:59:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:35 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig52139:6,683
INFO 2021-03-14 18:59:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:36 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig67549:4,154
INFO 2021-03-14 18:59:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:38 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig83429:7,741
INFO 2021-03-14 18:59:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:40 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig111435:3,659
INFO 2021-03-14 18:59:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:41 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig519002:3,179
INFO 2021-03-14 18:59:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 18:59:42 MarkDuplicates Read 8071252 records. 0 pairs never matched.
INFO 2021-03-14 18:59:42 MarkDuplicates After buildSortedReadEndLists freeMemory: 1059719624; totalMemory: 1509949440; maxMemory: 4294967296
INFO 2021-03-14 18:59:42 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 18:59:42 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 18:59:42 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 18:59:43 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 18:59:43 MarkDuplicates After generateDuplicateIndexes freeMemory: 1451730632; totalMemory: 2584739840; maxMemory: 4294967296
INFO 2021-03-14 18:59:43 MarkDuplicates Marking 2109331 records as duplicates.
INFO 2021-03-14 18:59:43 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 18:59:44 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 18:59:57 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 18:59:57 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 18:59:57 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 18:59:57 MarkDuplicates Before output close freeMemory: 263869760; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 18:59:57 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 18:59:58 MarkDuplicates After output close freeMemory: 262955928; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 18:59:58 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.52 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/332.sorted.bam O=../qc-processing/bowtie/deduplicated/332.dedup.bam M=../qc-processing/bowtie/deduplicated/332.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 18:59:59 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/333.sorted.bam -O ../qc-processing/bowtie/deduplicated/333.dedup.bam -M ../qc-processing/bowtie/deduplicated/333.dup_metrics.txt -REMOVE_DUPLICATES true
**********
17:59:59.761 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 18:59:59 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/333.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/333.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/333.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 6:59:59 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 18:59:59 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 18:59:59 MarkDuplicates Start of doWork freeMemory: 67872448; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 18:59:59 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 18:59:59 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:00:03 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 3s. Last read position: Contig15441:4,178
INFO 2021-03-14 19:00:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:05 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig28748:5,682
INFO 2021-03-14 19:00:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:07 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig37981:582
INFO 2021-03-14 19:00:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:08 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig54291:841
INFO 2021-03-14 19:00:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:10 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig68908:704
INFO 2021-03-14 19:00:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:12 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig85291:7,988
INFO 2021-03-14 19:00:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:13 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig123544:573
INFO 2021-03-14 19:00:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:15 MarkDuplicates Read 7938296 records. 0 pairs never matched.
INFO 2021-03-14 19:00:15 MarkDuplicates After buildSortedReadEndLists freeMemory: 1119354040; totalMemory: 1593835520; maxMemory: 4294967296
INFO 2021-03-14 19:00:15 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:00:15 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:00:15 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:00:16 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:00:16 MarkDuplicates After generateDuplicateIndexes freeMemory: 1535617824; totalMemory: 2668625920; maxMemory: 4294967296
INFO 2021-03-14 19:00:16 MarkDuplicates Marking 2418757 records as duplicates.
INFO 2021-03-14 19:00:16 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:00:17 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:00:30 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:00:30 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:00:30 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:00:30 MarkDuplicates Before output close freeMemory: 271598384; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 19:00:30 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:00:30 MarkDuplicates After output close freeMemory: 258762560; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:00:30 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.51 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/333.sorted.bam O=../qc-processing/bowtie/deduplicated/333.dedup.bam M=../qc-processing/bowtie/deduplicated/333.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:00:32 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/334.sorted.bam -O ../qc-processing/bowtie/deduplicated/334.dedup.bam -M ../qc-processing/bowtie/deduplicated/334.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:00:32.320 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:00:32 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/334.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/334.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/334.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:00:32 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:00:32 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:00:32 MarkDuplicates Start of doWork freeMemory: 75208936; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:00:32 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:00:32 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:00:35 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig12011:1,177
INFO 2021-03-14 19:00:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:37 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig26383:9,123
INFO 2021-03-14 19:00:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:38 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig34438:2,014
INFO 2021-03-14 19:00:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:40 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig47434:99
INFO 2021-03-14 19:00:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:42 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:12,102
INFO 2021-03-14 19:00:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:43 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68908:1,051
INFO 2021-03-14 19:00:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:45 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig76034:996
INFO 2021-03-14 19:00:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:47 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,745
INFO 2021-03-14 19:00:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:48 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig121681:3,679
INFO 2021-03-14 19:00:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:00:50 MarkDuplicates Read 9992160 records. 0 pairs never matched.
INFO 2021-03-14 19:00:50 MarkDuplicates After buildSortedReadEndLists freeMemory: 1089436912; totalMemory: 1558183936; maxMemory: 4294967296
INFO 2021-03-14 19:00:50 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:00:51 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:00:51 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:00:51 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:00:51 MarkDuplicates After generateDuplicateIndexes freeMemory: 1499965104; totalMemory: 2632974336; maxMemory: 4294967296
INFO 2021-03-14 19:00:51 MarkDuplicates Marking 2504699 records as duplicates.
INFO 2021-03-14 19:00:51 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:00:52 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:01:09 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:01:09 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:01:09 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:01:10 MarkDuplicates Before output close freeMemory: 266994512; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:01:10 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:01:10 MarkDuplicates After output close freeMemory: 258761104; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:01:10 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.63 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/334.sorted.bam O=../qc-processing/bowtie/deduplicated/334.dedup.bam M=../qc-processing/bowtie/deduplicated/334.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:01:12 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/335.sorted.bam -O ../qc-processing/bowtie/deduplicated/335.dedup.bam -M ../qc-processing/bowtie/deduplicated/335.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:01:12.081 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:01:12 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/335.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/335.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/335.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:01:12 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:01:12 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:01:12 MarkDuplicates Start of doWork freeMemory: 67868296; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:01:12 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:01:12 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:01:15 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15609:2,433
INFO 2021-03-14 19:01:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:17 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig28652:3,681
INFO 2021-03-14 19:01:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:18 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig37656:21,265
INFO 2021-03-14 19:01:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:20 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig54936:592
INFO 2021-03-14 19:01:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:22 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,784
INFO 2021-03-14 19:01:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:23 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,944
INFO 2021-03-14 19:01:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:25 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig130572:7
INFO 2021-03-14 19:01:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:26 MarkDuplicates Read 7774286 records. 0 pairs never matched.
INFO 2021-03-14 19:01:27 MarkDuplicates After buildSortedReadEndLists freeMemory: 1116295736; totalMemory: 1587544064; maxMemory: 4294967296
INFO 2021-03-14 19:01:27 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:01:27 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:01:27 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:01:28 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:01:28 MarkDuplicates After generateDuplicateIndexes freeMemory: 1529324696; totalMemory: 2662334464; maxMemory: 4294967296
INFO 2021-03-14 19:01:28 MarkDuplicates Marking 2325035 records as duplicates.
INFO 2021-03-14 19:01:28 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:01:29 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:01:41 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:01:41 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:01:41 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:01:41 MarkDuplicates Before output close freeMemory: 272358648; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 19:01:41 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:01:41 MarkDuplicates After output close freeMemory: 258759792; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:01:41 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.50 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/335.sorted.bam O=../qc-processing/bowtie/deduplicated/335.dedup.bam M=../qc-processing/bowtie/deduplicated/335.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:01:43 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/336.sorted.bam -O ../qc-processing/bowtie/deduplicated/336.dedup.bam -M ../qc-processing/bowtie/deduplicated/336.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:01:43.487 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:01:43 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/336.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/336.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/336.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:01:43 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:01:43 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:01:43 MarkDuplicates Start of doWork freeMemory: 72065440; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:01:43 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:01:43 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:01:46 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4198:13,228
INFO 2021-03-14 19:01:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:48 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig20601:4,530
INFO 2021-03-14 19:01:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:50 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30130:1,252
INFO 2021-03-14 19:01:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:51 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35774:1,112
INFO 2021-03-14 19:01:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:53 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig46328:8,937
INFO 2021-03-14 19:01:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:54 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig57779:4,170
INFO 2021-03-14 19:01:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:56 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig69234:11,734
INFO 2021-03-14 19:01:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:01:58 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig85156:27,318
INFO 2021-03-14 19:01:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:00 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig104373:1,317
INFO 2021-03-14 19:02:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:01 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig149299:9,260
INFO 2021-03-14 19:02:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:02 MarkDuplicates Read 10709147 records. 0 pairs never matched.
INFO 2021-03-14 19:02:02 MarkDuplicates After buildSortedReadEndLists freeMemory: 1003333224; totalMemory: 1545601024; maxMemory: 4294967296
INFO 2021-03-14 19:02:02 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:02:03 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:02:03 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:02:04 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:02:04 MarkDuplicates After generateDuplicateIndexes freeMemory: 1487379392; totalMemory: 2620391424; maxMemory: 4294967296
INFO 2021-03-14 19:02:04 MarkDuplicates Marking 2982703 records as duplicates.
INFO 2021-03-14 19:02:04 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:02:05 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:02:22 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:02:22 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:02:22 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:02:23 MarkDuplicates Before output close freeMemory: 270990824; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 19:02:23 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:02:23 MarkDuplicates After output close freeMemory: 262952576; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:02:23 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.66 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/336.sorted.bam O=../qc-processing/bowtie/deduplicated/336.dedup.bam M=../qc-processing/bowtie/deduplicated/336.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:02:25 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/337.sorted.bam -O ../qc-processing/bowtie/deduplicated/337.dedup.bam -M ../qc-processing/bowtie/deduplicated/337.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:02:25.041 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:02:25 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/337.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/337.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/337.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:02:25 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:02:25 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:02:25 MarkDuplicates Start of doWork freeMemory: 72063224; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:02:25 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:02:25 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:02:28 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2268:9,193
INFO 2021-03-14 19:02:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:29 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig10742:9,617
INFO 2021-03-14 19:02:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:31 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig21438:15,911
INFO 2021-03-14 19:02:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:33 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig28264:10,588
INFO 2021-03-14 19:02:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:34 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig33739:575
INFO 2021-03-14 19:02:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:36 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig37588:13,721
INFO 2021-03-14 19:02:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:37 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig45152:5,695
INFO 2021-03-14 19:02:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:39 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig54565:4,382
INFO 2021-03-14 19:02:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:40 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig61796:10,932
INFO 2021-03-14 19:02:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:42 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig68908:744
INFO 2021-03-14 19:02:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:44 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig74454:3,329
INFO 2021-03-14 19:02:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:45 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig90977:7,694
INFO 2021-03-14 19:02:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:47 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig111970:3,500
INFO 2021-03-14 19:02:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:49 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:23s. Time for last 1,000,000: 1s. Last read position: Contig147865:1,622
INFO 2021-03-14 19:02:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:50 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 1s. Last read position: Contig568337:1,178
INFO 2021-03-14 19:02:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:02:50 MarkDuplicates Read 15087210 records. 0 pairs never matched.
INFO 2021-03-14 19:02:51 MarkDuplicates After buildSortedReadEndLists freeMemory: 703574168; totalMemory: 1460666368; maxMemory: 4294967296
INFO 2021-03-14 19:02:51 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:02:51 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:02:51 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:02:52 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:02:53 MarkDuplicates After generateDuplicateIndexes freeMemory: 1402446360; totalMemory: 2535456768; maxMemory: 4294967296
INFO 2021-03-14 19:02:53 MarkDuplicates Marking 5440155 records as duplicates.
INFO 2021-03-14 19:02:53 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:02:54 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:03:16 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:03:16 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:03:16 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:03:16 MarkDuplicates Before output close freeMemory: 264843376; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:03:16 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:03:16 MarkDuplicates After output close freeMemory: 258760288; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:03:16 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.86 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/337.sorted.bam O=../qc-processing/bowtie/deduplicated/337.dedup.bam M=../qc-processing/bowtie/deduplicated/337.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:03:19 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/338.sorted.bam -O ../qc-processing/bowtie/deduplicated/338.dedup.bam -M ../qc-processing/bowtie/deduplicated/338.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:03:19.039 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:03:19 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/338.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/338.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/338.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:03:19 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:03:19 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:03:19 MarkDuplicates Start of doWork freeMemory: 75208648; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:03:19 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:03:19 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:03:22 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4522:6,868
INFO 2021-03-14 19:03:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:24 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig21128:6,809
INFO 2021-03-14 19:03:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:25 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30643:3,175
INFO 2021-03-14 19:03:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:27 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig37346:5,301
INFO 2021-03-14 19:03:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:29 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig48506:4,790
INFO 2021-03-14 19:03:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:30 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig56693:13,894
INFO 2021-03-14 19:03:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:32 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,790
INFO 2021-03-14 19:03:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:33 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig78855:5,070
INFO 2021-03-14 19:03:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:35 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig103849:2,598
INFO 2021-03-14 19:03:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:37 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig141397:7,560
INFO 2021-03-14 19:03:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:03:38 MarkDuplicates Read 10872680 records. 0 pairs never matched.
INFO 2021-03-14 19:03:38 MarkDuplicates After buildSortedReadEndLists freeMemory: 1207798896; totalMemory: 1735393280; maxMemory: 4294967296
INFO 2021-03-14 19:03:38 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:03:39 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:03:39 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:03:40 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:03:41 MarkDuplicates After generateDuplicateIndexes freeMemory: 1677173264; totalMemory: 2810183680; maxMemory: 4294967296
INFO 2021-03-14 19:03:41 MarkDuplicates Marking 2652536 records as duplicates.
INFO 2021-03-14 19:03:41 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:03:42 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:04:00 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:04:00 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:04:00 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:04:00 MarkDuplicates Before output close freeMemory: 257904400; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:04:00 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:04:00 MarkDuplicates After output close freeMemory: 255614112; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 19:04:00 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.70 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/338.sorted.bam O=../qc-processing/bowtie/deduplicated/338.dedup.bam M=../qc-processing/bowtie/deduplicated/338.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:04:02 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/339.sorted.bam -O ../qc-processing/bowtie/deduplicated/339.dedup.bam -M ../qc-processing/bowtie/deduplicated/339.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:04:02.598 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:04:02 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/339.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/339.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/339.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:04:02 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:04:02 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:04:02 MarkDuplicates Start of doWork freeMemory: 72063264; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:04:02 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:04:02 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:04:06 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3541:386
INFO 2021-03-14 19:04:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:07 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig19099:1,618
INFO 2021-03-14 19:04:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:09 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig27538:26,222
INFO 2021-03-14 19:04:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:10 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35213:220
INFO 2021-03-14 19:04:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:12 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig42669:5,728
INFO 2021-03-14 19:04:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:13 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig54957:22,764
INFO 2021-03-14 19:04:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:15 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig63443:2,571
INFO 2021-03-14 19:04:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:17 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,891
INFO 2021-03-14 19:04:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:18 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig86737:11,547
INFO 2021-03-14 19:04:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:20 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig109801:8,426
INFO 2021-03-14 19:04:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:21 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig163120:116
INFO 2021-03-14 19:04:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:22 MarkDuplicates Read 11666925 records. 0 pairs never matched.
INFO 2021-03-14 19:04:23 MarkDuplicates After buildSortedReadEndLists freeMemory: 912457992; totalMemory: 1488977920; maxMemory: 4294967296
INFO 2021-03-14 19:04:23 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:04:23 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:04:23 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:04:24 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:04:24 MarkDuplicates After generateDuplicateIndexes freeMemory: 1430757368; totalMemory: 2563768320; maxMemory: 4294967296
INFO 2021-03-14 19:04:24 MarkDuplicates Marking 3304830 records as duplicates.
INFO 2021-03-14 19:04:24 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:04:25 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:04:43 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:04:43 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:04:43 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:04:43 MarkDuplicates Before output close freeMemory: 271568320; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 19:04:43 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:04:43 MarkDuplicates After output close freeMemory: 258759264; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:04:43 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.69 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/339.sorted.bam O=../qc-processing/bowtie/deduplicated/339.dedup.bam M=../qc-processing/bowtie/deduplicated/339.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:04:45 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/34.sorted.bam -O ../qc-processing/bowtie/deduplicated/34..dedup.bam -M ../qc-processing/bowtie/deduplicated/34..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:04:45.639 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:04:45 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/34.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/34..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/34..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:04:45 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:04:45 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:04:45 MarkDuplicates Start of doWork freeMemory: 67868312; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:04:45 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:04:45 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:04:49 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 3s. Last read position: Contig16118:39,828
INFO 2021-03-14 19:04:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:51 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig22200:3,529
INFO 2021-03-14 19:04:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:53 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,735
INFO 2021-03-14 19:04:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:54 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35487:29,237
INFO 2021-03-14 19:04:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:56 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig49089:16,038
INFO 2021-03-14 19:04:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:57 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,301
INFO 2021-03-14 19:04:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:04:59 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig56910:792
INFO 2021-03-14 19:04:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:00 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig68037:820
INFO 2021-03-14 19:05:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:02 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,639
INFO 2021-03-14 19:05:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:04 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig83174:2,419
INFO 2021-03-14 19:05:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:06 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig88342:3,464
INFO 2021-03-14 19:05:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:07 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,477
INFO 2021-03-14 19:05:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:09 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig137636:6,616
INFO 2021-03-14 19:05:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:10 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 1s. Last read position: Contig686969:1,564
INFO 2021-03-14 19:05:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:10 MarkDuplicates Read 14027016 records. 0 pairs never matched.
INFO 2021-03-14 19:05:11 MarkDuplicates After buildSortedReadEndLists freeMemory: 887629656; totalMemory: 1440743424; maxMemory: 4294967296
INFO 2021-03-14 19:05:11 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:05:11 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:05:11 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:05:12 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:05:12 MarkDuplicates After generateDuplicateIndexes freeMemory: 1382523632; totalMemory: 2515533824; maxMemory: 4294967296
INFO 2021-03-14 19:05:12 MarkDuplicates Marking 4339647 records as duplicates.
INFO 2021-03-14 19:05:12 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:05:13 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:05:35 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:05:35 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:05:35 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:05:36 MarkDuplicates Before output close freeMemory: 264191760; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:05:36 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:05:36 MarkDuplicates After output close freeMemory: 258760376; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:05:36 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.84 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/34.sorted.bam O=../qc-processing/bowtie/deduplicated/34..dedup.bam M=../qc-processing/bowtie/deduplicated/34..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:05:37 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/341.sorted.bam -O ../qc-processing/bowtie/deduplicated/341.dedup.bam -M ../qc-processing/bowtie/deduplicated/341.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:05:37.886 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:05:38 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/341.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/341.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/341.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:05:38 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:05:38 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:05:38 MarkDuplicates Start of doWork freeMemory: 75212504; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:05:38 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:05:38 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:05:41 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig8839:1,517
INFO 2021-03-14 19:05:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:43 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig27044:9,626
INFO 2021-03-14 19:05:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:44 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35213:2,490
INFO 2021-03-14 19:05:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:46 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig48729:2,828
INFO 2021-03-14 19:05:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:47 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig63927:1,155
INFO 2021-03-14 19:05:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:49 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig75617:1,510
INFO 2021-03-14 19:05:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:50 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:05:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:52 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig171609:1,761
INFO 2021-03-14 19:05:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:05:53 MarkDuplicates Read 8398656 records. 0 pairs never matched.
INFO 2021-03-14 19:05:53 MarkDuplicates After buildSortedReadEndLists freeMemory: 1125705768; totalMemory: 1604321280; maxMemory: 4294967296
INFO 2021-03-14 19:05:53 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:05:53 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:05:53 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:05:54 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:05:54 MarkDuplicates After generateDuplicateIndexes freeMemory: 1546102344; totalMemory: 2679111680; maxMemory: 4294967296
INFO 2021-03-14 19:05:54 MarkDuplicates Marking 2296026 records as duplicates.
INFO 2021-03-14 19:05:54 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:05:55 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:06:11 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:06:11 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:06:11 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:06:12 MarkDuplicates Before output close freeMemory: 266096808; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:06:12 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:06:12 MarkDuplicates After output close freeMemory: 258761640; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:06:12 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.57 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/341.sorted.bam O=../qc-processing/bowtie/deduplicated/341.dedup.bam M=../qc-processing/bowtie/deduplicated/341.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:06:14 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/342.sorted.bam -O ../qc-processing/bowtie/deduplicated/342.dedup.bam -M ../qc-processing/bowtie/deduplicated/342.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:06:14.074 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:06:14 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/342.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/342.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/342.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:06:14 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:06:14 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:06:14 MarkDuplicates Start of doWork freeMemory: 75207888; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:06:14 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:06:14 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:06:17 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2870:6,266
INFO 2021-03-14 19:06:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:19 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig16972:8,967
INFO 2021-03-14 19:06:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:20 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig24377:29,697
INFO 2021-03-14 19:06:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:22 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,671
INFO 2021-03-14 19:06:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:23 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig35849:9,012
INFO 2021-03-14 19:06:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:25 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig42785:5,344
INFO 2021-03-14 19:06:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:26 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig52854:16,045
INFO 2021-03-14 19:06:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:28 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig60992:1,575
INFO 2021-03-14 19:06:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:30 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,921
INFO 2021-03-14 19:06:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:31 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig78849:13,991
INFO 2021-03-14 19:06:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:33 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,750
INFO 2021-03-14 19:06:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:34 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig124836:2,052
INFO 2021-03-14 19:06:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:36 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig181638:2,112
INFO 2021-03-14 19:06:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:06:37 MarkDuplicates Read 13558049 records. 0 pairs never matched.
INFO 2021-03-14 19:06:37 MarkDuplicates After buildSortedReadEndLists freeMemory: 886071896; totalMemory: 1560281088; maxMemory: 4294967296
INFO 2021-03-14 19:06:37 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:06:37 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:06:37 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:06:38 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:06:39 MarkDuplicates After generateDuplicateIndexes freeMemory: 1502060072; totalMemory: 2635071488; maxMemory: 4294967296
INFO 2021-03-14 19:06:39 MarkDuplicates Marking 4429135 records as duplicates.
INFO 2021-03-14 19:06:39 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:06:39 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:06:59 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:06:59 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:06:59 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:07:00 MarkDuplicates Before output close freeMemory: 264690256; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:07:00 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:07:00 MarkDuplicates After output close freeMemory: 266099080; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 19:07:00 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.77 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/342.sorted.bam O=../qc-processing/bowtie/deduplicated/342.dedup.bam M=../qc-processing/bowtie/deduplicated/342.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:07:01 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/343.sorted.bam -O ../qc-processing/bowtie/deduplicated/343.dedup.bam -M ../qc-processing/bowtie/deduplicated/343.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:07:01.969 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:07:02 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/343.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/343.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/343.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:07:02 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:07:02 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:07:02 MarkDuplicates Start of doWork freeMemory: 72062992; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:07:02 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:07:02 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:07:05 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3112:1,249
INFO 2021-03-14 19:07:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:06 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig18609:16,597
INFO 2021-03-14 19:07:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:08 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig26808:20,746
INFO 2021-03-14 19:07:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:10 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig34501:5,667
INFO 2021-03-14 19:07:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:11 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig41518:3,084
INFO 2021-03-14 19:07:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:13 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig52503:7,310
INFO 2021-03-14 19:07:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:14 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig64198:1,271
INFO 2021-03-14 19:07:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:16 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig76529:556
INFO 2021-03-14 19:07:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:18 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig101000:5,412
INFO 2021-03-14 19:07:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:19 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig103886:900
INFO 2021-03-14 19:07:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:21 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig145734:2,268
INFO 2021-03-14 19:07:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:22 MarkDuplicates Read 11884225 records. 0 pairs never matched.
INFO 2021-03-14 19:07:22 MarkDuplicates After buildSortedReadEndLists freeMemory: 1008765944; totalMemory: 1605369856; maxMemory: 4294967296
INFO 2021-03-14 19:07:22 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:07:23 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:07:23 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:07:24 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:07:24 MarkDuplicates After generateDuplicateIndexes freeMemory: 1547150856; totalMemory: 2680160256; maxMemory: 4294967296
INFO 2021-03-14 19:07:24 MarkDuplicates Marking 3050107 records as duplicates.
INFO 2021-03-14 19:07:24 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:07:25 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:07:44 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:07:44 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:07:44 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:07:44 MarkDuplicates Before output close freeMemory: 261412144; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:07:44 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:07:44 MarkDuplicates After output close freeMemory: 262955248; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:07:44 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.72 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/343.sorted.bam O=../qc-processing/bowtie/deduplicated/343.dedup.bam M=../qc-processing/bowtie/deduplicated/343.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:07:46 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/344.sorted.bam -O ../qc-processing/bowtie/deduplicated/344.dedup.bam -M ../qc-processing/bowtie/deduplicated/344.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:07:46.713 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:07:46 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/344.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/344.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/344.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:07:47 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:07:47 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:07:47 MarkDuplicates Start of doWork freeMemory: 72061680; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:07:47 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:07:47 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:07:50 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig2891:2,342
INFO 2021-03-14 19:07:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:52 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig18486:16,022
INFO 2021-03-14 19:07:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:53 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig26204:23,722
INFO 2021-03-14 19:07:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:55 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,720
INFO 2021-03-14 19:07:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:56 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig36557:12,137
INFO 2021-03-14 19:07:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:07:58 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig44640:3,717
INFO 2021-03-14 19:07:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:00 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,649
INFO 2021-03-14 19:08:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:01 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig56693:16,401
INFO 2021-03-14 19:08:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:03 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig66255:1,482
INFO 2021-03-14 19:08:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:04 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig69234:11,761
INFO 2021-03-14 19:08:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:06 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig70638:19,186
INFO 2021-03-14 19:08:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:07 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig81563:3,618
INFO 2021-03-14 19:08:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:08 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig98935:2,018
INFO 2021-03-14 19:08:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:10 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig109802:560
INFO 2021-03-14 19:08:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:12 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 1s. Last read position: Contig149236:96
INFO 2021-03-14 19:08:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:13 MarkDuplicates Read 15876307 records. 0 pairs never matched.
INFO 2021-03-14 19:08:13 MarkDuplicates After buildSortedReadEndLists freeMemory: 703387256; totalMemory: 1390411776; maxMemory: 4294967296
INFO 2021-03-14 19:08:13 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:08:14 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:08:14 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:08:15 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:08:15 MarkDuplicates After generateDuplicateIndexes freeMemory: 1332189856; totalMemory: 2465202176; maxMemory: 4294967296
INFO 2021-03-14 19:08:15 MarkDuplicates Marking 4709300 records as duplicates.
INFO 2021-03-14 19:08:15 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:08:16 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:08:37 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:21s. Time for last 10,000,000: 21s. Last read position: Contig123544:573
INFO 2021-03-14 19:08:40 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:08:40 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:08:40 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:08:40 MarkDuplicates Before output close freeMemory: 265471328; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:08:40 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:08:40 MarkDuplicates After output close freeMemory: 262952064; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:08:40 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.90 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/344.sorted.bam O=../qc-processing/bowtie/deduplicated/344.dedup.bam M=../qc-processing/bowtie/deduplicated/344.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:08:42 MarkDuplicates
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/345.sorted.bam -O ../qc-processing/bowtie/deduplicated/345.dedup.bam -M ../qc-processing/bowtie/deduplicated/345.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:08:42.483 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:08:42 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/345.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/345.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/345.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:08:42 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:08:42 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:08:42 MarkDuplicates Start of doWork freeMemory: 75210904; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:08:42 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:08:42 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:08:45 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig7085:163
INFO 2021-03-14 19:08:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:47 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig25068:12,662
INFO 2021-03-14 19:08:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:48 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35213:854
INFO 2021-03-14 19:08:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:50 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig47067:8,692
INFO 2021-03-14 19:08:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:52 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig61796:10,932
INFO 2021-03-14 19:08:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:53 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig77522:6,757
INFO 2021-03-14 19:08:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:55 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig111954:463
INFO 2021-03-14 19:08:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:56 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig211325:446
INFO 2021-03-14 19:08:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:08:57 MarkDuplicates Read 8198694 records. 0 pairs never matched.
INFO 2021-03-14 19:08:57 MarkDuplicates After buildSortedReadEndLists freeMemory: 1160393168; totalMemory: 1650458624; maxMemory: 4294967296
INFO 2021-03-14 19:08:57 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:08:57 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:08:57 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:08:58 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:08:58 MarkDuplicates After generateDuplicateIndexes freeMemory: 1592240624; totalMemory: 2725249024; maxMemory: 4294967296
INFO 2021-03-14 19:08:58 MarkDuplicates Marking 2298868 records as duplicates.
INFO 2021-03-14 19:08:58 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:08:59 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:09:12 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:09:12 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:09:12 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:09:13 MarkDuplicates Before output close freeMemory: 262181984; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:09:13 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:09:13 MarkDuplicates After output close freeMemory: 258762048; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:09:13 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.51 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/345.sorted.bam O=../qc-processing/bowtie/deduplicated/345.dedup.bam M=../qc-processing/bowtie/deduplicated/345.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:09:14 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/346.sorted.bam -O ../qc-processing/bowtie/deduplicated/346.dedup.bam -M ../qc-processing/bowtie/deduplicated/346.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:09:14.894 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:09:14 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/346.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/346.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/346.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:09:15 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:09:15 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:09:15 MarkDuplicates Start of doWork freeMemory: 72062032; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:09:15 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:09:15 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:09:18 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5628:2,433
INFO 2021-03-14 19:09:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:19 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig23182:22,794
INFO 2021-03-14 19:09:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:21 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig33124:3,757
INFO 2021-03-14 19:09:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:23 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig42187:42,959
INFO 2021-03-14 19:09:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:24 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56152:920
INFO 2021-03-14 19:09:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:26 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68037:506
INFO 2021-03-14 19:09:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:27 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig78298:4,973
INFO 2021-03-14 19:09:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:29 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:09:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:30 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig149666:3,390
INFO 2021-03-14 19:09:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:32 MarkDuplicates Read 9633922 records. 0 pairs never matched.
INFO 2021-03-14 19:09:32 MarkDuplicates After buildSortedReadEndLists freeMemory: 1058808968; totalMemory: 1569718272; maxMemory: 4294967296
INFO 2021-03-14 19:09:32 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:09:32 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:09:32 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:09:33 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:09:33 MarkDuplicates After generateDuplicateIndexes freeMemory: 1511497760; totalMemory: 2644508672; maxMemory: 4294967296
INFO 2021-03-14 19:09:33 MarkDuplicates Marking 2450085 records as duplicates.
INFO 2021-03-14 19:09:33 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:09:34 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:09:50 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:09:50 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:09:50 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:09:50 MarkDuplicates Before output close freeMemory: 2531845664; totalMemory: 2644508672; maxMemory: 4294967296
INFO 2021-03-14 19:09:50 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:09:50 MarkDuplicates After output close freeMemory: 262954344; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:09:50 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.59 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/346.sorted.bam O=../qc-processing/bowtie/deduplicated/346.dedup.bam M=../qc-processing/bowtie/deduplicated/346.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:09:52 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/347.sorted.bam -O ../qc-processing/bowtie/deduplicated/347.dedup.bam -M ../qc-processing/bowtie/deduplicated/347.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:09:52.227 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:09:52 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/347.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/347.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/347.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:09:52 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:09:52 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:09:52 MarkDuplicates Start of doWork freeMemory: 75209480; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:09:52 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:09:52 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:09:55 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5475:1,646
INFO 2021-03-14 19:09:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:57 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig22598:1,468
INFO 2021-03-14 19:09:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:09:59 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig32497:7,675
INFO 2021-03-14 19:09:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:00 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig40564:1,249
INFO 2021-03-14 19:10:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:02 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig54243:26,029
INFO 2021-03-14 19:10:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:03 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68037:802
INFO 2021-03-14 19:10:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:05 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig81791:23,527
INFO 2021-03-14 19:10:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:07 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig105282:2,949
INFO 2021-03-14 19:10:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:08 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig171204:3,385
INFO 2021-03-14 19:10:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:09 MarkDuplicates Read 9472397 records. 0 pairs never matched.
INFO 2021-03-14 19:10:09 MarkDuplicates After buildSortedReadEndLists freeMemory: 798784824; totalMemory: 1345323008; maxMemory: 4294967296
INFO 2021-03-14 19:10:09 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:10:10 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:10:10 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:10:11 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:10:11 MarkDuplicates After generateDuplicateIndexes freeMemory: 1287103680; totalMemory: 2420113408; maxMemory: 4294967296
INFO 2021-03-14 19:10:11 MarkDuplicates Marking 3067496 records as duplicates.
INFO 2021-03-14 19:10:11 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:10:12 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:10:27 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:10:27 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:10:27 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:10:27 MarkDuplicates Before output close freeMemory: 715648704; totalMemory: 2420113408; maxMemory: 4294967296
INFO 2021-03-14 19:10:27 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:10:27 MarkDuplicates After output close freeMemory: 262954912; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:10:27 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.59 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/347.sorted.bam O=../qc-processing/bowtie/deduplicated/347.dedup.bam M=../qc-processing/bowtie/deduplicated/347.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:10:29 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/348.sorted.bam -O ../qc-processing/bowtie/deduplicated/348.dedup.bam -M ../qc-processing/bowtie/deduplicated/348.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:10:29.066 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:10:29 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/348.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/348.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/348.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:10:29 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:10:29 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:10:29 MarkDuplicates Start of doWork freeMemory: 75211600; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:10:29 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:10:29 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:10:32 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3849:6,634
INFO 2021-03-14 19:10:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:34 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig20021:7,008
INFO 2021-03-14 19:10:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:35 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig28839:522
INFO 2021-03-14 19:10:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:37 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35213:861
INFO 2021-03-14 19:10:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:38 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig42679:23,804
INFO 2021-03-14 19:10:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:40 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig55181:5,897
INFO 2021-03-14 19:10:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:42 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig61245:2,268
INFO 2021-03-14 19:10:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:43 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig69234:11,777
INFO 2021-03-14 19:10:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:45 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig80162:3,345
INFO 2021-03-14 19:10:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:46 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig101302:2,438
INFO 2021-03-14 19:10:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:48 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig130956:12,390
INFO 2021-03-14 19:10:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:50 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig269830:105
INFO 2021-03-14 19:10:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:10:50 MarkDuplicates Read 12212793 records. 0 pairs never matched.
INFO 2021-03-14 19:10:50 MarkDuplicates After buildSortedReadEndLists freeMemory: 1300395568; totalMemory: 1886388224; maxMemory: 4294967296
INFO 2021-03-14 19:10:50 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:10:51 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:10:51 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:10:51 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:10:52 MarkDuplicates After generateDuplicateIndexes freeMemory: 1828168120; totalMemory: 2961178624; maxMemory: 4294967296
INFO 2021-03-14 19:10:52 MarkDuplicates Marking 3430130 records as duplicates.
INFO 2021-03-14 19:10:52 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:10:52 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:11:12 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:11:12 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:11:12 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:11:12 MarkDuplicates Before output close freeMemory: 268165272; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:11:12 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:11:12 MarkDuplicates After output close freeMemory: 262954400; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:11:12 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.73 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/348.sorted.bam O=../qc-processing/bowtie/deduplicated/348.dedup.bam M=../qc-processing/bowtie/deduplicated/348.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:11:14 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/349.sorted.bam -O ../qc-processing/bowtie/deduplicated/349.dedup.bam -M ../qc-processing/bowtie/deduplicated/349.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:11:14.607 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:11:14 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/349.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/349.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/349.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:11:14 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:11:14 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:11:14 MarkDuplicates Start of doWork freeMemory: 75209832; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:11:14 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:11:14 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:11:17 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig7517:6,298
INFO 2021-03-14 19:11:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:19 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig25175:9,283
INFO 2021-03-14 19:11:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:21 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig34897:6,022
INFO 2021-03-14 19:11:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:22 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig46478:7,244
INFO 2021-03-14 19:11:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:24 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig57693:20,359
INFO 2021-03-14 19:11:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:25 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig69525:5,325
INFO 2021-03-14 19:11:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:27 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig89106:10,211
INFO 2021-03-14 19:11:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:29 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig131920:3,931
INFO 2021-03-14 19:11:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:30 MarkDuplicates Read 8897134 records. 0 pairs never matched.
INFO 2021-03-14 19:11:30 MarkDuplicates After buildSortedReadEndLists freeMemory: 1168837504; totalMemory: 1664090112; maxMemory: 4294967296
INFO 2021-03-14 19:11:30 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:11:31 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:11:31 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:11:31 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:11:31 MarkDuplicates After generateDuplicateIndexes freeMemory: 1605869968; totalMemory: 2738880512; maxMemory: 4294967296
INFO 2021-03-14 19:11:31 MarkDuplicates Marking 2519877 records as duplicates.
INFO 2021-03-14 19:11:31 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:11:32 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:11:46 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:11:46 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:11:46 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:11:46 MarkDuplicates Before output close freeMemory: 265055072; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:11:46 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:11:46 MarkDuplicates After output close freeMemory: 258760304; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:11:46 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.54 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/349.sorted.bam O=../qc-processing/bowtie/deduplicated/349.dedup.bam M=../qc-processing/bowtie/deduplicated/349.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:11:48 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/35.sorted.bam -O ../qc-processing/bowtie/deduplicated/35..dedup.bam -M ../qc-processing/bowtie/deduplicated/35..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:11:48.481 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:11:48 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/35.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/35..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/35..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:11:48 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:11:48 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:11:48 MarkDuplicates Start of doWork freeMemory: 72060416; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:11:48 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:11:48 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:11:51 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig42187:43,034
INFO 2021-03-14 19:11:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:53 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig70638:19,331
INFO 2021-03-14 19:11:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:54 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:11:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:55 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig686969:1,573
INFO 2021-03-14 19:11:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:11:55 MarkDuplicates Read 4008896 records. 0 pairs never matched.
INFO 2021-03-14 19:11:56 MarkDuplicates After buildSortedReadEndLists freeMemory: 698867232; totalMemory: 993001472; maxMemory: 4294967296
INFO 2021-03-14 19:11:56 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:11:56 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:11:56 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:11:56 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:11:57 MarkDuplicates After generateDuplicateIndexes freeMemory: 934778984; totalMemory: 2067791872; maxMemory: 4294967296
INFO 2021-03-14 19:11:57 MarkDuplicates Marking 1350158 records as duplicates.
INFO 2021-03-14 19:11:57 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:11:57 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:12:04 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:12:04 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:12:04 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:12:04 MarkDuplicates Before output close freeMemory: 271488712; totalMemory: 381681664; maxMemory: 4294967296
INFO 2021-03-14 19:12:04 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:12:04 MarkDuplicates After output close freeMemory: 262951768; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:12:04 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.26 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/35.sorted.bam O=../qc-processing/bowtie/deduplicated/35..dedup.bam M=../qc-processing/bowtie/deduplicated/35..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:12:05 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/37.sorted.bam -O ../qc-processing/bowtie/deduplicated/37..dedup.bam -M ../qc-processing/bowtie/deduplicated/37..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:12:05.902 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:12:06 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/37.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/37..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/37..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:12:06 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:12:06 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:12:06 MarkDuplicates Start of doWork freeMemory: 67870616; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:12:06 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:12:06 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:12:09 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig18167:41,612
INFO 2021-03-14 19:12:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:10 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,216
INFO 2021-03-14 19:12:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:12 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35213:853
INFO 2021-03-14 19:12:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:13 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig42289:563
INFO 2021-03-14 19:12:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:15 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,502
INFO 2021-03-14 19:12:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:16 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig56693:13,888
INFO 2021-03-14 19:12:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:18 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig68037:348
INFO 2021-03-14 19:12:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:20 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,637
INFO 2021-03-14 19:12:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:21 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,537
INFO 2021-03-14 19:12:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:23 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig98184:1,973
INFO 2021-03-14 19:12:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:25 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:12:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:26 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig686969:632
INFO 2021-03-14 19:12:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:27 MarkDuplicates Read 12522185 records. 0 pairs never matched.
INFO 2021-03-14 19:12:27 MarkDuplicates After buildSortedReadEndLists freeMemory: 1055673088; totalMemory: 1560281088; maxMemory: 4294967296
INFO 2021-03-14 19:12:27 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:12:27 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:12:27 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:12:28 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:12:28 MarkDuplicates After generateDuplicateIndexes freeMemory: 1502063400; totalMemory: 2635071488; maxMemory: 4294967296
INFO 2021-03-14 19:12:28 MarkDuplicates Marking 3999771 records as duplicates.
INFO 2021-03-14 19:12:28 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:12:29 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:12:49 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:12:49 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:12:49 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:12:49 MarkDuplicates Before output close freeMemory: 654532512; totalMemory: 2635071488; maxMemory: 4294967296
INFO 2021-03-14 19:12:49 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:12:49 MarkDuplicates After output close freeMemory: 262956424; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:12:49 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.73 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/37.sorted.bam O=../qc-processing/bowtie/deduplicated/37..dedup.bam M=../qc-processing/bowtie/deduplicated/37..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:12:51 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/39.sorted.bam -O ../qc-processing/bowtie/deduplicated/39..dedup.bam -M ../qc-processing/bowtie/deduplicated/39..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:12:51.310 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:12:51 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/39.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/39..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/39..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:12:51 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:12:51 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:12:51 MarkDuplicates Start of doWork freeMemory: 67870840; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:12:51 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:12:51 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:12:54 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16189:8,795
INFO 2021-03-14 19:12:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:56 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig30144:2,661
INFO 2021-03-14 19:12:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:57 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig39846:4,370
INFO 2021-03-14 19:12:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:12:59 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56598:3,113
INFO 2021-03-14 19:12:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:01 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig67826:3,318
INFO 2021-03-14 19:13:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:02 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig80484:1,529
INFO 2021-03-14 19:13:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:04 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig106636:5,951
INFO 2021-03-14 19:13:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:05 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig686969:1,319
INFO 2021-03-14 19:13:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:05 MarkDuplicates Read 8028351 records. 0 pairs never matched.
INFO 2021-03-14 19:13:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 999117976; totalMemory: 1471152128; maxMemory: 4294967296
INFO 2021-03-14 19:13:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:13:06 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:13:06 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:13:07 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:13:07 MarkDuplicates After generateDuplicateIndexes freeMemory: 1412933112; totalMemory: 2545942528; maxMemory: 4294967296
INFO 2021-03-14 19:13:07 MarkDuplicates Marking 2443042 records as duplicates.
INFO 2021-03-14 19:13:07 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:13:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:13:21 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:13:21 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:13:21 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:13:21 MarkDuplicates Before output close freeMemory: 264858440; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:13:21 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:13:21 MarkDuplicates After output close freeMemory: 262955696; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:13:21 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.50 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/39.sorted.bam O=../qc-processing/bowtie/deduplicated/39..dedup.bam M=../qc-processing/bowtie/deduplicated/39..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:13:23 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/401.sorted.bam -O ../qc-processing/bowtie/deduplicated/401.dedup.bam -M ../qc-processing/bowtie/deduplicated/401.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:13:23.234 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:13:23 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/401.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/401.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/401.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:13:23 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:13:23 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:13:23 MarkDuplicates Start of doWork freeMemory: 72064312; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:13:23 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:13:23 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:13:26 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig24892:4,131
INFO 2021-03-14 19:13:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:28 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig42187:43,004
INFO 2021-03-14 19:13:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:29 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig56693:12,071
INFO 2021-03-14 19:13:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:31 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig68777:4,032
INFO 2021-03-14 19:13:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:32 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,936
INFO 2021-03-14 19:13:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:34 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,750
INFO 2021-03-14 19:13:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:36 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig148303:704
INFO 2021-03-14 19:13:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:36 MarkDuplicates Read 7308746 records. 0 pairs never matched.
INFO 2021-03-14 19:13:36 MarkDuplicates After buildSortedReadEndLists freeMemory: 926028712; totalMemory: 1318060032; maxMemory: 4294967296
INFO 2021-03-14 19:13:36 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:13:37 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:13:37 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:13:37 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:13:38 MarkDuplicates After generateDuplicateIndexes freeMemory: 1259839536; totalMemory: 2392850432; maxMemory: 4294967296
INFO 2021-03-14 19:13:38 MarkDuplicates Marking 2256756 records as duplicates.
INFO 2021-03-14 19:13:38 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:13:38 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:13:50 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:13:50 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:13:50 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:13:50 MarkDuplicates Before output close freeMemory: 266094912; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:13:51 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:13:51 MarkDuplicates After output close freeMemory: 262954048; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:13:51 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.46 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/401.sorted.bam O=../qc-processing/bowtie/deduplicated/401.dedup.bam M=../qc-processing/bowtie/deduplicated/401.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:13:52 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/402.sorted.bam -O ../qc-processing/bowtie/deduplicated/402.dedup.bam -M ../qc-processing/bowtie/deduplicated/402.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:13:52.766 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:13:52 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/402.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/402.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/402.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:13:52 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:13:52 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:13:52 MarkDuplicates Start of doWork freeMemory: 72063888; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:13:52 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:13:52 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:13:56 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 3s. Last read position: Contig15959:44,961
INFO 2021-03-14 19:13:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:13:58 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig28042:3,696
INFO 2021-03-14 19:13:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:00 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig37726:25,023
INFO 2021-03-14 19:14:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:01 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig53348:4,541
INFO 2021-03-14 19:14:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:03 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig66247:2,670
INFO 2021-03-14 19:14:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:04 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig70819:7,358
INFO 2021-03-14 19:14:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:06 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig101302:2,343
INFO 2021-03-14 19:14:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:07 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig174154:6,402
INFO 2021-03-14 19:14:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:08 MarkDuplicates Read 8371875 records. 0 pairs never matched.
INFO 2021-03-14 19:14:08 MarkDuplicates After buildSortedReadEndLists freeMemory: 824309880; totalMemory: 1309671424; maxMemory: 4294967296
INFO 2021-03-14 19:14:08 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:14:09 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:14:09 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:14:09 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:14:10 MarkDuplicates After generateDuplicateIndexes freeMemory: 1251451192; totalMemory: 2384461824; maxMemory: 4294967296
INFO 2021-03-14 19:14:10 MarkDuplicates Marking 2605655 records as duplicates.
INFO 2021-03-14 19:14:10 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:14:10 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:14:23 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:14:23 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:14:23 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:14:24 MarkDuplicates Before output close freeMemory: 261640976; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:14:24 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:14:24 MarkDuplicates After output close freeMemory: 262954304; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:14:24 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.52 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/402.sorted.bam O=../qc-processing/bowtie/deduplicated/402.dedup.bam M=../qc-processing/bowtie/deduplicated/402.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:14:25 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/403.sorted.bam -O ../qc-processing/bowtie/deduplicated/403.dedup.bam -M ../qc-processing/bowtie/deduplicated/403.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:14:25.955 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:14:26 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/403.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/403.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/403.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:14:26 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:14:26 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:14:26 MarkDuplicates Start of doWork freeMemory: 75209776; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:14:26 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:14:26 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:14:29 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5482:1,682
INFO 2021-03-14 19:14:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:31 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig23485:22,261
INFO 2021-03-14 19:14:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:32 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig33895:2,123
INFO 2021-03-14 19:14:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:34 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig42719:4,137
INFO 2021-03-14 19:14:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:35 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,649
INFO 2021-03-14 19:14:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:37 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,794
INFO 2021-03-14 19:14:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:39 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig90849:8,040
INFO 2021-03-14 19:14:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:40 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig133490:1,930
INFO 2021-03-14 19:14:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:14:42 MarkDuplicates Read 8843600 records. 0 pairs never matched.
INFO 2021-03-14 19:14:42 MarkDuplicates After buildSortedReadEndLists freeMemory: 1097442568; totalMemory: 1643118592; maxMemory: 4294967296
INFO 2021-03-14 19:14:42 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:14:42 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:14:42 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:14:43 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:14:43 MarkDuplicates After generateDuplicateIndexes freeMemory: 1584899016; totalMemory: 2717908992; maxMemory: 4294967296
INFO 2021-03-14 19:14:43 MarkDuplicates Marking 2958023 records as duplicates.
INFO 2021-03-14 19:14:43 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:14:44 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:14:58 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:14:58 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:14:58 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:14:59 MarkDuplicates Before output close freeMemory: 264203712; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:14:59 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:14:59 MarkDuplicates After output close freeMemory: 262954696; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:14:59 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.55 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/403.sorted.bam O=../qc-processing/bowtie/deduplicated/403.dedup.bam M=../qc-processing/bowtie/deduplicated/403.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:15:01 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/404.sorted.bam -O ../qc-processing/bowtie/deduplicated/404.dedup.bam -M ../qc-processing/bowtie/deduplicated/404.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:15:01.113 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:15:01 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/404.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/404.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/404.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:15:01 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:15:01 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:15:01 MarkDuplicates Start of doWork freeMemory: 64726928; totalMemory: 91226112; maxMemory: 4294967296
INFO 2021-03-14 19:15:01 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:15:01 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:15:04 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15298:857
INFO 2021-03-14 19:15:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:06 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig29038:10,845
INFO 2021-03-14 19:15:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:08 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig39636:2,918
INFO 2021-03-14 19:15:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:09 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig55763:2,206
INFO 2021-03-14 19:15:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:11 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig67389:9,966
INFO 2021-03-14 19:15:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:12 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig80410:1,869
INFO 2021-03-14 19:15:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:14 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig103849:2,735
INFO 2021-03-14 19:15:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:16 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig174154:7,169
INFO 2021-03-14 19:15:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:16 MarkDuplicates Read 8359296 records. 0 pairs never matched.
INFO 2021-03-14 19:15:16 MarkDuplicates After buildSortedReadEndLists freeMemory: 826235112; totalMemory: 1361051648; maxMemory: 4294967296
INFO 2021-03-14 19:15:16 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:15:17 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:15:17 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:15:18 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:15:18 MarkDuplicates After generateDuplicateIndexes freeMemory: 1302833160; totalMemory: 2435842048; maxMemory: 4294967296
INFO 2021-03-14 19:15:18 MarkDuplicates Marking 3562318 records as duplicates.
INFO 2021-03-14 19:15:18 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:15:19 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:15:32 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:15:32 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:15:32 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:15:32 MarkDuplicates Before output close freeMemory: 266096592; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:15:32 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:15:32 MarkDuplicates After output close freeMemory: 262955728; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:15:32 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.52 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/404.sorted.bam O=../qc-processing/bowtie/deduplicated/404.dedup.bam M=../qc-processing/bowtie/deduplicated/404.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:15:33 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/41.sorted.bam -O ../qc-processing/bowtie/deduplicated/41..dedup.bam -M ../qc-processing/bowtie/deduplicated/41..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:15:33.995 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:15:34 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/41.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/41..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/41..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:15:34 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:15:34 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:15:34 MarkDuplicates Start of doWork freeMemory: 72061680; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:15:34 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:15:34 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:15:37 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15836:31,933
INFO 2021-03-14 19:15:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:38 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig21019:6,270
INFO 2021-03-14 19:15:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:40 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,659
INFO 2021-03-14 19:15:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:41 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig32900:15,019
INFO 2021-03-14 19:15:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:43 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig42187:43,004
INFO 2021-03-14 19:15:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:44 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig56033:1,117
INFO 2021-03-14 19:15:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:46 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig56693:11,741
INFO 2021-03-14 19:15:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:48 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig62580:2,530
INFO 2021-03-14 19:15:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:50 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig68037:873
INFO 2021-03-14 19:15:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:51 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,603
INFO 2021-03-14 19:15:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:53 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig71494:3,713
INFO 2021-03-14 19:15:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:55 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,669
INFO 2021-03-14 19:15:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:56 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig103849:2,598
INFO 2021-03-14 19:15:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:58 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:23s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,511
INFO 2021-03-14 19:15:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:15:59 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 1s. Last read position: Contig148184:614
INFO 2021-03-14 19:15:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:16:01 MarkDuplicates Read 15914293 records. 0 pairs never matched.
INFO 2021-03-14 19:16:01 MarkDuplicates After buildSortedReadEndLists freeMemory: 894912848; totalMemory: 1521483776; maxMemory: 4294967296
INFO 2021-03-14 19:16:01 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:16:01 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:16:01 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:16:02 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:16:03 MarkDuplicates After generateDuplicateIndexes freeMemory: 1463262376; totalMemory: 2596274176; maxMemory: 4294967296
INFO 2021-03-14 19:16:03 MarkDuplicates Marking 5224154 records as duplicates.
INFO 2021-03-14 19:16:03 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:16:03 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:16:28 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:24s. Time for last 10,000,000: 24s. Last read position: Contig137778:6,438
INFO 2021-03-14 19:16:29 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:16:29 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:16:29 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:16:29 MarkDuplicates Before output close freeMemory: 2485635240; totalMemory: 2596274176; maxMemory: 4294967296
INFO 2021-03-14 19:16:29 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:16:29 MarkDuplicates After output close freeMemory: 262952856; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:16:29 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.93 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/41.sorted.bam O=../qc-processing/bowtie/deduplicated/41..dedup.bam M=../qc-processing/bowtie/deduplicated/41..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:16:31 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/411.sorted.bam -O ../qc-processing/bowtie/deduplicated/411.dedup.bam -M ../qc-processing/bowtie/deduplicated/411.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:16:31.710 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:16:32 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/411.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/411.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/411.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:16:32 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:16:32 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:16:32 MarkDuplicates Start of doWork freeMemory: 72061656; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:16:32 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:16:32 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:16:35 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig7297:1,954
INFO 2021-03-14 19:16:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:16:36 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig25638:4,795
INFO 2021-03-14 19:16:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:16:38 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35213:877
INFO 2021-03-14 19:16:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:16:40 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig46083:7,960
INFO 2021-03-14 19:16:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:16:41 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig57693:20,365
INFO 2021-03-14 19:16:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:16:43 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig69234:11,734
INFO 2021-03-14 19:16:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:16:44 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig85851:5,624
INFO 2021-03-14 19:16:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:16:46 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig121105:2,537
INFO 2021-03-14 19:16:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:16:47 MarkDuplicates Read 8939401 records. 0 pairs never matched.
INFO 2021-03-14 19:16:48 MarkDuplicates After buildSortedReadEndLists freeMemory: 1127202176; totalMemory: 1650458624; maxMemory: 4294967296
INFO 2021-03-14 19:16:48 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:16:48 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:16:48 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:16:49 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:16:49 MarkDuplicates After generateDuplicateIndexes freeMemory: 1592238544; totalMemory: 2725249024; maxMemory: 4294967296
INFO 2021-03-14 19:16:49 MarkDuplicates Marking 3029121 records as duplicates.
INFO 2021-03-14 19:16:49 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:16:50 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:17:04 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:17:04 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:17:04 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:17:04 MarkDuplicates Before output close freeMemory: 267486320; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:17:04 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:17:04 MarkDuplicates After output close freeMemory: 258759808; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:17:04 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.54 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/411.sorted.bam O=../qc-processing/bowtie/deduplicated/411.dedup.bam M=../qc-processing/bowtie/deduplicated/411.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:17:06 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/412.sorted.bam -O ../qc-processing/bowtie/deduplicated/412.dedup.bam -M ../qc-processing/bowtie/deduplicated/412.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:17:06.076 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:17:06 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/412.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/412.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/412.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:17:06 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:17:06 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:17:06 MarkDuplicates Start of doWork freeMemory: 72063640; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:17:06 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:17:06 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:17:09 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig35213:853
INFO 2021-03-14 19:17:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:10 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig68133:6,494
INFO 2021-03-14 19:17:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:12 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:17:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:13 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig103886:900
INFO 2021-03-14 19:17:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:14 MarkDuplicates Read 4784064 records. 0 pairs never matched.
INFO 2021-03-14 19:17:15 MarkDuplicates After buildSortedReadEndLists freeMemory: 839900552; totalMemory: 1192230912; maxMemory: 4294967296
INFO 2021-03-14 19:17:15 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:17:15 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:17:15 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:17:16 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:17:16 MarkDuplicates After generateDuplicateIndexes freeMemory: 1134010576; totalMemory: 2267021312; maxMemory: 4294967296
INFO 2021-03-14 19:17:16 MarkDuplicates Marking 2225608 records as duplicates.
INFO 2021-03-14 19:17:16 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:17:17 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:17:23 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:17:23 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:17:23 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:17:24 MarkDuplicates Before output close freeMemory: 266095112; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:17:24 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:17:24 MarkDuplicates After output close freeMemory: 262954248; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:17:24 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.30 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/412.sorted.bam O=../qc-processing/bowtie/deduplicated/412.dedup.bam M=../qc-processing/bowtie/deduplicated/412.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:17:25 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/413.sorted.bam -O ../qc-processing/bowtie/deduplicated/413.dedup.bam -M ../qc-processing/bowtie/deduplicated/413.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:17:25.905 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:17:25 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/413.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/413.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/413.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:17:26 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:17:26 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:17:26 MarkDuplicates Start of doWork freeMemory: 72061184; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:17:26 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:17:26 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:17:29 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig6483:6,175
INFO 2021-03-14 19:17:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:30 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig25603:6,380
INFO 2021-03-14 19:17:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:32 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35967:93
INFO 2021-03-14 19:17:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:34 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig47617:5,006
INFO 2021-03-14 19:17:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:35 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig64789:2,789
INFO 2021-03-14 19:17:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:37 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,944
INFO 2021-03-14 19:17:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:38 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig134210:619
INFO 2021-03-14 19:17:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:17:40 MarkDuplicates Read 7749399 records. 0 pairs never matched.
INFO 2021-03-14 19:17:40 MarkDuplicates After buildSortedReadEndLists freeMemory: 1236907056; totalMemory: 1761607680; maxMemory: 4294967296
INFO 2021-03-14 19:17:40 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:17:40 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:17:40 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:17:41 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:17:41 MarkDuplicates After generateDuplicateIndexes freeMemory: 1703385856; totalMemory: 2836398080; maxMemory: 4294967296
INFO 2021-03-14 19:17:41 MarkDuplicates Marking 3139760 records as duplicates.
INFO 2021-03-14 19:17:41 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:17:42 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:17:53 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:17:53 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:17:53 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:17:53 MarkDuplicates Before output close freeMemory: 262115264; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:17:53 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:17:53 MarkDuplicates After output close freeMemory: 258758112; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:17:53 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.47 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/413.sorted.bam O=../qc-processing/bowtie/deduplicated/413.dedup.bam M=../qc-processing/bowtie/deduplicated/413.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:17:55 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/414.sorted.bam -O ../qc-processing/bowtie/deduplicated/414.dedup.bam -M ../qc-processing/bowtie/deduplicated/414.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:17:55.611 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:17:55 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/414.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/414.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/414.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:17:55 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:17:55 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:17:55 MarkDuplicates Start of doWork freeMemory: 75210264; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:17:55 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:17:55 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:17:58 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig18183:3,156
INFO 2021-03-14 19:17:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:00 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig32665:6,351
INFO 2021-03-14 19:18:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:01 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig48580:5,666
INFO 2021-03-14 19:18:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:03 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig68716:4,703
INFO 2021-03-14 19:18:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:04 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig89303:7,448
INFO 2021-03-14 19:18:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:06 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig170266:2,396
INFO 2021-03-14 19:18:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:06 MarkDuplicates Read 6324460 records. 0 pairs never matched.
INFO 2021-03-14 19:18:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 1010900888; totalMemory: 1440743424; maxMemory: 4294967296
INFO 2021-03-14 19:18:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:18:07 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:18:07 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:18:08 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:18:08 MarkDuplicates After generateDuplicateIndexes freeMemory: 1382524376; totalMemory: 2515533824; maxMemory: 4294967296
INFO 2021-03-14 19:18:08 MarkDuplicates Marking 2021644 records as duplicates.
INFO 2021-03-14 19:18:08 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:18:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:18:18 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:18:18 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:18:18 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:18:18 MarkDuplicates Before output close freeMemory: 257789608; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:18:18 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:18:18 MarkDuplicates After output close freeMemory: 255615312; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 19:18:18 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.38 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/414.sorted.bam O=../qc-processing/bowtie/deduplicated/414.dedup.bam M=../qc-processing/bowtie/deduplicated/414.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:18:20 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/421.sorted.bam -O ../qc-processing/bowtie/deduplicated/421.dedup.bam -M ../qc-processing/bowtie/deduplicated/421.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:18:20.121 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:18:20 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/421.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/421.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/421.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:18:20 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:18:20 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:18:20 MarkDuplicates Start of doWork freeMemory: 72061352; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:18:20 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:18:20 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:18:23 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4583:2,684
INFO 2021-03-14 19:18:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:25 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig20021:7,076
INFO 2021-03-14 19:18:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:26 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig29369:25,118
INFO 2021-03-14 19:18:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:28 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig36186:33,309
INFO 2021-03-14 19:18:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:29 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig45529:8,300
INFO 2021-03-14 19:18:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:31 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,480
INFO 2021-03-14 19:18:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:33 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig64644:333
INFO 2021-03-14 19:18:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:34 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig69234:14,487
INFO 2021-03-14 19:18:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:36 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig82614:3,298
INFO 2021-03-14 19:18:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:37 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig103849:2,681
INFO 2021-03-14 19:18:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:39 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig136973:1,666
INFO 2021-03-14 19:18:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:41 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig686969:1,537
INFO 2021-03-14 19:18:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:18:41 MarkDuplicates Read 12014744 records. 0 pairs never matched.
INFO 2021-03-14 19:18:41 MarkDuplicates After buildSortedReadEndLists freeMemory: 1041307320; totalMemory: 1632632832; maxMemory: 4294967296
INFO 2021-03-14 19:18:41 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:18:41 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:18:41 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:18:42 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:18:42 MarkDuplicates After generateDuplicateIndexes freeMemory: 1574411088; totalMemory: 2707423232; maxMemory: 4294967296
INFO 2021-03-14 19:18:42 MarkDuplicates Marking 3287648 records as duplicates.
INFO 2021-03-14 19:18:42 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:18:43 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:19:03 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:19:03 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:19:03 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:19:03 MarkDuplicates Before output close freeMemory: 267205608; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:19:03 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:19:03 MarkDuplicates After output close freeMemory: 258758592; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:19:03 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.72 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/421.sorted.bam O=../qc-processing/bowtie/deduplicated/421.dedup.bam M=../qc-processing/bowtie/deduplicated/421.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:19:05 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/43.sorted.bam -O ../qc-processing/bowtie/deduplicated/43..dedup.bam -M ../qc-processing/bowtie/deduplicated/43..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:19:05.157 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:19:05 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/43.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/43..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/43..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:19:05 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:19:05 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:19:05 MarkDuplicates Start of doWork freeMemory: 72062008; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:19:05 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:19:05 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:19:08 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5148:760
INFO 2021-03-14 19:19:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:10 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig21438:15,911
INFO 2021-03-14 19:19:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:11 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30879:28,000
INFO 2021-03-14 19:19:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:13 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig38357:21,416
INFO 2021-03-14 19:19:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:14 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig49969:18,826
INFO 2021-03-14 19:19:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:16 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig59806:2,737
INFO 2021-03-14 19:19:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:18 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig69234:11,734
INFO 2021-03-14 19:19:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:19 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig84841:4,235
INFO 2021-03-14 19:19:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:21 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig107599:7,345
INFO 2021-03-14 19:19:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:22 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig187982:4,099
INFO 2021-03-14 19:19:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:23 MarkDuplicates Read 10392878 records. 0 pairs never matched.
INFO 2021-03-14 19:19:23 MarkDuplicates After buildSortedReadEndLists freeMemory: 995424280; totalMemory: 1556086784; maxMemory: 4294967296
INFO 2021-03-14 19:19:23 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:19:23 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:19:23 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:19:24 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:19:25 MarkDuplicates After generateDuplicateIndexes freeMemory: 1497866808; totalMemory: 2630877184; maxMemory: 4294967296
INFO 2021-03-14 19:19:25 MarkDuplicates Marking 3195288 records as duplicates.
INFO 2021-03-14 19:19:25 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:19:25 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:19:41 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:19:41 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:19:41 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:19:41 MarkDuplicates Before output close freeMemory: 267649096; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:19:41 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:19:41 MarkDuplicates After output close freeMemory: 266100064; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 19:19:41 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.61 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/43.sorted.bam O=../qc-processing/bowtie/deduplicated/43..dedup.bam M=../qc-processing/bowtie/deduplicated/43..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:19:43 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/431b.sorted.bam -O ../qc-processing/bowtie/deduplicated/431.dedup.bam -M ../qc-processing/bowtie/deduplicated/431.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:19:43.942 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
Mar 14, 2021 7:19:44 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:19:44 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:19:44 MarkDuplicates Start of doWork freeMemory: 72064616; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:19:44 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:19:44 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:19:47 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15994:7,628
INFO 2021-03-14 19:19:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:48 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig29931:21,345
INFO 2021-03-14 19:19:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:50 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig38003:16,521
INFO 2021-03-14 19:19:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:51 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig53225:768
INFO 2021-03-14 19:19:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:53 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:12,071
INFO 2021-03-14 19:19:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:55 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig66871:14,215
INFO 2021-03-14 19:19:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:56 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig69234:17,340
INFO 2021-03-14 19:19:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:19:58 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,537
INFO 2021-03-14 19:19:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:00 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,406
INFO 2021-03-14 19:20:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:01 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig147865:530
INFO 2021-03-14 19:20:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:02 MarkDuplicates Read 10561880 records. 0 pairs never matched.
INFO 2021-03-14 19:20:02 MarkDuplicates After buildSortedReadEndLists freeMemory: 1048684296; totalMemory: 1576009728; maxMemory: 4294967296
INFO 2021-03-14 19:20:02 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:20:03 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:20:03 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:20:04 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:20:04 MarkDuplicates After generateDuplicateIndexes freeMemory: 1517788584; totalMemory: 2650800128; maxMemory: 4294967296
INFO 2021-03-14 19:20:04 MarkDuplicates Marking 3480826 records as duplicates.
INFO 2021-03-14 19:20:04 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:20:04 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:20:23 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:20:23 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:20:23 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:20:23 MarkDuplicates Before output close freeMemory: 255608528; totalMemory: 363855872; maxMemory: 4294967296
INFO 2021-03-14 19:20:23 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:20:23 MarkDuplicates After output close freeMemory: 255613392; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 19:20:23 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.66 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/431b.sorted.bam O=../qc-processing/bowtie/deduplicated/431.dedup.bam M=../qc-processing/bowtie/deduplicated/431.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:20:25 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/432.sorted.bam -O ../qc-processing/bowtie/deduplicated/432.dedup.bam -M ../qc-processing/bowtie/deduplicated/432.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:20:25.321 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:20:25 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/432.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/432.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/432.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:20:25 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:20:25 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:20:25 MarkDuplicates Start of doWork freeMemory: 67868672; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:20:25 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:20:25 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:20:28 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig20021:6,825
INFO 2021-03-14 19:20:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:30 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,671
INFO 2021-03-14 19:20:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:31 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30879:28,010
INFO 2021-03-14 19:20:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:32 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig42187:43,019
INFO 2021-03-14 19:20:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:34 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,649
INFO 2021-03-14 19:20:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:35 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig56693:11,741
INFO 2021-03-14 19:20:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:37 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig64644:488
INFO 2021-03-14 19:20:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:39 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig68037:873
INFO 2021-03-14 19:20:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:40 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,510
INFO 2021-03-14 19:20:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:42 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,891
INFO 2021-03-14 19:20:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:44 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig83174:2,161
INFO 2021-03-14 19:20:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:45 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,537
INFO 2021-03-14 19:20:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:47 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,944
INFO 2021-03-14 19:20:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:48 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,229
INFO 2021-03-14 19:20:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:50 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:23s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,346
INFO 2021-03-14 19:20:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:51 MarkDuplicates Read 16,000,000 records. Elapsed time: 00:00:25s. Time for last 1,000,000: 1s. Last read position: Contig146366:3,809
INFO 2021-03-14 19:20:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:53 MarkDuplicates Read 17,000,000 records. Elapsed time: 00:00:26s. Time for last 1,000,000: 1s. Last read position: Contig686969:1,563
INFO 2021-03-14 19:20:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:20:53 MarkDuplicates Read 17208981 records. 0 pairs never matched.
INFO 2021-03-14 19:20:53 MarkDuplicates After buildSortedReadEndLists freeMemory: 997578304; totalMemory: 1529872384; maxMemory: 4294967296
INFO 2021-03-14 19:20:53 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:20:54 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:20:54 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:20:55 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:20:55 MarkDuplicates After generateDuplicateIndexes freeMemory: 1471651872; totalMemory: 2604662784; maxMemory: 4294967296
INFO 2021-03-14 19:20:55 MarkDuplicates Marking 4727429 records as duplicates.
INFO 2021-03-14 19:20:55 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:20:56 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:21:18 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:22s. Time for last 10,000,000: 22s. Last read position: Contig85486:1,944
INFO 2021-03-14 19:21:24 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:21:24 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:21:24 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:21:25 MarkDuplicates Before output close freeMemory: 264878216; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:21:25 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:21:25 MarkDuplicates After output close freeMemory: 262953792; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:21:25 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 1.00 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/432.sorted.bam O=../qc-processing/bowtie/deduplicated/432.dedup.bam M=../qc-processing/bowtie/deduplicated/432.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:21:26 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/434.sorted.bam -O ../qc-processing/bowtie/deduplicated/434.dedup.bam -M ../qc-processing/bowtie/deduplicated/434.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:21:26.829 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:21:27 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/434.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/434.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/434.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:21:27 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:21:27 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:21:27 MarkDuplicates Start of doWork freeMemory: 67868656; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:21:27 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:21:27 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:21:30 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15298:865
INFO 2021-03-14 19:21:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:21:31 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig31265:2,720
INFO 2021-03-14 19:21:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:21:33 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig40873:4,380
INFO 2021-03-14 19:21:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:21:34 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig57718:12,557
INFO 2021-03-14 19:21:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:21:36 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig74387:2,042
INFO 2021-03-14 19:21:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:21:38 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig108137:164
INFO 2021-03-14 19:21:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:21:39 MarkDuplicates Read 6914225 records. 0 pairs never matched.
INFO 2021-03-14 19:21:39 MarkDuplicates After buildSortedReadEndLists freeMemory: 959427760; totalMemory: 1469054976; maxMemory: 4294967296
INFO 2021-03-14 19:21:39 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:21:40 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:21:40 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:21:41 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:21:41 MarkDuplicates After generateDuplicateIndexes freeMemory: 1410835312; totalMemory: 2543845376; maxMemory: 4294967296
INFO 2021-03-14 19:21:41 MarkDuplicates Marking 3423588 records as duplicates.
INFO 2021-03-14 19:21:41 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:21:42 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:21:52 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:21:52 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:21:52 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:21:52 MarkDuplicates Before output close freeMemory: 262949608; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:21:52 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:21:52 MarkDuplicates After output close freeMemory: 258760168; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:21:52 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.43 minutes.
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/434.sorted.bam O=../qc-processing/bowtie/deduplicated/434.dedup.bam M=../qc-processing/bowtie/deduplicated/434.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:21:54 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/44.sorted.bam -O ../qc-processing/bowtie/deduplicated/44..dedup.bam -M ../qc-processing/bowtie/deduplicated/44..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:21:54.480 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:21:54 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/44.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/44..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/44..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:21:54 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:21:54 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:21:54 MarkDuplicates Start of doWork freeMemory: 67874104; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:21:54 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:21:54 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:21:57 MarkDuplicates Read 915416 records. 0 pairs never matched.
INFO 2021-03-14 19:21:57 MarkDuplicates After buildSortedReadEndLists freeMemory: 514973576; totalMemory: 727711744; maxMemory: 4294967296
INFO 2021-03-14 19:21:57 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:21:57 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:21:57 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:21:58 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:21:58 MarkDuplicates After generateDuplicateIndexes freeMemory: 761767992; totalMemory: 1894776832; maxMemory: 4294967296
INFO 2021-03-14 19:21:58 MarkDuplicates Marking 343899 records as duplicates.
INFO 2021-03-14 19:21:58 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:21:58 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:22:00 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:22:00 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:22:00 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:22:00 MarkDuplicates Before output close freeMemory: 264637504; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:22:00 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:22:00 MarkDuplicates After output close freeMemory: 258761568; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:22:00 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.11 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/44.sorted.bam O=../qc-processing/bowtie/deduplicated/44..dedup.bam M=../qc-processing/bowtie/deduplicated/44..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:22:02 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/441.sorted.bam -O ../qc-processing/bowtie/deduplicated/441.dedup.bam -M ../qc-processing/bowtie/deduplicated/441.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:22:02.552 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:22:02 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/441.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/441.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/441.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:22:02 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:22:02 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:22:02 MarkDuplicates Start of doWork freeMemory: 72063552; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:22:02 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:22:02 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:22:05 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig20043:18,889
INFO 2021-03-14 19:22:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:07 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig36186:16,958
INFO 2021-03-14 19:22:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:08 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig53881:929
INFO 2021-03-14 19:22:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:10 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig76256:34
INFO 2021-03-14 19:22:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:12 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:22:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:13 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:22:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:14 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:22:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:16 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig103886:900
INFO 2021-03-14 19:22:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:17 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig103886:900
INFO 2021-03-14 19:22:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:18 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig103886:900
INFO 2021-03-14 19:22:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:20 MarkDuplicates Read 10826756 records. 0 pairs never matched.
INFO 2021-03-14 19:22:20 MarkDuplicates After buildSortedReadEndLists freeMemory: 973945552; totalMemory: 1624244224; maxMemory: 4294967296
INFO 2021-03-14 19:22:20 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:22:20 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:22:20 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:22:22 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:22:22 MarkDuplicates After generateDuplicateIndexes freeMemory: 1566024696; totalMemory: 2699034624; maxMemory: 4294967296
INFO 2021-03-14 19:22:22 MarkDuplicates Marking 6401934 records as duplicates.
INFO 2021-03-14 19:22:22 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:22:23 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:22:36 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:22:36 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:22:36 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:22:36 MarkDuplicates Before output close freeMemory: 262551680; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:22:36 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:22:36 MarkDuplicates After output close freeMemory: 262954992; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:22:36 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.57 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/441.sorted.bam O=../qc-processing/bowtie/deduplicated/441.dedup.bam M=../qc-processing/bowtie/deduplicated/441.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:22:38 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/442b.sorted.bam -O ../qc-processing/bowtie/deduplicated/442.dedup.bam -M ../qc-processing/bowtie/deduplicated/442.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:22:38.151 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:22:38 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/442b.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/442.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/442.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:22:38 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:22:38 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:22:38 MarkDuplicates Start of doWork freeMemory: 67873296; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:22:38 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:22:38 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:22:41 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig20021:6,825
INFO 2021-03-14 19:22:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:42 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig30879:28,089
INFO 2021-03-14 19:22:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:44 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig50339:20,905
INFO 2021-03-14 19:22:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:46 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig66349:6,445
INFO 2021-03-14 19:22:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:47 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,891
INFO 2021-03-14 19:22:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:49 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig85486:2,335
INFO 2021-03-14 19:22:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:51 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig105506:8,199
INFO 2021-03-14 19:22:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:22:52 MarkDuplicates Read 7815845 records. 0 pairs never matched.
INFO 2021-03-14 19:22:52 MarkDuplicates After buildSortedReadEndLists freeMemory: 1033005496; totalMemory: 1472200704; maxMemory: 4294967296
INFO 2021-03-14 19:22:52 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:22:52 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:22:52 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:22:53 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:22:53 MarkDuplicates After generateDuplicateIndexes freeMemory: 1413982368; totalMemory: 2546991104; maxMemory: 4294967296
INFO 2021-03-14 19:22:53 MarkDuplicates Marking 2611210 records as duplicates.
INFO 2021-03-14 19:22:53 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:22:54 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:23:06 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:23:06 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:23:06 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:23:07 MarkDuplicates Before output close freeMemory: 262950816; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:23:07 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:23:07 MarkDuplicates After output close freeMemory: 258761376; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:23:07 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.49 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/442b.sorted.bam O=../qc-processing/bowtie/deduplicated/442.dedup.bam M=../qc-processing/bowtie/deduplicated/442.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:23:08 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/443.sorted.bam -O ../qc-processing/bowtie/deduplicated/443.dedup.bam -M ../qc-processing/bowtie/deduplicated/443.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:23:08.973 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:23:09 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/443.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/443.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/443.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:23:09 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:23:09 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:23:09 MarkDuplicates Start of doWork freeMemory: 64722568; totalMemory: 91226112; maxMemory: 4294967296
INFO 2021-03-14 19:23:09 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:23:09 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:23:12 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15439:47,440
INFO 2021-03-14 19:23:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:13 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,639
INFO 2021-03-14 19:23:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:15 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig42375:3,433
INFO 2021-03-14 19:23:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:17 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig59424:1,399
INFO 2021-03-14 19:23:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:18 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig81792:4,920
INFO 2021-03-14 19:23:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:20 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig135300:18,411
INFO 2021-03-14 19:23:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:21 MarkDuplicates Read 6616242 records. 0 pairs never matched.
INFO 2021-03-14 19:23:21 MarkDuplicates After buildSortedReadEndLists freeMemory: 1155898248; totalMemory: 1646264320; maxMemory: 4294967296
INFO 2021-03-14 19:23:21 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:23:21 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:23:21 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:23:22 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:23:22 MarkDuplicates After generateDuplicateIndexes freeMemory: 1588044144; totalMemory: 2721054720; maxMemory: 4294967296
INFO 2021-03-14 19:23:22 MarkDuplicates Marking 3012328 records as duplicates.
INFO 2021-03-14 19:23:22 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:23:23 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:23:32 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:23:32 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:23:32 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:23:32 MarkDuplicates Before output close freeMemory: 262649512; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:23:32 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:23:32 MarkDuplicates After output close freeMemory: 262954256; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:23:32 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.40 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/443.sorted.bam O=../qc-processing/bowtie/deduplicated/443.dedup.bam M=../qc-processing/bowtie/deduplicated/443.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:23:34 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/444.sorted.bam -O ../qc-processing/bowtie/deduplicated/444.dedup.bam -M ../qc-processing/bowtie/deduplicated/444.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:23:34.349 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:23:34 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/444.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/444.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/444.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:23:34 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:23:34 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:23:34 MarkDuplicates Start of doWork freeMemory: 72062296; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:23:34 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:23:34 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:23:37 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5115:4,519
INFO 2021-03-14 19:23:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:39 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig24129:4,034
INFO 2021-03-14 19:23:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:40 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35213:855
INFO 2021-03-14 19:23:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:42 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig45006:159
INFO 2021-03-14 19:23:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:43 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig58498:3,665
INFO 2021-03-14 19:23:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:45 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig75533:15,480
INFO 2021-03-14 19:23:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:47 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig104654:5,377
INFO 2021-03-14 19:23:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:48 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig205407:2,693
INFO 2021-03-14 19:23:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:23:48 MarkDuplicates Read 8199010 records. 0 pairs never matched.
INFO 2021-03-14 19:23:49 MarkDuplicates After buildSortedReadEndLists freeMemory: 844507688; totalMemory: 1428160512; maxMemory: 4294967296
INFO 2021-03-14 19:23:49 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:23:49 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:23:49 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:23:50 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:23:50 MarkDuplicates After generateDuplicateIndexes freeMemory: 1369940168; totalMemory: 2502950912; maxMemory: 4294967296
INFO 2021-03-14 19:23:50 MarkDuplicates Marking 4252788 records as duplicates.
INFO 2021-03-14 19:23:50 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:23:51 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:24:03 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:24:03 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:24:03 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:24:03 MarkDuplicates Before output close freeMemory: 258754680; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:24:03 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:24:03 MarkDuplicates After output close freeMemory: 258759544; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:24:03 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.48 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/444.sorted.bam O=../qc-processing/bowtie/deduplicated/444.dedup.bam M=../qc-processing/bowtie/deduplicated/444.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:24:05 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/445.sorted.bam -O ../qc-processing/bowtie/deduplicated/445.dedup.bam -M ../qc-processing/bowtie/deduplicated/445.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:24:05.148 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:24:05 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/445.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/445.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/445.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:24:05 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:24:05 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:24:05 MarkDuplicates Start of doWork freeMemory: 67868752; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:24:05 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:24:05 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:24:08 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5705:6,465
INFO 2021-03-14 19:24:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:10 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig23498:20,136
INFO 2021-03-14 19:24:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:11 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig33765:9,163
INFO 2021-03-14 19:24:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:13 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig42936:6,885
INFO 2021-03-14 19:24:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:14 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:4,516
INFO 2021-03-14 19:24:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:16 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68908:739
INFO 2021-03-14 19:24:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:18 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig85426:2,226
INFO 2021-03-14 19:24:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:19 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig110898:6,083
INFO 2021-03-14 19:24:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:21 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig313765:1,651
INFO 2021-03-14 19:24:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:21 MarkDuplicates Read 9154329 records. 0 pairs never matched.
INFO 2021-03-14 19:24:21 MarkDuplicates After buildSortedReadEndLists freeMemory: 958662080; totalMemory: 1504706560; maxMemory: 4294967296
INFO 2021-03-14 19:24:21 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:24:22 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:24:22 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:24:22 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:24:23 MarkDuplicates After generateDuplicateIndexes freeMemory: 1446486368; totalMemory: 2579496960; maxMemory: 4294967296
INFO 2021-03-14 19:24:23 MarkDuplicates Marking 3045674 records as duplicates.
INFO 2021-03-14 19:24:23 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:24:23 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:24:37 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:24:37 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:24:37 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:24:38 MarkDuplicates Before output close freeMemory: 264594496; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:24:38 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:24:38 MarkDuplicates After output close freeMemory: 258760344; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:24:38 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.55 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/445.sorted.bam O=../qc-processing/bowtie/deduplicated/445.dedup.bam M=../qc-processing/bowtie/deduplicated/445.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:24:39 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/45.sorted.bam -O ../qc-processing/bowtie/deduplicated/45..dedup.bam -M ../qc-processing/bowtie/deduplicated/45..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:24:39.940 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:24:40 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/45.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/45..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/45..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:24:40 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:24:40 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:24:40 MarkDuplicates Start of doWork freeMemory: 64725560; totalMemory: 91226112; maxMemory: 4294967296
INFO 2021-03-14 19:24:40 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:24:40 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:24:43 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5839:2,047
INFO 2021-03-14 19:24:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:44 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig22359:2,556
INFO 2021-03-14 19:24:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:46 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig31928:6,944
INFO 2021-03-14 19:24:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:47 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig41537:1,221
INFO 2021-03-14 19:24:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:49 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig55040:6,345
INFO 2021-03-14 19:24:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:50 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig66450:1,167
INFO 2021-03-14 19:24:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:52 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig70882:7,693
INFO 2021-03-14 19:24:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:54 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig92992:14,808
INFO 2021-03-14 19:24:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:56 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig123992:1,952
INFO 2021-03-14 19:24:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:57 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig492755:82
INFO 2021-03-14 19:24:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:24:57 MarkDuplicates Read 10126095 records. 0 pairs never matched.
INFO 2021-03-14 19:24:57 MarkDuplicates After buildSortedReadEndLists freeMemory: 869769680; totalMemory: 1379926016; maxMemory: 4294967296
INFO 2021-03-14 19:24:57 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:24:58 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:24:58 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:24:59 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:24:59 MarkDuplicates After generateDuplicateIndexes freeMemory: 1321706008; totalMemory: 2454716416; maxMemory: 4294967296
INFO 2021-03-14 19:24:59 MarkDuplicates Marking 3430318 records as duplicates.
INFO 2021-03-14 19:24:59 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:25:00 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:25:15 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:25:15 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:25:15 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:25:15 MarkDuplicates Before output close freeMemory: 375217864; totalMemory: 2454716416; maxMemory: 4294967296
INFO 2021-03-14 19:25:15 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:25:15 MarkDuplicates After output close freeMemory: 262954320; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:25:15 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.59 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/45.sorted.bam O=../qc-processing/bowtie/deduplicated/45..dedup.bam M=../qc-processing/bowtie/deduplicated/45..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:25:17 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/451.sorted.bam -O ../qc-processing/bowtie/deduplicated/451.dedup.bam -M ../qc-processing/bowtie/deduplicated/451.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:25:17.071 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:25:17 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/451.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/451.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/451.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:25:17 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:25:17 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:25:17 MarkDuplicates Start of doWork freeMemory: 67868464; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:25:17 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:25:17 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:25:20 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3255:600
INFO 2021-03-14 19:25:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:22 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig20015:3,210
INFO 2021-03-14 19:25:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:23 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig29566:785
INFO 2021-03-14 19:25:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:25 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig36557:11,314
INFO 2021-03-14 19:25:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:26 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig45811:6,132
INFO 2021-03-14 19:25:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:28 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig56910:900
INFO 2021-03-14 19:25:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:29 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,805
INFO 2021-03-14 19:25:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:31 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig89302:1,507
INFO 2021-03-14 19:25:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:33 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig115433:3,440
INFO 2021-03-14 19:25:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:34 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig206328:1,420
INFO 2021-03-14 19:25:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:35 MarkDuplicates Read 10291693 records. 0 pairs never matched.
INFO 2021-03-14 19:25:35 MarkDuplicates After buildSortedReadEndLists freeMemory: 936725688; totalMemory: 1577058304; maxMemory: 4294967296
INFO 2021-03-14 19:25:35 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:25:35 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:25:35 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:25:37 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:25:37 MarkDuplicates After generateDuplicateIndexes freeMemory: 1518838448; totalMemory: 2651848704; maxMemory: 4294967296
INFO 2021-03-14 19:25:37 MarkDuplicates Marking 4258107 records as duplicates.
INFO 2021-03-14 19:25:37 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:25:37 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:25:52 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:25:52 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:25:52 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:25:52 MarkDuplicates Before output close freeMemory: 267630192; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:25:52 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:25:52 MarkDuplicates After output close freeMemory: 262954528; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:25:52 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.60 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/451.sorted.bam O=../qc-processing/bowtie/deduplicated/451.dedup.bam M=../qc-processing/bowtie/deduplicated/451.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:25:54 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/452b.sorted.bam -O ../qc-processing/bowtie/deduplicated/452.dedup.bam -M ../qc-processing/bowtie/deduplicated/452.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:25:54.561 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:25:54 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/452b.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/452.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/452.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:25:54 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:25:54 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:25:54 MarkDuplicates Start of doWork freeMemory: 72067080; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:25:54 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:25:54 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:25:57 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig17782:9,886
INFO 2021-03-14 19:25:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:25:59 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,672
INFO 2021-03-14 19:25:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:01 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig43103:377
INFO 2021-03-14 19:26:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:02 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56693:12,060
INFO 2021-03-14 19:26:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:04 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig68908:704
INFO 2021-03-14 19:26:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:06 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig73535:1,225
INFO 2021-03-14 19:26:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:07 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,334
INFO 2021-03-14 19:26:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:09 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig189001:5,385
INFO 2021-03-14 19:26:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:10 MarkDuplicates Read 8281261 records. 0 pairs never matched.
INFO 2021-03-14 19:26:10 MarkDuplicates After buildSortedReadEndLists freeMemory: 1118778832; totalMemory: 1590689792; maxMemory: 4294967296
INFO 2021-03-14 19:26:10 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:26:10 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:26:10 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:26:11 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:26:11 MarkDuplicates After generateDuplicateIndexes freeMemory: 1532472640; totalMemory: 2665480192; maxMemory: 4294967296
INFO 2021-03-14 19:26:11 MarkDuplicates Marking 2686431 records as duplicates.
INFO 2021-03-14 19:26:11 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:26:12 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:26:24 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:26:24 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:26:24 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:26:24 MarkDuplicates Before output close freeMemory: 258757760; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:26:24 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:26:25 MarkDuplicates After output close freeMemory: 258762624; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:26:25 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.51 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/452b.sorted.bam O=../qc-processing/bowtie/deduplicated/452.dedup.bam M=../qc-processing/bowtie/deduplicated/452.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:26:26 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/453.sorted.bam -O ../qc-processing/bowtie/deduplicated/453.dedup.bam -M ../qc-processing/bowtie/deduplicated/453.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:26:26.711 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:26:26 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/453.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/453.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/453.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:26:26 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:26:26 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:26:26 MarkDuplicates Start of doWork freeMemory: 75209784; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:26:26 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:26:26 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:26:30 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 3s. Last read position: Contig15978:2,686
INFO 2021-03-14 19:26:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:32 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 2s. Last read position: Contig29882:20,225
INFO 2021-03-14 19:26:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:34 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig40873:4,349
INFO 2021-03-14 19:26:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:36 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig57009:2,287
INFO 2021-03-14 19:26:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:37 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig71579:4,565
INFO 2021-03-14 19:26:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:39 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig107599:7,355
INFO 2021-03-14 19:26:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:26:41 MarkDuplicates Read 6936824 records. 0 pairs never matched.
INFO 2021-03-14 19:26:41 MarkDuplicates After buildSortedReadEndLists freeMemory: 1113340328; totalMemory: 1583349760; maxMemory: 4294967296
INFO 2021-03-14 19:26:41 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:26:41 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:26:41 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:26:42 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:26:42 MarkDuplicates After generateDuplicateIndexes freeMemory: 1525130376; totalMemory: 2658140160; maxMemory: 4294967296
INFO 2021-03-14 19:26:42 MarkDuplicates Marking 2420412 records as duplicates.
INFO 2021-03-14 19:26:42 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:26:43 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:26:53 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:26:53 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:26:53 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:26:53 MarkDuplicates Before output close freeMemory: 1471128712; totalMemory: 2658140160; maxMemory: 4294967296
INFO 2021-03-14 19:26:53 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:26:54 MarkDuplicates After output close freeMemory: 262954920; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:26:54 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.46 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/453.sorted.bam O=../qc-processing/bowtie/deduplicated/453.dedup.bam M=../qc-processing/bowtie/deduplicated/453.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:26:55 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/46.sorted.bam -O ../qc-processing/bowtie/deduplicated/46..dedup.bam -M ../qc-processing/bowtie/deduplicated/46..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:26:55.779 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:26:55 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/46.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/46..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/46..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:26:56 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:26:56 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:26:56 MarkDuplicates Start of doWork freeMemory: 67869344; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:26:56 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:26:56 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:26:58 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16202:1,865
INFO 2021-03-14 19:26:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:00 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig25496:5,545
INFO 2021-03-14 19:27:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:02 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30879:28,098
INFO 2021-03-14 19:27:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:03 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig43261:14,832
INFO 2021-03-14 19:27:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:05 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,649
INFO 2021-03-14 19:27:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:06 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig63193:15,275
INFO 2021-03-14 19:27:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:08 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig69234:14,487
INFO 2021-03-14 19:27:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:10 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig77069:9,020
INFO 2021-03-14 19:27:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:11 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig95501:1,591
INFO 2021-03-14 19:27:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:13 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,495
INFO 2021-03-14 19:27:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:15 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig186382:383
INFO 2021-03-14 19:27:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:15 MarkDuplicates Read 11459236 records. 0 pairs never matched.
INFO 2021-03-14 19:27:16 MarkDuplicates After buildSortedReadEndLists freeMemory: 1142263264; totalMemory: 1699741696; maxMemory: 4294967296
INFO 2021-03-14 19:27:16 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:27:16 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:27:16 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:27:17 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:27:17 MarkDuplicates After generateDuplicateIndexes freeMemory: 1641522880; totalMemory: 2774532096; maxMemory: 4294967296
INFO 2021-03-14 19:27:17 MarkDuplicates Marking 4365146 records as duplicates.
INFO 2021-03-14 19:27:17 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:27:18 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:27:35 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:27:35 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:27:35 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:27:36 MarkDuplicates Before output close freeMemory: 265584600; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:27:36 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:27:36 MarkDuplicates After output close freeMemory: 266101624; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 19:27:36 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.67 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/46.sorted.bam O=../qc-processing/bowtie/deduplicated/46..dedup.bam M=../qc-processing/bowtie/deduplicated/46..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:27:38 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/461b.sorted.bam -O ../qc-processing/bowtie/deduplicated/461.dedup.bam -M ../qc-processing/bowtie/deduplicated/461.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:27:38.203 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:27:38 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/461b.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/461.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/461.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:27:38 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:27:38 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:27:38 MarkDuplicates Start of doWork freeMemory: 72062608; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:27:38 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:27:38 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:27:41 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig21665:33,329
INFO 2021-03-14 19:27:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:43 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig38227:6,038
INFO 2021-03-14 19:27:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:45 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig56693:9,308
INFO 2021-03-14 19:27:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:46 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig70595:999
INFO 2021-03-14 19:27:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:48 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig101783:541
INFO 2021-03-14 19:27:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:27:50 MarkDuplicates Read 5982926 records. 0 pairs never matched.
INFO 2021-03-14 19:27:50 MarkDuplicates After buildSortedReadEndLists freeMemory: 999495200; totalMemory: 1422917632; maxMemory: 4294967296
INFO 2021-03-14 19:27:50 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:27:50 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:27:50 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:27:51 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:27:51 MarkDuplicates After generateDuplicateIndexes freeMemory: 1364697656; totalMemory: 2497708032; maxMemory: 4294967296
INFO 2021-03-14 19:27:51 MarkDuplicates Marking 2253446 records as duplicates.
INFO 2021-03-14 19:27:51 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:27:52 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:28:01 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:28:01 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:28:01 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:28:01 MarkDuplicates Before output close freeMemory: 262949600; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:28:01 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:28:01 MarkDuplicates After output close freeMemory: 258760032; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:28:01 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.40 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/461b.sorted.bam O=../qc-processing/bowtie/deduplicated/461.dedup.bam M=../qc-processing/bowtie/deduplicated/461.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:28:03 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/462b.sorted.bam -O ../qc-processing/bowtie/deduplicated/462.dedup.bam -M ../qc-processing/bowtie/deduplicated/462.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:28:03.580 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:28:03 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/462b.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/462.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/462.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:28:03 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:28:03 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:28:03 MarkDuplicates Start of doWork freeMemory: 75209184; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:28:03 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:28:03 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:28:06 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5738:10,717
INFO 2021-03-14 19:28:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:08 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig24129:3,965
INFO 2021-03-14 19:28:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:10 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig32730:7,350
INFO 2021-03-14 19:28:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:11 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig42187:43,004
INFO 2021-03-14 19:28:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:13 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56598:3,105
INFO 2021-03-14 19:28:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:14 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68037:876
INFO 2021-03-14 19:28:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:16 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig81479:1,748
INFO 2021-03-14 19:28:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:18 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,705
INFO 2021-03-14 19:28:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:19 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig109966:244
INFO 2021-03-14 19:28:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:21 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig492138:5,850
INFO 2021-03-14 19:28:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:21 MarkDuplicates Read 10143015 records. 0 pairs never matched.
INFO 2021-03-14 19:28:21 MarkDuplicates After buildSortedReadEndLists freeMemory: 1003491312; totalMemory: 1659895808; maxMemory: 4294967296
INFO 2021-03-14 19:28:21 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:28:22 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:28:22 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:28:23 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:28:23 MarkDuplicates After generateDuplicateIndexes freeMemory: 1601676120; totalMemory: 2734686208; maxMemory: 4294967296
INFO 2021-03-14 19:28:23 MarkDuplicates Marking 5251407 records as duplicates.
INFO 2021-03-14 19:28:23 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:28:24 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:28:37 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:28:37 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:28:37 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:28:38 MarkDuplicates Before output close freeMemory: 262949192; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:28:38 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:28:38 MarkDuplicates After output close freeMemory: 258759752; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:28:38 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.58 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/462b.sorted.bam O=../qc-processing/bowtie/deduplicated/462.dedup.bam M=../qc-processing/bowtie/deduplicated/462.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:28:39 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/47.sorted.bam -O ../qc-processing/bowtie/deduplicated/47..dedup.bam -M ../qc-processing/bowtie/deduplicated/47..dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:28:39.918 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:28:40 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/47.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/47..dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/47..dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:28:40 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:28:40 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:28:40 MarkDuplicates Start of doWork freeMemory: 67875296; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:28:40 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:28:40 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:28:43 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16373:12,206
INFO 2021-03-14 19:28:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:45 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig29931:28,092
INFO 2021-03-14 19:28:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:46 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig39679:11,238
INFO 2021-03-14 19:28:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:48 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,500
INFO 2021-03-14 19:28:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:49 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,805
INFO 2021-03-14 19:28:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:51 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig92992:19,305
INFO 2021-03-14 19:28:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:53 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig139531:5,444
INFO 2021-03-14 19:28:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:28:54 MarkDuplicates Read 7591272 records. 0 pairs never matched.
INFO 2021-03-14 19:28:54 MarkDuplicates After buildSortedReadEndLists freeMemory: 1087722016; totalMemory: 1548746752; maxMemory: 4294967296
INFO 2021-03-14 19:28:54 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:28:54 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:28:54 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:28:55 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:28:55 MarkDuplicates After generateDuplicateIndexes freeMemory: 1490528128; totalMemory: 2623537152; maxMemory: 4294967296
INFO 2021-03-14 19:28:55 MarkDuplicates Marking 2432924 records as duplicates.
INFO 2021-03-14 19:28:55 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:28:56 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:29:08 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:29:08 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:29:08 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:29:08 MarkDuplicates Before output close freeMemory: 267430896; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:29:08 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:29:08 MarkDuplicates After output close freeMemory: 258761320; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:29:08 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.48 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/47.sorted.bam O=../qc-processing/bowtie/deduplicated/47..dedup.bam M=../qc-processing/bowtie/deduplicated/47..dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:29:10 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/471b.sorted.bam -O ../qc-processing/bowtie/deduplicated/471.dedup.bam -M ../qc-processing/bowtie/deduplicated/471.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:29:10.437 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:29:10 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/471b.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/471.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/471.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:29:10 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:29:10 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:29:10 MarkDuplicates Start of doWork freeMemory: 67871112; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:29:10 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:29:10 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:29:13 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16668:7,094
INFO 2021-03-14 19:29:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:15 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,420
INFO 2021-03-14 19:29:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:17 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig40873:2,646
INFO 2021-03-14 19:29:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:18 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,649
INFO 2021-03-14 19:29:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:20 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig66871:14,257
INFO 2021-03-14 19:29:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:21 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig71414:3,444
INFO 2021-03-14 19:29:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:23 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,738
INFO 2021-03-14 19:29:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:25 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig138882:2,496
INFO 2021-03-14 19:29:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:26 MarkDuplicates Read 8581233 records. 0 pairs never matched.
INFO 2021-03-14 19:29:26 MarkDuplicates After buildSortedReadEndLists freeMemory: 875142040; totalMemory: 1390411776; maxMemory: 4294967296
INFO 2021-03-14 19:29:26 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:29:26 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:29:26 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:29:27 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:29:27 MarkDuplicates After generateDuplicateIndexes freeMemory: 1332192672; totalMemory: 2465202176; maxMemory: 4294967296
INFO 2021-03-14 19:29:27 MarkDuplicates Marking 3150487 records as duplicates.
INFO 2021-03-14 19:29:27 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:29:28 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:29:41 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:29:41 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:29:41 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:29:42 MarkDuplicates Before output close freeMemory: 258756224; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:29:42 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:29:42 MarkDuplicates After output close freeMemory: 258761088; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:29:42 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.53 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/471b.sorted.bam O=../qc-processing/bowtie/deduplicated/471.dedup.bam M=../qc-processing/bowtie/deduplicated/471.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:29:44 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/472b.sorted.bam -O ../qc-processing/bowtie/deduplicated/472.dedup.bam -M ../qc-processing/bowtie/deduplicated/472.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:29:44.044 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:29:44 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/472b.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/472.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/472.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:29:44 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:29:44 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:29:44 MarkDuplicates Start of doWork freeMemory: 67874360; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:29:44 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:29:44 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:29:47 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16509:24,097
INFO 2021-03-14 19:29:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:49 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,413
INFO 2021-03-14 19:29:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:51 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig41407:5,826
INFO 2021-03-14 19:29:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:52 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56693:5,048
INFO 2021-03-14 19:29:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:54 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig68908:700
INFO 2021-03-14 19:29:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:56 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig79944:2,798
INFO 2021-03-14 19:29:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:57 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig104005:11,137
INFO 2021-03-14 19:29:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:59 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig673102:167
INFO 2021-03-14 19:29:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:29:59 MarkDuplicates Read 8047176 records. 0 pairs never matched.
INFO 2021-03-14 19:29:59 MarkDuplicates After buildSortedReadEndLists freeMemory: 1171049136; totalMemory: 1678770176; maxMemory: 4294967296
INFO 2021-03-14 19:29:59 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:29:59 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:29:59 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:30:00 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:30:01 MarkDuplicates After generateDuplicateIndexes freeMemory: 1620551064; totalMemory: 2753560576; maxMemory: 4294967296
INFO 2021-03-14 19:30:01 MarkDuplicates Marking 3092587 records as duplicates.
INFO 2021-03-14 19:30:01 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:30:01 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:30:14 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:30:14 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:30:14 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:30:14 MarkDuplicates Before output close freeMemory: 262950648; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:30:14 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:30:14 MarkDuplicates After output close freeMemory: 258761208; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:30:14 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.51 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/472b.sorted.bam O=../qc-processing/bowtie/deduplicated/472.dedup.bam M=../qc-processing/bowtie/deduplicated/472.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:30:16 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/473.sorted.bam -O ../qc-processing/bowtie/deduplicated/473.dedup.bam -M ../qc-processing/bowtie/deduplicated/473.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:30:16.562 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:30:16 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/473.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/473.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/473.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:30:16 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:30:16 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:30:16 MarkDuplicates Start of doWork freeMemory: 72064600; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:30:16 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:30:16 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:30:20 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig7085:2,274
INFO 2021-03-14 19:30:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:21 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig25713:11,312
INFO 2021-03-14 19:30:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:23 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35774:1,112
INFO 2021-03-14 19:30:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:25 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig47617:8,854
INFO 2021-03-14 19:30:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:26 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig62744:3,367
INFO 2021-03-14 19:30:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:28 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig77734:588
INFO 2021-03-14 19:30:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:30 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig103849:5,739
INFO 2021-03-14 19:30:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:31 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig205580:2,117
INFO 2021-03-14 19:30:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:31 MarkDuplicates Read 8219914 records. 0 pairs never matched.
INFO 2021-03-14 19:30:32 MarkDuplicates After buildSortedReadEndLists freeMemory: 856754704; totalMemory: 1390411776; maxMemory: 4294967296
INFO 2021-03-14 19:30:32 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:30:32 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:30:32 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:30:33 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:30:33 MarkDuplicates After generateDuplicateIndexes freeMemory: 1332192904; totalMemory: 2465202176; maxMemory: 4294967296
INFO 2021-03-14 19:30:33 MarkDuplicates Marking 3188510 records as duplicates.
INFO 2021-03-14 19:30:33 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:30:34 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:30:46 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:30:46 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:30:46 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:30:47 MarkDuplicates Before output close freeMemory: 265097432; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:30:47 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:30:47 MarkDuplicates After output close freeMemory: 258760600; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:30:47 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.51 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/473.sorted.bam O=../qc-processing/bowtie/deduplicated/473.dedup.bam M=../qc-processing/bowtie/deduplicated/473.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:30:48 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/474.sorted.bam -O ../qc-processing/bowtie/deduplicated/474.dedup.bam -M ../qc-processing/bowtie/deduplicated/474.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:30:48.882 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:30:48 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/474.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/474.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/474.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:30:49 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:30:49 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:30:49 MarkDuplicates Start of doWork freeMemory: 75208320; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:30:49 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:30:49 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:30:52 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3058:2,096
INFO 2021-03-14 19:30:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:53 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig20453:9,506
INFO 2021-03-14 19:30:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:55 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30266:21,848
INFO 2021-03-14 19:30:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:57 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig37186:10,622
INFO 2021-03-14 19:30:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:30:58 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig47544:1,295
INFO 2021-03-14 19:30:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:00 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig59424:2,860
INFO 2021-03-14 19:31:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:01 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig73878:4,356
INFO 2021-03-14 19:31:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:03 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,750
INFO 2021-03-14 19:31:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:05 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig145172:932
INFO 2021-03-14 19:31:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:06 MarkDuplicates Read 9698119 records. 0 pairs never matched.
INFO 2021-03-14 19:31:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 996001384; totalMemory: 1619001344; maxMemory: 4294967296
INFO 2021-03-14 19:31:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:31:06 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:31:06 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:31:07 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:31:07 MarkDuplicates After generateDuplicateIndexes freeMemory: 1560780288; totalMemory: 2693791744; maxMemory: 4294967296
INFO 2021-03-14 19:31:07 MarkDuplicates Marking 4192238 records as duplicates.
INFO 2021-03-14 19:31:07 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:31:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:31:22 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:31:22 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:31:22 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:31:22 MarkDuplicates Before output close freeMemory: 258753944; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:31:22 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:31:22 MarkDuplicates After output close freeMemory: 258758808; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:31:22 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.56 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/474.sorted.bam O=../qc-processing/bowtie/deduplicated/474.dedup.bam M=../qc-processing/bowtie/deduplicated/474.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:31:24 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/475.sorted.bam -O ../qc-processing/bowtie/deduplicated/475.dedup.bam -M ../qc-processing/bowtie/deduplicated/475.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:31:24.207 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:31:24 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/475.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/475.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/475.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:31:24 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:31:24 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:31:24 MarkDuplicates Start of doWork freeMemory: 67872704; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:31:24 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:31:24 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:31:27 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3894:418
INFO 2021-03-14 19:31:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:29 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig22282:19,521
INFO 2021-03-14 19:31:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:30 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig33224:2,648
INFO 2021-03-14 19:31:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:32 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig41196:1,682
INFO 2021-03-14 19:31:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:33 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig54227:13,730
INFO 2021-03-14 19:31:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:35 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68908:739
INFO 2021-03-14 19:31:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:36 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig94163:7,985
INFO 2021-03-14 19:31:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:38 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig137890:7,842
INFO 2021-03-14 19:31:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:31:39 MarkDuplicates Read 8713997 records. 0 pairs never matched.
INFO 2021-03-14 19:31:39 MarkDuplicates After buildSortedReadEndLists freeMemory: 956256912; totalMemory: 1541406720; maxMemory: 4294967296
INFO 2021-03-14 19:31:39 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:31:40 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:31:40 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:31:41 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:31:41 MarkDuplicates After generateDuplicateIndexes freeMemory: 1483185624; totalMemory: 2616197120; maxMemory: 4294967296
INFO 2021-03-14 19:31:41 MarkDuplicates Marking 3806886 records as duplicates.
INFO 2021-03-14 19:31:41 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:31:42 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:31:54 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:31:54 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:31:54 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:31:54 MarkDuplicates Before output close freeMemory: 258754016; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:31:54 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:31:54 MarkDuplicates After output close freeMemory: 258758752; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:31:54 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.51 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/475.sorted.bam O=../qc-processing/bowtie/deduplicated/475.dedup.bam M=../qc-processing/bowtie/deduplicated/475.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:31:56 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/476.sorted.bam -O ../qc-processing/bowtie/deduplicated/476.dedup.bam -M ../qc-processing/bowtie/deduplicated/476.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:31:56.371 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:31:56 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/476.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/476.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/476.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:31:56 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:31:56 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:31:56 MarkDuplicates Start of doWork freeMemory: 72063472; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:31:56 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:31:56 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:31:59 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig19245:31,067
INFO 2021-03-14 19:31:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:01 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig35611:2,642
INFO 2021-03-14 19:32:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:02 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig54868:3,474
INFO 2021-03-14 19:32:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:04 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig81594:9,315
INFO 2021-03-14 19:32:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:06 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig171746:500
INFO 2021-03-14 19:32:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:06 MarkDuplicates Read 5228074 records. 0 pairs never matched.
INFO 2021-03-14 19:32:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 1036697328; totalMemory: 1475346432; maxMemory: 4294967296
INFO 2021-03-14 19:32:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:32:07 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:32:07 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:32:07 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:32:07 MarkDuplicates After generateDuplicateIndexes freeMemory: 1417126560; totalMemory: 2550136832; maxMemory: 4294967296
INFO 2021-03-14 19:32:07 MarkDuplicates Marking 2588387 records as duplicates.
INFO 2021-03-14 19:32:07 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:32:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:32:15 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:32:15 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:32:15 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:32:16 MarkDuplicates Before output close freeMemory: 258755248; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:32:16 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:32:16 MarkDuplicates After output close freeMemory: 258760112; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:32:16 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.33 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/476.sorted.bam O=../qc-processing/bowtie/deduplicated/476.dedup.bam M=../qc-processing/bowtie/deduplicated/476.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:32:17 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/477.sorted.bam -O ../qc-processing/bowtie/deduplicated/477.dedup.bam -M ../qc-processing/bowtie/deduplicated/477.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:32:17.886 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:32:17 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/477.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/477.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/477.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:32:18 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:32:18 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:32:18 MarkDuplicates Start of doWork freeMemory: 75209112; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:32:18 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:32:18 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:32:21 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5738:10,531
INFO 2021-03-14 19:32:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:22 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig23145:17,026
INFO 2021-03-14 19:32:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:24 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig32979:8,232
INFO 2021-03-14 19:32:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:25 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig42901:13,574
INFO 2021-03-14 19:32:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:27 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,567
INFO 2021-03-14 19:32:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:29 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig67725:7,957
INFO 2021-03-14 19:32:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:30 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig70819:7,362
INFO 2021-03-14 19:32:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:32 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig96963:44
INFO 2021-03-14 19:32:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:34 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig130752:1,764
INFO 2021-03-14 19:32:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:32:35 MarkDuplicates Read 9914264 records. 0 pairs never matched.
INFO 2021-03-14 19:32:35 MarkDuplicates After buildSortedReadEndLists freeMemory: 964600328; totalMemory: 1506803712; maxMemory: 4294967296
INFO 2021-03-14 19:32:35 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:32:36 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:32:36 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:32:37 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:32:37 MarkDuplicates After generateDuplicateIndexes freeMemory: 1448583488; totalMemory: 2581594112; maxMemory: 4294967296
INFO 2021-03-14 19:32:37 MarkDuplicates Marking 3025760 records as duplicates.
INFO 2021-03-14 19:32:37 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:32:37 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:32:53 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:32:53 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:32:53 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:32:53 MarkDuplicates Before output close freeMemory: 608302224; totalMemory: 2581594112; maxMemory: 4294967296
INFO 2021-03-14 19:32:53 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:32:54 MarkDuplicates After output close freeMemory: 262953888; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:32:54 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.60 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/477.sorted.bam O=../qc-processing/bowtie/deduplicated/477.dedup.bam M=../qc-processing/bowtie/deduplicated/477.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:32:55 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/481.sorted.bam -O ../qc-processing/bowtie/deduplicated/481.dedup.bam -M ../qc-processing/bowtie/deduplicated/481.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:32:55.845 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:32:55 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/481.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/481.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/481.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:32:56 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:32:56 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:32:56 MarkDuplicates Start of doWork freeMemory: 72061800; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:32:56 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:32:56 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:32:59 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig9588:937
INFO 2021-03-14 19:32:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:00 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig26849:3,199
INFO 2021-03-14 19:33:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:02 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35213:857
INFO 2021-03-14 19:33:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:04 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig47071:3,297
INFO 2021-03-14 19:33:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:05 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig61691:10,621
INFO 2021-03-14 19:33:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:07 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig70638:19,280
INFO 2021-03-14 19:33:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:08 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,703
INFO 2021-03-14 19:33:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:10 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig127092:1,360
INFO 2021-03-14 19:33:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:11 MarkDuplicates Read 8841925 records. 0 pairs never matched.
INFO 2021-03-14 19:33:11 MarkDuplicates After buildSortedReadEndLists freeMemory: 1048113824; totalMemory: 1580204032; maxMemory: 4294967296
INFO 2021-03-14 19:33:11 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:33:12 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:33:12 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:33:13 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:33:13 MarkDuplicates After generateDuplicateIndexes freeMemory: 1521983632; totalMemory: 2654994432; maxMemory: 4294967296
INFO 2021-03-14 19:33:13 MarkDuplicates Marking 3276199 records as duplicates.
INFO 2021-03-14 19:33:13 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:33:14 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:33:27 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:33:27 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:33:27 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:33:27 MarkDuplicates Before output close freeMemory: 258754544; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:33:27 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:33:27 MarkDuplicates After output close freeMemory: 258759408; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:33:27 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.53 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/481.sorted.bam O=../qc-processing/bowtie/deduplicated/481.dedup.bam M=../qc-processing/bowtie/deduplicated/481.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:33:29 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/482.sorted.bam -O ../qc-processing/bowtie/deduplicated/482.dedup.bam -M ../qc-processing/bowtie/deduplicated/482.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:33:29.156 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:33:29 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/482.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/482.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/482.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:33:29 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:33:29 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:33:29 MarkDuplicates Start of doWork freeMemory: 78354624; totalMemory: 104857600; maxMemory: 4294967296
INFO 2021-03-14 19:33:29 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:33:29 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:33:32 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig17598:27,552
INFO 2021-03-14 19:33:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:34 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig34828:35,108
INFO 2021-03-14 19:33:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:35 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig47617:7,008
INFO 2021-03-14 19:33:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:37 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig68908:721
INFO 2021-03-14 19:33:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:38 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig107026:2,482
INFO 2021-03-14 19:33:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:40 MarkDuplicates Read 5814560 records. 0 pairs never matched.
INFO 2021-03-14 19:33:40 MarkDuplicates After buildSortedReadEndLists freeMemory: 829146888; totalMemory: 1291845632; maxMemory: 4294967296
INFO 2021-03-14 19:33:40 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:33:40 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:33:40 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:33:41 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:33:41 MarkDuplicates After generateDuplicateIndexes freeMemory: 1233625136; totalMemory: 2366636032; maxMemory: 4294967296
INFO 2021-03-14 19:33:41 MarkDuplicates Marking 2852549 records as duplicates.
INFO 2021-03-14 19:33:41 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:33:42 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:33:50 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:33:50 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:33:50 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:33:50 MarkDuplicates Before output close freeMemory: 258754896; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:33:50 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:33:50 MarkDuplicates After output close freeMemory: 258759760; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:33:50 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.36 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/482.sorted.bam O=../qc-processing/bowtie/deduplicated/482.dedup.bam M=../qc-processing/bowtie/deduplicated/482.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:33:52 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/483.sorted.bam -O ../qc-processing/bowtie/deduplicated/483.dedup.bam -M ../qc-processing/bowtie/deduplicated/483.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:33:52.451 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:33:52 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/483.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/483.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/483.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:33:52 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:33:52 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:33:52 MarkDuplicates Start of doWork freeMemory: 72063480; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:33:52 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:33:52 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:33:55 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig18494:2,867
INFO 2021-03-14 19:33:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:57 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig33873:45,317
INFO 2021-03-14 19:33:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:33:58 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig48849:2,416
INFO 2021-03-14 19:33:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:00 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig61379:9,375
INFO 2021-03-14 19:34:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:01 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig72108:728
INFO 2021-03-14 19:34:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:03 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig103849:4,264
INFO 2021-03-14 19:34:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:04 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig686969:1,541
INFO 2021-03-14 19:34:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:04 MarkDuplicates Read 7022197 records. 0 pairs never matched.
INFO 2021-03-14 19:34:05 MarkDuplicates After buildSortedReadEndLists freeMemory: 861859400; totalMemory: 1317011456; maxMemory: 4294967296
INFO 2021-03-14 19:34:05 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:34:05 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:34:05 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:34:06 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:34:06 MarkDuplicates After generateDuplicateIndexes freeMemory: 1258790440; totalMemory: 2391801856; maxMemory: 4294967296
INFO 2021-03-14 19:34:06 MarkDuplicates Marking 2684548 records as duplicates.
INFO 2021-03-14 19:34:06 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:34:07 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:34:17 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:34:17 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:34:17 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:34:17 MarkDuplicates Before output close freeMemory: 266094800; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:34:17 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:34:17 MarkDuplicates After output close freeMemory: 258759632; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:34:17 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.42 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/483.sorted.bam O=../qc-processing/bowtie/deduplicated/483.dedup.bam M=../qc-processing/bowtie/deduplicated/483.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:34:19 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/484.sorted.bam -O ../qc-processing/bowtie/deduplicated/484.dedup.bam -M ../qc-processing/bowtie/deduplicated/484.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:34:19.487 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:34:19 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/484.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/484.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/484.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:34:19 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:34:19 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:34:19 MarkDuplicates Start of doWork freeMemory: 75209624; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:34:19 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:34:19 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:34:22 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig17421:16,748
INFO 2021-03-14 19:34:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:24 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,400
INFO 2021-03-14 19:34:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:26 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig42185:16,551
INFO 2021-03-14 19:34:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:27 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,510
INFO 2021-03-14 19:34:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:29 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig68226:1,172
INFO 2021-03-14 19:34:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:31 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig76179:5,879
INFO 2021-03-14 19:34:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:32 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,590
INFO 2021-03-14 19:34:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:34 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig216445:3,168
INFO 2021-03-14 19:34:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:34 MarkDuplicates Read 8210507 records. 0 pairs never matched.
INFO 2021-03-14 19:34:34 MarkDuplicates After buildSortedReadEndLists freeMemory: 939329536; totalMemory: 1394606080; maxMemory: 4294967296
INFO 2021-03-14 19:34:34 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:34:35 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:34:35 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:34:36 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:34:36 MarkDuplicates After generateDuplicateIndexes freeMemory: 1336385944; totalMemory: 2469396480; maxMemory: 4294967296
INFO 2021-03-14 19:34:36 MarkDuplicates Marking 2315855 records as duplicates.
INFO 2021-03-14 19:34:36 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:34:36 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:34:50 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:34:50 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:34:50 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:34:50 MarkDuplicates Before output close freeMemory: 268316432; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:34:50 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:34:50 MarkDuplicates After output close freeMemory: 262954520; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:34:50 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.52 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/484.sorted.bam O=../qc-processing/bowtie/deduplicated/484.dedup.bam M=../qc-processing/bowtie/deduplicated/484.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:34:52 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/485.sorted.bam -O ../qc-processing/bowtie/deduplicated/485.dedup.bam -M ../qc-processing/bowtie/deduplicated/485.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:34:52.533 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:34:52 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/485.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/485.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/485.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:34:52 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:34:52 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:34:52 MarkDuplicates Start of doWork freeMemory: 72063160; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:34:52 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:34:52 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:34:56 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4875:1,914
INFO 2021-03-14 19:34:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:57 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig22210:6,986
INFO 2021-03-14 19:34:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:34:59 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig32285:3,690
INFO 2021-03-14 19:34:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:01 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig40564:1,249
INFO 2021-03-14 19:35:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:02 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig53186:14,305
INFO 2021-03-14 19:35:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:04 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig66753:649
INFO 2021-03-14 19:35:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:06 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig79286:172
INFO 2021-03-14 19:35:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:07 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig103849:5,693
INFO 2021-03-14 19:35:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:09 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig170943:2,048
INFO 2021-03-14 19:35:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:10 MarkDuplicates Read 9461956 records. 0 pairs never matched.
INFO 2021-03-14 19:35:10 MarkDuplicates After buildSortedReadEndLists freeMemory: 885281456; totalMemory: 1447034880; maxMemory: 4294967296
INFO 2021-03-14 19:35:10 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:35:11 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:35:11 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:35:12 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:35:12 MarkDuplicates After generateDuplicateIndexes freeMemory: 1388814352; totalMemory: 2521825280; maxMemory: 4294967296
INFO 2021-03-14 19:35:12 MarkDuplicates Marking 3254875 records as duplicates.
INFO 2021-03-14 19:35:12 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:35:13 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:35:28 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:35:28 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:35:28 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:35:28 MarkDuplicates Before output close freeMemory: 268501840; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:35:28 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:35:28 MarkDuplicates After output close freeMemory: 258759472; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:35:28 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.60 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/485.sorted.bam O=../qc-processing/bowtie/deduplicated/485.dedup.bam M=../qc-processing/bowtie/deduplicated/485.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:35:30 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/487.sorted.bam -O ../qc-processing/bowtie/deduplicated/487.dedup.bam -M ../qc-processing/bowtie/deduplicated/487.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:35:30.087 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:35:30 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/487.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/487.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/487.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:35:30 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:35:30 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:35:30 MarkDuplicates Start of doWork freeMemory: 72063992; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:35:30 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:35:30 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:35:33 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4039:2,328
INFO 2021-03-14 19:35:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:34 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig20807:15,474
INFO 2021-03-14 19:35:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:36 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30354:336
INFO 2021-03-14 19:35:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:38 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig37352:9,663
INFO 2021-03-14 19:35:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:39 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig47662:11,151
INFO 2021-03-14 19:35:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:41 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig58365:4,740
INFO 2021-03-14 19:35:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:42 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig69925:5,140
INFO 2021-03-14 19:35:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:44 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig90554:2,739
INFO 2021-03-14 19:35:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:45 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig130284:6,300
INFO 2021-03-14 19:35:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:47 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig686969:942
INFO 2021-03-14 19:35:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:35:47 MarkDuplicates Read 10023321 records. 0 pairs never matched.
INFO 2021-03-14 19:35:47 MarkDuplicates After buildSortedReadEndLists freeMemory: 878890176; totalMemory: 1462763520; maxMemory: 4294967296
INFO 2021-03-14 19:35:47 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:35:48 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:35:48 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:35:49 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:35:49 MarkDuplicates After generateDuplicateIndexes freeMemory: 1404544480; totalMemory: 2537553920; maxMemory: 4294967296
INFO 2021-03-14 19:35:49 MarkDuplicates Marking 3669196 records as duplicates.
INFO 2021-03-14 19:35:49 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:35:50 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:36:10 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:36:10 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:36:10 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:36:10 MarkDuplicates Before output close freeMemory: 262949768; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:36:11 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:36:11 MarkDuplicates After output close freeMemory: 262954632; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:36:11 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.68 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/487.sorted.bam O=../qc-processing/bowtie/deduplicated/487.dedup.bam M=../qc-processing/bowtie/deduplicated/487.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:36:13 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/488.sorted.bam -O ../qc-processing/bowtie/deduplicated/488.dedup.bam -M ../qc-processing/bowtie/deduplicated/488.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:36:13.187 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:36:13 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/488.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/488.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/488.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:36:13 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:36:13 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:36:13 MarkDuplicates Start of doWork freeMemory: 72063320; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:36:13 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:36:13 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:36:16 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5705:6,477
INFO 2021-03-14 19:36:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:18 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig25135:13,109
INFO 2021-03-14 19:36:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:20 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig35897:1,267
INFO 2021-03-14 19:36:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:22 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig46998:5,895
INFO 2021-03-14 19:36:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:23 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig63531:4,723
INFO 2021-03-14 19:36:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:25 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig84462:3,128
INFO 2021-03-14 19:36:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:27 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig131913:3,111
INFO 2021-03-14 19:36:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:28 MarkDuplicates Read 7690308 records. 0 pairs never matched.
INFO 2021-03-14 19:36:28 MarkDuplicates After buildSortedReadEndLists freeMemory: 1082453320; totalMemory: 1639972864; maxMemory: 4294967296
INFO 2021-03-14 19:36:28 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:36:29 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:36:29 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:36:29 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:36:30 MarkDuplicates After generateDuplicateIndexes freeMemory: 1581754120; totalMemory: 2714763264; maxMemory: 4294967296
INFO 2021-03-14 19:36:30 MarkDuplicates Marking 3934291 records as duplicates.
INFO 2021-03-14 19:36:30 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:36:30 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:36:41 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:36:41 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:36:41 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:36:42 MarkDuplicates Before output close freeMemory: 262951064; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:36:42 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:36:42 MarkDuplicates After output close freeMemory: 262955928; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:36:42 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.48 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/488.sorted.bam O=../qc-processing/bowtie/deduplicated/488.dedup.bam M=../qc-processing/bowtie/deduplicated/488.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:36:43 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/489.sorted.bam -O ../qc-processing/bowtie/deduplicated/489.dedup.bam -M ../qc-processing/bowtie/deduplicated/489.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:36:44.000 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:36:44 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/489.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/489.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/489.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:36:44 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:36:44 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:36:44 MarkDuplicates Start of doWork freeMemory: 72063232; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:36:44 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:36:44 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:36:47 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5117:2,581
INFO 2021-03-14 19:36:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:49 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig21309:39,060
INFO 2021-03-14 19:36:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:51 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,647
INFO 2021-03-14 19:36:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:52 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig37707:7,419
INFO 2021-03-14 19:36:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:54 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig48737:5,387
INFO 2021-03-14 19:36:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:56 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig56693:16,400
INFO 2021-03-14 19:36:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:57 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig68037:397
INFO 2021-03-14 19:36:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:36:59 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig69234:14,487
INFO 2021-03-14 19:36:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:00 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig80528:7,130
INFO 2021-03-14 19:37:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:02 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig102196:3,221
INFO 2021-03-14 19:37:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:04 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig124730:2,506
INFO 2021-03-14 19:37:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:05 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:20s. Time for last 1,000,000: 1s. Last read position: Contig686969:920
INFO 2021-03-14 19:37:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:05 MarkDuplicates Read 12071698 records. 0 pairs never matched.
INFO 2021-03-14 19:37:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 857598720; totalMemory: 1448083456; maxMemory: 4294967296
INFO 2021-03-14 19:37:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:37:06 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:37:06 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:37:07 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:37:07 MarkDuplicates After generateDuplicateIndexes freeMemory: 1389863720; totalMemory: 2522873856; maxMemory: 4294967296
INFO 2021-03-14 19:37:07 MarkDuplicates Marking 3518731 records as duplicates.
INFO 2021-03-14 19:37:07 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:37:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:37:28 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:37:28 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:37:28 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:37:28 MarkDuplicates Before output close freeMemory: 261526072; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:37:28 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:37:28 MarkDuplicates After output close freeMemory: 258760216; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:37:28 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.74 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/489.sorted.bam O=../qc-processing/bowtie/deduplicated/489.dedup.bam M=../qc-processing/bowtie/deduplicated/489.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:37:30 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/490.sorted.bam -O ../qc-processing/bowtie/deduplicated/490.dedup.bam -M ../qc-processing/bowtie/deduplicated/490.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:37:30.323 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:37:30 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/490.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/490.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/490.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:37:30 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:37:30 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:37:30 MarkDuplicates Start of doWork freeMemory: 75211472; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:37:30 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:37:30 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:37:33 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig9662:3,258
INFO 2021-03-14 19:37:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:35 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig27262:3,772
INFO 2021-03-14 19:37:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:37 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig37777:9,221
INFO 2021-03-14 19:37:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:38 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig53624:3,055
INFO 2021-03-14 19:37:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:40 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,780
INFO 2021-03-14 19:37:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:42 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig95784:8,907
INFO 2021-03-14 19:37:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:43 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig149651:1,566
INFO 2021-03-14 19:37:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:37:44 MarkDuplicates Read 7472616 records. 0 pairs never matched.
INFO 2021-03-14 19:37:44 MarkDuplicates After buildSortedReadEndLists freeMemory: 953443192; totalMemory: 1442840576; maxMemory: 4294967296
INFO 2021-03-14 19:37:44 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:37:45 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:37:45 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:37:46 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:37:46 MarkDuplicates After generateDuplicateIndexes freeMemory: 1384621560; totalMemory: 2517630976; maxMemory: 4294967296
INFO 2021-03-14 19:37:46 MarkDuplicates Marking 2462323 records as duplicates.
INFO 2021-03-14 19:37:46 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:37:46 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:37:58 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:37:58 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:37:58 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:37:58 MarkDuplicates Before output close freeMemory: 320572408; totalMemory: 2517630976; maxMemory: 4294967296
INFO 2021-03-14 19:37:58 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:37:58 MarkDuplicates After output close freeMemory: 266100864; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 19:37:58 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.48 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/490.sorted.bam O=../qc-processing/bowtie/deduplicated/490.dedup.bam M=../qc-processing/bowtie/deduplicated/490.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:38:00 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/491.sorted.bam -O ../qc-processing/bowtie/deduplicated/491.dedup.bam -M ../qc-processing/bowtie/deduplicated/491.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:38:00.575 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:38:00 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/491.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/491.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/491.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:38:00 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:38:00 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:38:00 MarkDuplicates Start of doWork freeMemory: 75209008; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:38:00 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:38:00 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:38:03 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig17195:10,948
INFO 2021-03-14 19:38:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:05 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig33914:6,873
INFO 2021-03-14 19:38:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:07 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig45737:5,029
INFO 2021-03-14 19:38:07 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:08 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig68037:867
INFO 2021-03-14 19:38:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:10 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,705
INFO 2021-03-14 19:38:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:11 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig194027:711
INFO 2021-03-14 19:38:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:12 MarkDuplicates Read 6206096 records. 0 pairs never matched.
INFO 2021-03-14 19:38:12 MarkDuplicates After buildSortedReadEndLists freeMemory: 1129904288; totalMemory: 1611661312; maxMemory: 4294967296
INFO 2021-03-14 19:38:12 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:38:12 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:38:12 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:38:13 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:38:13 MarkDuplicates After generateDuplicateIndexes freeMemory: 1553441224; totalMemory: 2686451712; maxMemory: 4294967296
INFO 2021-03-14 19:38:13 MarkDuplicates Marking 3066255 records as duplicates.
INFO 2021-03-14 19:38:13 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:38:14 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:38:23 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:38:23 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:38:23 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:38:23 MarkDuplicates Before output close freeMemory: 255609624; totalMemory: 363855872; maxMemory: 4294967296
INFO 2021-03-14 19:38:23 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:38:23 MarkDuplicates After output close freeMemory: 255614488; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 19:38:23 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.38 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/491.sorted.bam O=../qc-processing/bowtie/deduplicated/491.dedup.bam M=../qc-processing/bowtie/deduplicated/491.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:38:25 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/492.sorted.bam -O ../qc-processing/bowtie/deduplicated/492.dedup.bam -M ../qc-processing/bowtie/deduplicated/492.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:38:25.559 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:38:25 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/492.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/492.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/492.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:38:25 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:38:25 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:38:25 MarkDuplicates Start of doWork freeMemory: 67871328; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:38:25 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:38:25 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:38:28 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5559:6,533
INFO 2021-03-14 19:38:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:30 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig23936:31,994
INFO 2021-03-14 19:38:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:31 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig34828:35,116
INFO 2021-03-14 19:38:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:33 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig42633:5,779
INFO 2021-03-14 19:38:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:35 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,466
INFO 2021-03-14 19:38:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:36 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,805
INFO 2021-03-14 19:38:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:38 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig97708:7,857
INFO 2021-03-14 19:38:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:39 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig144152:1,020
INFO 2021-03-14 19:38:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:38:40 MarkDuplicates Read 8632452 records. 0 pairs never matched.
INFO 2021-03-14 19:38:40 MarkDuplicates After buildSortedReadEndLists freeMemory: 929927048; totalMemory: 1517289472; maxMemory: 4294967296
INFO 2021-03-14 19:38:40 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:38:41 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:38:41 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:38:42 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:38:42 MarkDuplicates After generateDuplicateIndexes freeMemory: 1459069040; totalMemory: 2592079872; maxMemory: 4294967296
INFO 2021-03-14 19:38:42 MarkDuplicates Marking 4153952 records as duplicates.
INFO 2021-03-14 19:38:42 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:38:43 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:38:54 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:38:54 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:38:54 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:38:55 MarkDuplicates Before output close freeMemory: 269239976; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:38:55 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:38:55 MarkDuplicates After output close freeMemory: 262953256; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:38:55 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.49 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/492.sorted.bam O=../qc-processing/bowtie/deduplicated/492.dedup.bam M=../qc-processing/bowtie/deduplicated/492.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:38:56 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/506.sorted.bam -O ../qc-processing/bowtie/deduplicated/506.dedup.bam -M ../qc-processing/bowtie/deduplicated/506.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:38:56.843 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:38:56 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/506.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/506.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/506.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:38:57 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:38:57 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:38:57 MarkDuplicates Start of doWork freeMemory: 72061904; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:38:57 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:38:57 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:39:00 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig23487:5,041
INFO 2021-03-14 19:39:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:01 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig36557:11,339
INFO 2021-03-14 19:39:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:03 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig58273:4,648
INFO 2021-03-14 19:39:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:04 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig83649:2,416
INFO 2021-03-14 19:39:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:06 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig244871:164
INFO 2021-03-14 19:39:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:06 MarkDuplicates Read 5059600 records. 0 pairs never matched.
INFO 2021-03-14 19:39:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 986795048; totalMemory: 1405091840; maxMemory: 4294967296
INFO 2021-03-14 19:39:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:39:07 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:39:07 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:39:07 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:39:08 MarkDuplicates After generateDuplicateIndexes freeMemory: 1346871888; totalMemory: 2479882240; maxMemory: 4294967296
INFO 2021-03-14 19:39:08 MarkDuplicates Marking 2413267 records as duplicates.
INFO 2021-03-14 19:39:08 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:39:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:39:16 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:39:16 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:39:16 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:39:16 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:39:16 MarkDuplicates After output close freeMemory: 258760128; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:39:16 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.33 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/506.sorted.bam O=../qc-processing/bowtie/deduplicated/506.dedup.bam M=../qc-processing/bowtie/deduplicated/506.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:39:18 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/513.sorted.bam -O ../qc-processing/bowtie/deduplicated/513.dedup.bam -M ../qc-processing/bowtie/deduplicated/513.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:39:18.231 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:39:18 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/513.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/513.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/513.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:39:18 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:39:18 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:39:18 MarkDuplicates Start of doWork freeMemory: 75211216; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:39:18 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:39:18 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:39:21 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig7070:4,091
INFO 2021-03-14 19:39:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:23 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig25623:6,867
INFO 2021-03-14 19:39:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:24 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig35422:217
INFO 2021-03-14 19:39:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:26 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig47071:3,296
INFO 2021-03-14 19:39:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:27 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig62580:2,530
INFO 2021-03-14 19:39:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:29 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig77522:6,730
INFO 2021-03-14 19:39:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:31 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig109779:8,288
INFO 2021-03-14 19:39:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:32 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig464614:3,320
INFO 2021-03-14 19:39:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:32 MarkDuplicates Read 8095344 records. 0 pairs never matched.
INFO 2021-03-14 19:39:32 MarkDuplicates After buildSortedReadEndLists freeMemory: 1018064072; totalMemory: 1526726656; maxMemory: 4294967296
INFO 2021-03-14 19:39:32 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:39:33 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:39:33 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:39:34 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:39:34 MarkDuplicates After generateDuplicateIndexes freeMemory: 1468507688; totalMemory: 2601517056; maxMemory: 4294967296
INFO 2021-03-14 19:39:34 MarkDuplicates Marking 2671834 records as duplicates.
INFO 2021-03-14 19:39:34 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:39:35 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:39:47 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:39:47 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:39:47 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:39:47 MarkDuplicates Before output close freeMemory: 1074788392; totalMemory: 2601517056; maxMemory: 4294967296
INFO 2021-03-14 19:39:47 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:39:47 MarkDuplicates After output close freeMemory: 266101576; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 19:39:47 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.49 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/513.sorted.bam O=../qc-processing/bowtie/deduplicated/513.dedup.bam M=../qc-processing/bowtie/deduplicated/513.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:39:49 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/521.sorted.bam -O ../qc-processing/bowtie/deduplicated/521.dedup.bam -M ../qc-processing/bowtie/deduplicated/521.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:39:49.433 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:39:49 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/521.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/521.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/521.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:39:49 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:39:49 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:39:49 MarkDuplicates Start of doWork freeMemory: 64723368; totalMemory: 91226112; maxMemory: 4294967296
INFO 2021-03-14 19:39:49 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:39:49 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:39:52 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig5738:10,717
INFO 2021-03-14 19:39:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:54 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig22619:184
INFO 2021-03-14 19:39:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:55 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,671
INFO 2021-03-14 19:39:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:57 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig35213:849
INFO 2021-03-14 19:39:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:39:58 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig43834:3,625
INFO 2021-03-14 19:39:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:00 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,649
INFO 2021-03-14 19:40:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:01 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig59424:1,399
INFO 2021-03-14 19:40:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:03 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig68037:874
INFO 2021-03-14 19:40:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:05 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,891
INFO 2021-03-14 19:40:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:06 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig81807:6,003
INFO 2021-03-14 19:40:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:08 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,944
INFO 2021-03-14 19:40:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:10 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,216
INFO 2021-03-14 19:40:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:11 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig104701:4,801
INFO 2021-03-14 19:40:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:12 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:22s. Time for last 1,000,000: 1s. Last read position: Contig189029:2,207
INFO 2021-03-14 19:40:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:13 MarkDuplicates Read 14567238 records. 0 pairs never matched.
INFO 2021-03-14 19:40:14 MarkDuplicates After buildSortedReadEndLists freeMemory: 756731480; totalMemory: 1447034880; maxMemory: 4294967296
INFO 2021-03-14 19:40:14 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:40:14 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:40:14 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:40:15 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:40:16 MarkDuplicates After generateDuplicateIndexes freeMemory: 1388814896; totalMemory: 2521825280; maxMemory: 4294967296
INFO 2021-03-14 19:40:16 MarkDuplicates Marking 5999263 records as duplicates.
INFO 2021-03-14 19:40:16 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:40:17 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:40:38 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:40:38 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:40:38 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:40:38 MarkDuplicates Before output close freeMemory: 262948984; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:40:38 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:40:38 MarkDuplicates After output close freeMemory: 262953760; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:40:38 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.83 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/521.sorted.bam O=../qc-processing/bowtie/deduplicated/521.dedup.bam M=../qc-processing/bowtie/deduplicated/521.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:40:40 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/522.sorted.bam -O ../qc-processing/bowtie/deduplicated/522.dedup.bam -M ../qc-processing/bowtie/deduplicated/522.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:40:41.029 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:40:41 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/522.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/522.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/522.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:40:41 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:40:41 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:40:41 MarkDuplicates Start of doWork freeMemory: 72067024; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:40:41 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:40:41 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:40:44 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig17677:6,479
INFO 2021-03-14 19:40:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:45 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig34544:16,924
INFO 2021-03-14 19:40:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:47 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig48849:4,658
INFO 2021-03-14 19:40:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:49 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,795
INFO 2021-03-14 19:40:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:50 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig104701:4,801
INFO 2021-03-14 19:40:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:40:52 MarkDuplicates Read 5936529 records. 0 pairs never matched.
INFO 2021-03-14 19:40:52 MarkDuplicates After buildSortedReadEndLists freeMemory: 1063852136; totalMemory: 1514143744; maxMemory: 4294967296
INFO 2021-03-14 19:40:52 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:40:52 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:40:52 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:40:53 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:40:53 MarkDuplicates After generateDuplicateIndexes freeMemory: 1455923912; totalMemory: 2588934144; maxMemory: 4294967296
INFO 2021-03-14 19:40:53 MarkDuplicates Marking 2563826 records as duplicates.
INFO 2021-03-14 19:40:53 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:40:54 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:41:03 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:41:03 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:41:03 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:41:03 MarkDuplicates Before output close freeMemory: 2477761224; totalMemory: 2588934144; maxMemory: 4294967296
INFO 2021-03-14 19:41:03 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:41:03 MarkDuplicates After output close freeMemory: 266100072; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 19:41:03 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.37 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/522.sorted.bam O=../qc-processing/bowtie/deduplicated/522.dedup.bam M=../qc-processing/bowtie/deduplicated/522.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:41:04 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/523.sorted.bam -O ../qc-processing/bowtie/deduplicated/523.dedup.bam -M ../qc-processing/bowtie/deduplicated/523.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:41:04.972 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:41:05 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/523.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/523.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/523.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:41:05 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:41:05 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:41:05 MarkDuplicates Start of doWork freeMemory: 72062848; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:41:05 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:41:05 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:41:08 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15994:7,626
INFO 2021-03-14 19:41:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:09 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig28498:8,653
INFO 2021-03-14 19:41:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:11 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig37163:7,513
INFO 2021-03-14 19:41:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:12 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig51055:23,495
INFO 2021-03-14 19:41:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:14 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig56693:12,116
INFO 2021-03-14 19:41:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:16 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68037:457
INFO 2021-03-14 19:41:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:17 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,594
INFO 2021-03-14 19:41:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:19 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,673
INFO 2021-03-14 19:41:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:21 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig103849:5,732
INFO 2021-03-14 19:41:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:22 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig377195:7,036
INFO 2021-03-14 19:41:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:23 MarkDuplicates Read 10277262 records. 0 pairs never matched.
INFO 2021-03-14 19:41:23 MarkDuplicates After buildSortedReadEndLists freeMemory: 880220848; totalMemory: 1421869056; maxMemory: 4294967296
INFO 2021-03-14 19:41:23 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:41:23 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:41:23 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:41:24 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:41:24 MarkDuplicates After generateDuplicateIndexes freeMemory: 1363648672; totalMemory: 2496659456; maxMemory: 4294967296
INFO 2021-03-14 19:41:24 MarkDuplicates Marking 3740353 records as duplicates.
INFO 2021-03-14 19:41:24 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:41:25 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:41:41 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:41:41 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:41:41 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:41:41 MarkDuplicates Before output close freeMemory: 258755128; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:41:41 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:41:41 MarkDuplicates After output close freeMemory: 258759992; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:41:41 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.61 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/523.sorted.bam O=../qc-processing/bowtie/deduplicated/523.dedup.bam M=../qc-processing/bowtie/deduplicated/523.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:41:43 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/524.sorted.bam -O ../qc-processing/bowtie/deduplicated/524.dedup.bam -M ../qc-processing/bowtie/deduplicated/524.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:41:43.310 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:41:43 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/524.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/524.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/524.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:41:43 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:41:43 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:41:43 MarkDuplicates Start of doWork freeMemory: 72062536; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:41:43 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:41:43 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:41:46 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig8780:3,008
INFO 2021-03-14 19:41:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:48 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig29072:5,189
INFO 2021-03-14 19:41:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:49 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig40107:5,359
INFO 2021-03-14 19:41:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:51 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56878:5,805
INFO 2021-03-14 19:41:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:53 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig76520:6,511
INFO 2021-03-14 19:41:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:54 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig112919:3,766
INFO 2021-03-14 19:41:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:41:56 MarkDuplicates Read 6859685 records. 0 pairs never matched.
INFO 2021-03-14 19:41:56 MarkDuplicates After buildSortedReadEndLists freeMemory: 1175702504; totalMemory: 1674575872; maxMemory: 4294967296
INFO 2021-03-14 19:41:56 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:41:56 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:41:56 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:41:57 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:41:57 MarkDuplicates After generateDuplicateIndexes freeMemory: 1616356648; totalMemory: 2749366272; maxMemory: 4294967296
INFO 2021-03-14 19:41:57 MarkDuplicates Marking 3041510 records as duplicates.
INFO 2021-03-14 19:41:57 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:41:58 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:42:08 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:42:08 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:42:08 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:42:08 MarkDuplicates Before output close freeMemory: 262950104; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:42:08 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:42:08 MarkDuplicates After output close freeMemory: 262954968; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:42:08 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.42 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/524.sorted.bam O=../qc-processing/bowtie/deduplicated/524.dedup.bam M=../qc-processing/bowtie/deduplicated/524.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:42:10 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/525.sorted.bam -O ../qc-processing/bowtie/deduplicated/525.dedup.bam -M ../qc-processing/bowtie/deduplicated/525.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:42:10.110 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:42:10 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/525.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/525.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/525.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:42:10 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:42:10 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:42:10 MarkDuplicates Start of doWork freeMemory: 67869904; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:42:10 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:42:10 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:42:13 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15297:4,755
INFO 2021-03-14 19:42:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:14 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig27493:40,061
INFO 2021-03-14 19:42:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:16 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig37495:4,255
INFO 2021-03-14 19:42:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:18 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig52773:8,956
INFO 2021-03-14 19:42:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:19 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig66871:14,256
INFO 2021-03-14 19:42:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:21 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig77508:4,185
INFO 2021-03-14 19:42:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:23 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig104654:5,353
INFO 2021-03-14 19:42:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:24 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig350376:931
INFO 2021-03-14 19:42:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:24 MarkDuplicates Read 8136272 records. 0 pairs never matched.
INFO 2021-03-14 19:42:25 MarkDuplicates After buildSortedReadEndLists freeMemory: 910256072; totalMemory: 1394606080; maxMemory: 4294967296
INFO 2021-03-14 19:42:25 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:42:25 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:42:25 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:42:26 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:42:26 MarkDuplicates After generateDuplicateIndexes freeMemory: 1336387160; totalMemory: 2469396480; maxMemory: 4294967296
INFO 2021-03-14 19:42:26 MarkDuplicates Marking 2365706 records as duplicates.
INFO 2021-03-14 19:42:26 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:42:27 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:42:40 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:42:40 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:42:40 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:42:40 MarkDuplicates Before output close freeMemory: 263810496; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:42:40 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:42:40 MarkDuplicates After output close freeMemory: 258760800; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:42:40 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.51 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/525.sorted.bam O=../qc-processing/bowtie/deduplicated/525.dedup.bam M=../qc-processing/bowtie/deduplicated/525.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:42:42 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/526.sorted.bam -O ../qc-processing/bowtie/deduplicated/526.dedup.bam -M ../qc-processing/bowtie/deduplicated/526.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:42:42.794 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:42:42 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/526.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/526.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/526.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:42:43 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:42:43 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:42:43 MarkDuplicates Start of doWork freeMemory: 75207672; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:42:43 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:42:43 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:42:45 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16408:90
INFO 2021-03-14 19:42:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:47 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig30392:16,014
INFO 2021-03-14 19:42:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:49 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig42187:43,004
INFO 2021-03-14 19:42:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:51 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig57693:20,418
INFO 2021-03-14 19:42:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:52 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig72311:9,116
INFO 2021-03-14 19:42:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:54 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig106614:2,363
INFO 2021-03-14 19:42:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:42:55 MarkDuplicates Read 6971456 records. 0 pairs never matched.
INFO 2021-03-14 19:42:56 MarkDuplicates After buildSortedReadEndLists freeMemory: 1067266448; totalMemory: 1520435200; maxMemory: 4294967296
INFO 2021-03-14 19:42:56 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:42:56 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:42:56 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:42:57 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:42:57 MarkDuplicates After generateDuplicateIndexes freeMemory: 1462215200; totalMemory: 2595225600; maxMemory: 4294967296
INFO 2021-03-14 19:42:57 MarkDuplicates Marking 2039137 records as duplicates.
INFO 2021-03-14 19:42:57 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:42:57 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:43:09 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:43:09 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:43:09 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:43:09 MarkDuplicates Before output close freeMemory: 262949648; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:43:09 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:43:09 MarkDuplicates After output close freeMemory: 258760080; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:43:09 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.45 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/526.sorted.bam O=../qc-processing/bowtie/deduplicated/526.dedup.bam M=../qc-processing/bowtie/deduplicated/526.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:43:11 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/527.sorted.bam -O ../qc-processing/bowtie/deduplicated/527.dedup.bam -M ../qc-processing/bowtie/deduplicated/527.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:43:11.329 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:43:11 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/527.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/527.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/527.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:43:11 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:43:11 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:43:11 MarkDuplicates Start of doWork freeMemory: 67866032; totalMemory: 94371840; maxMemory: 4294967296
INFO 2021-03-14 19:43:11 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:43:11 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:43:14 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4963:3,727
INFO 2021-03-14 19:43:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:16 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig21655:48,522
INFO 2021-03-14 19:43:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:17 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig31694:9,160
INFO 2021-03-14 19:43:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:19 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig38003:36,047
INFO 2021-03-14 19:43:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:20 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig48415:702
INFO 2021-03-14 19:43:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:22 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig62449:8,893
INFO 2021-03-14 19:43:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:24 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig74232:7,680
INFO 2021-03-14 19:43:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:25 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig100843:1,750
INFO 2021-03-14 19:43:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:27 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig147865:528
INFO 2021-03-14 19:43:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:28 MarkDuplicates Read 9691585 records. 0 pairs never matched.
INFO 2021-03-14 19:43:29 MarkDuplicates After buildSortedReadEndLists freeMemory: 927788072; totalMemory: 1536163840; maxMemory: 4294967296
INFO 2021-03-14 19:43:29 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:43:29 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:43:29 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:43:30 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:43:30 MarkDuplicates After generateDuplicateIndexes freeMemory: 1477941832; totalMemory: 2610954240; maxMemory: 4294967296
INFO 2021-03-14 19:43:30 MarkDuplicates Marking 4248363 records as duplicates.
INFO 2021-03-14 19:43:30 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:43:31 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:43:44 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:43:44 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:43:44 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:43:45 MarkDuplicates Before output close freeMemory: 265940696; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:43:45 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:43:45 MarkDuplicates After output close freeMemory: 255611680; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 19:43:45 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.57 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/527.sorted.bam O=../qc-processing/bowtie/deduplicated/527.dedup.bam M=../qc-processing/bowtie/deduplicated/527.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:43:46 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/528.sorted.bam -O ../qc-processing/bowtie/deduplicated/528.dedup.bam -M ../qc-processing/bowtie/deduplicated/528.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:43:46.997 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:43:47 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/528.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/528.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/528.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:43:47 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:43:47 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:43:47 MarkDuplicates Start of doWork freeMemory: 72066856; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:43:47 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:43:47 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:43:51 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 3s. Last read position: Contig3033:32,612
INFO 2021-03-14 19:43:51 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:52 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig19473:29,368
INFO 2021-03-14 19:43:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:54 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig28449:44,557
INFO 2021-03-14 19:43:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:56 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig35611:2,642
INFO 2021-03-14 19:43:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:57 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig44034:8,556
INFO 2021-03-14 19:43:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:43:59 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig56205:4,740
INFO 2021-03-14 19:43:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:00 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig68908:720
INFO 2021-03-14 19:44:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:02 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig86903:2,595
INFO 2021-03-14 19:44:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:04 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig113539:2,903
INFO 2021-03-14 19:44:04 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:05 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:17s. Time for last 1,000,000: 1s. Last read position: Contig199624:4,427
INFO 2021-03-14 19:44:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:06 MarkDuplicates Read 10311803 records. 0 pairs never matched.
INFO 2021-03-14 19:44:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 715378688; totalMemory: 1355808768; maxMemory: 4294967296
INFO 2021-03-14 19:44:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:44:06 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:44:06 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:44:08 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:44:08 MarkDuplicates After generateDuplicateIndexes freeMemory: 1297591448; totalMemory: 2430599168; maxMemory: 4294967296
INFO 2021-03-14 19:44:08 MarkDuplicates Marking 4262114 records as duplicates.
INFO 2021-03-14 19:44:08 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:44:09 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:44:23 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:44:23 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:44:23 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:44:24 MarkDuplicates Before output close freeMemory: 262891456; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:44:24 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:44:24 MarkDuplicates After output close freeMemory: 262956864; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:44:24 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.62 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/528.sorted.bam O=../qc-processing/bowtie/deduplicated/528.dedup.bam M=../qc-processing/bowtie/deduplicated/528.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:44:25 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/529.sorted.bam -O ../qc-processing/bowtie/deduplicated/529.dedup.bam -M ../qc-processing/bowtie/deduplicated/529.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:44:25.905 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:44:26 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/529.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/529.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/529.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:44:26 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:44:26 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:44:26 MarkDuplicates Start of doWork freeMemory: 75209896; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:44:26 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:44:26 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:44:29 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15296:92
INFO 2021-03-14 19:44:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:30 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig30824:4,544
INFO 2021-03-14 19:44:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:32 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig41405:3,231
INFO 2021-03-14 19:44:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:33 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig56910:832
INFO 2021-03-14 19:44:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:35 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig77027:2,901
INFO 2021-03-14 19:44:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:37 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig110969:6,329
INFO 2021-03-14 19:44:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:38 MarkDuplicates Read 6828865 records. 0 pairs never matched.
INFO 2021-03-14 19:44:38 MarkDuplicates After buildSortedReadEndLists freeMemory: 909367136; totalMemory: 1428160512; maxMemory: 4294967296
INFO 2021-03-14 19:44:38 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:44:38 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:44:38 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:44:39 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:44:40 MarkDuplicates After generateDuplicateIndexes freeMemory: 1369941080; totalMemory: 2502950912; maxMemory: 4294967296
INFO 2021-03-14 19:44:40 MarkDuplicates Marking 3660497 records as duplicates.
INFO 2021-03-14 19:44:40 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:44:40 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:44:50 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:44:50 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:44:50 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:44:50 MarkDuplicates Before output close freeMemory: 262949928; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:44:50 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:44:50 MarkDuplicates After output close freeMemory: 262954792; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:44:50 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.41 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/529.sorted.bam O=../qc-processing/bowtie/deduplicated/529.dedup.bam M=../qc-processing/bowtie/deduplicated/529.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:44:52 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/531.sorted.bam -O ../qc-processing/bowtie/deduplicated/531.dedup.bam -M ../qc-processing/bowtie/deduplicated/531.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:44:52.080 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:44:52 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/531.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/531.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/531.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:44:52 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:44:52 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:44:52 MarkDuplicates Start of doWork freeMemory: 75211040; totalMemory: 101711872; maxMemory: 4294967296
INFO 2021-03-14 19:44:52 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:44:52 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:44:55 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig17280:32,422
INFO 2021-03-14 19:44:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:57 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig30702:6,241
INFO 2021-03-14 19:44:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:44:58 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig42187:42,961
INFO 2021-03-14 19:44:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:00 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56693:16,402
INFO 2021-03-14 19:45:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:01 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig69234:11,758
INFO 2021-03-14 19:45:01 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:03 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig89106:10,211
INFO 2021-03-14 19:45:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:05 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig142345:5,341
INFO 2021-03-14 19:45:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:06 MarkDuplicates Read 7531668 records. 0 pairs never matched.
INFO 2021-03-14 19:45:06 MarkDuplicates After buildSortedReadEndLists freeMemory: 1049901792; totalMemory: 1498415104; maxMemory: 4294967296
INFO 2021-03-14 19:45:06 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:45:06 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:45:06 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:45:07 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:45:07 MarkDuplicates After generateDuplicateIndexes freeMemory: 1440196544; totalMemory: 2573205504; maxMemory: 4294967296
INFO 2021-03-14 19:45:07 MarkDuplicates Marking 2058927 records as duplicates.
INFO 2021-03-14 19:45:07 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:45:08 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:45:20 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:45:20 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:45:20 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:45:20 MarkDuplicates Before output close freeMemory: 267371208; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:45:20 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:45:20 MarkDuplicates After output close freeMemory: 262955200; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:45:20 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.48 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/531.sorted.bam O=../qc-processing/bowtie/deduplicated/531.dedup.bam M=../qc-processing/bowtie/deduplicated/531.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:45:22 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/532.sorted.bam -O ../qc-processing/bowtie/deduplicated/532.dedup.bam -M ../qc-processing/bowtie/deduplicated/532.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:45:22.637 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:45:22 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/532.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/532.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/532.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:45:22 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:45:22 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:45:22 MarkDuplicates Start of doWork freeMemory: 64726040; totalMemory: 91226112; maxMemory: 4294967296
INFO 2021-03-14 19:45:22 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:45:22 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:45:26 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig17971:12,287
INFO 2021-03-14 19:45:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:28 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig34828:35,158
INFO 2021-03-14 19:45:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:29 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig48546:1,559
INFO 2021-03-14 19:45:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:31 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,805
INFO 2021-03-14 19:45:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:33 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig106636:3,219
INFO 2021-03-14 19:45:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:34 MarkDuplicates Read 5853149 records. 0 pairs never matched.
INFO 2021-03-14 19:45:35 MarkDuplicates After buildSortedReadEndLists freeMemory: 1043488136; totalMemory: 1485832192; maxMemory: 4294967296
INFO 2021-03-14 19:45:35 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:45:35 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:45:35 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:45:36 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:45:36 MarkDuplicates After generateDuplicateIndexes freeMemory: 1427612984; totalMemory: 2560622592; maxMemory: 4294967296
INFO 2021-03-14 19:45:36 MarkDuplicates Marking 2396922 records as duplicates.
INFO 2021-03-14 19:45:36 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:45:37 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:45:45 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:45:45 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:45:45 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:45:45 MarkDuplicates Before output close freeMemory: 262439720; totalMemory: 371195904; maxMemory: 4294967296
INFO 2021-03-14 19:45:45 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:45:45 MarkDuplicates After output close freeMemory: 258761192; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:45:45 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.39 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/532.sorted.bam O=../qc-processing/bowtie/deduplicated/532.dedup.bam M=../qc-processing/bowtie/deduplicated/532.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:45:47 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/533.sorted.bam -O ../qc-processing/bowtie/deduplicated/533.dedup.bam -M ../qc-processing/bowtie/deduplicated/533.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:45:47.582 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:45:47 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/533.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/533.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/533.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:45:47 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:45:47 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:45:47 MarkDuplicates Start of doWork freeMemory: 72063656; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:45:47 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:45:47 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:45:50 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig17423:1,904
INFO 2021-03-14 19:45:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:52 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,623
INFO 2021-03-14 19:45:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:54 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig42187:43,004
INFO 2021-03-14 19:45:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:55 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig56914:3,015
INFO 2021-03-14 19:45:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:57 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig69234:14,487
INFO 2021-03-14 19:45:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:45:59 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,944
INFO 2021-03-14 19:45:59 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:00 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig105506:8,208
INFO 2021-03-14 19:46:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:02 MarkDuplicates Read 7945481 records. 0 pairs never matched.
INFO 2021-03-14 19:46:02 MarkDuplicates After buildSortedReadEndLists freeMemory: 1101950312; totalMemory: 1566572544; maxMemory: 4294967296
INFO 2021-03-14 19:46:02 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:46:03 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:46:03 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:46:03 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:46:03 MarkDuplicates After generateDuplicateIndexes freeMemory: 1508352888; totalMemory: 2641362944; maxMemory: 4294967296
INFO 2021-03-14 19:46:03 MarkDuplicates Marking 2653991 records as duplicates.
INFO 2021-03-14 19:46:03 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:46:04 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:46:16 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:46:16 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:46:16 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:46:17 MarkDuplicates Before output close freeMemory: 264182944; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:46:17 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:46:17 MarkDuplicates After output close freeMemory: 258760136; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:46:17 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.49 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/533.sorted.bam O=../qc-processing/bowtie/deduplicated/533.dedup.bam M=../qc-processing/bowtie/deduplicated/533.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:46:18 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/541.sorted.bam -O ../qc-processing/bowtie/deduplicated/541.dedup.bam -M ../qc-processing/bowtie/deduplicated/541.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:46:18.920 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:46:18 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/541.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/541.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/541.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:46:19 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:46:19 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:46:19 MarkDuplicates Start of doWork freeMemory: 72063160; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:46:19 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:46:19 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:46:22 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig7365:3,399
INFO 2021-03-14 19:46:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:23 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig26524:20,789
INFO 2021-03-14 19:46:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:25 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig36557:11,314
INFO 2021-03-14 19:46:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:26 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig49166:6,379
INFO 2021-03-14 19:46:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:28 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig64626:2,316
INFO 2021-03-14 19:46:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:30 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig80528:5,723
INFO 2021-03-14 19:46:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:31 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig108457:1,295
INFO 2021-03-14 19:46:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:33 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig519002:3,294
INFO 2021-03-14 19:46:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:33 MarkDuplicates Read 8077792 records. 0 pairs never matched.
INFO 2021-03-14 19:46:33 MarkDuplicates After buildSortedReadEndLists freeMemory: 1175975472; totalMemory: 1703936000; maxMemory: 4294967296
INFO 2021-03-14 19:46:33 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:46:33 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:46:33 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:46:34 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:46:34 MarkDuplicates After generateDuplicateIndexes freeMemory: 1645716968; totalMemory: 2778726400; maxMemory: 4294967296
INFO 2021-03-14 19:46:34 MarkDuplicates Marking 3317152 records as duplicates.
INFO 2021-03-14 19:46:34 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:46:35 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:46:46 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:46:46 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:46:46 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:46:47 MarkDuplicates Before output close freeMemory: 264727344; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:46:47 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:46:47 MarkDuplicates After output close freeMemory: 258760848; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:46:47 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.47 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/541.sorted.bam O=../qc-processing/bowtie/deduplicated/541.dedup.bam M=../qc-processing/bowtie/deduplicated/541.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:46:49 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/542.sorted.bam -O ../qc-processing/bowtie/deduplicated/542.dedup.bam -M ../qc-processing/bowtie/deduplicated/542.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:46:49.063 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:46:49 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/542.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/542.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/542.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:46:49 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:46:49 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:46:49 MarkDuplicates Start of doWork freeMemory: 72061744; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:46:49 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:46:49 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:46:52 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig21157:3,497
INFO 2021-03-14 19:46:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:53 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig36439:1,862
INFO 2021-03-14 19:46:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:55 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig56320:792
INFO 2021-03-14 19:46:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:57 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig75494:648
INFO 2021-03-14 19:46:57 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:58 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig138833:1,679
INFO 2021-03-14 19:46:58 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:46:59 MarkDuplicates Read 5408988 records. 0 pairs never matched.
INFO 2021-03-14 19:46:59 MarkDuplicates After buildSortedReadEndLists freeMemory: 971684304; totalMemory: 1384120320; maxMemory: 4294967296
INFO 2021-03-14 19:46:59 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:46:59 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:46:59 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:47:00 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:47:00 MarkDuplicates After generateDuplicateIndexes freeMemory: 1325899632; totalMemory: 2458910720; maxMemory: 4294967296
INFO 2021-03-14 19:47:00 MarkDuplicates Marking 2196611 records as duplicates.
INFO 2021-03-14 19:47:00 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:47:01 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:47:09 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:47:09 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:47:09 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:47:10 MarkDuplicates Before output close freeMemory: 263833704; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:47:10 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:47:10 MarkDuplicates After output close freeMemory: 258760008; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:47:10 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.35 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/542.sorted.bam O=../qc-processing/bowtie/deduplicated/542.dedup.bam M=../qc-processing/bowtie/deduplicated/542.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:47:11 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/543.sorted.bam -O ../qc-processing/bowtie/deduplicated/543.dedup.bam -M ../qc-processing/bowtie/deduplicated/543.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:47:11.847 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:47:11 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/543.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/543.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/543.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:47:12 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:47:12 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:47:12 MarkDuplicates Start of doWork freeMemory: 72063848; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:47:12 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:47:12 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:47:15 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig21789:3,523
INFO 2021-03-14 19:47:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:16 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig39991:25,645
INFO 2021-03-14 19:47:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:18 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig68040:2,113
INFO 2021-03-14 19:47:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:20 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig121360:1,120
INFO 2021-03-14 19:47:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:21 MarkDuplicates Read 4536531 records. 0 pairs never matched.
INFO 2021-03-14 19:47:21 MarkDuplicates After buildSortedReadEndLists freeMemory: 911915584; totalMemory: 1300234240; maxMemory: 4294967296
INFO 2021-03-14 19:47:21 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:47:21 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:47:21 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:47:22 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:47:22 MarkDuplicates After generateDuplicateIndexes freeMemory: 1242014952; totalMemory: 2375024640; maxMemory: 4294967296
INFO 2021-03-14 19:47:22 MarkDuplicates Marking 1678199 records as duplicates.
INFO 2021-03-14 19:47:22 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:47:23 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:47:30 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:47:30 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:47:30 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:47:30 MarkDuplicates Before output close freeMemory: 264429440; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:47:30 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:47:30 MarkDuplicates After output close freeMemory: 255614256; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 19:47:30 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.31 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/543.sorted.bam O=../qc-processing/bowtie/deduplicated/543.dedup.bam M=../qc-processing/bowtie/deduplicated/543.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:47:32 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/551.sorted.bam -O ../qc-processing/bowtie/deduplicated/551.dedup.bam -M ../qc-processing/bowtie/deduplicated/551.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:47:32.305 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:47:32 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/551.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/551.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/551.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:47:32 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:47:32 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:47:32 MarkDuplicates Start of doWork freeMemory: 72065912; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:47:32 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:47:32 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:47:36 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 3s. Last read position: Contig5240:2,172
INFO 2021-03-14 19:47:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:38 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig23691:9,001
INFO 2021-03-14 19:47:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:39 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig34634:538
INFO 2021-03-14 19:47:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:41 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig43024:7,290
INFO 2021-03-14 19:47:41 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:42 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:09s. Time for last 1,000,000: 1s. Last read position: Contig56693:18,541
INFO 2021-03-14 19:47:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:44 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,842
INFO 2021-03-14 19:47:44 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:46 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig97936:72
INFO 2021-03-14 19:47:46 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:47 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:14s. Time for last 1,000,000: 1s. Last read position: Contig137674:181
INFO 2021-03-14 19:47:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:47:49 MarkDuplicates Read 8767880 records. 0 pairs never matched.
INFO 2021-03-14 19:47:49 MarkDuplicates After buildSortedReadEndLists freeMemory: 941298480; totalMemory: 1515192320; maxMemory: 4294967296
INFO 2021-03-14 19:47:49 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:47:49 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:47:49 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:47:50 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:47:51 MarkDuplicates After generateDuplicateIndexes freeMemory: 1456972424; totalMemory: 2589982720; maxMemory: 4294967296
INFO 2021-03-14 19:47:51 MarkDuplicates Marking 3670108 records as duplicates.
INFO 2021-03-14 19:47:51 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:47:51 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:48:04 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:48:04 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:48:04 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:48:04 MarkDuplicates Before output close freeMemory: 266095368; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:48:04 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:48:04 MarkDuplicates After output close freeMemory: 262954376; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:48:04 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.54 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/551.sorted.bam O=../qc-processing/bowtie/deduplicated/551.dedup.bam M=../qc-processing/bowtie/deduplicated/551.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:48:06 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/552b.sorted.bam -O ../qc-processing/bowtie/deduplicated/552.dedup.bam -M ../qc-processing/bowtie/deduplicated/552.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:48:06.378 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:48:06 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/552b.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/552.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/552.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:48:06 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:48:06 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:48:06 MarkDuplicates Start of doWork freeMemory: 72064160; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:48:06 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:48:06 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:48:09 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig23081:7,479
INFO 2021-03-14 19:48:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:11 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig41246:5,505
INFO 2021-03-14 19:48:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:12 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig63634:12,888
INFO 2021-03-14 19:48:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:14 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig95451:4,647
INFO 2021-03-14 19:48:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:16 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig328985:1,525
INFO 2021-03-14 19:48:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:16 MarkDuplicates Read 5084900 records. 0 pairs never matched.
INFO 2021-03-14 19:48:16 MarkDuplicates After buildSortedReadEndLists freeMemory: 932566936; totalMemory: 1325400064; maxMemory: 4294967296
INFO 2021-03-14 19:48:16 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:48:16 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:48:16 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:48:17 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:48:17 MarkDuplicates After generateDuplicateIndexes freeMemory: 1267180128; totalMemory: 2400190464; maxMemory: 4294967296
INFO 2021-03-14 19:48:17 MarkDuplicates Marking 1927245 records as duplicates.
INFO 2021-03-14 19:48:17 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:48:18 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:48:26 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:48:26 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:48:26 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:48:26 MarkDuplicates Before output close freeMemory: 266095344; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:48:26 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:48:27 MarkDuplicates After output close freeMemory: 266100080; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 19:48:27 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.34 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/552b.sorted.bam O=../qc-processing/bowtie/deduplicated/552.dedup.bam M=../qc-processing/bowtie/deduplicated/552.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:48:28 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/553.sorted.bam -O ../qc-processing/bowtie/deduplicated/553.dedup.bam -M ../qc-processing/bowtie/deduplicated/553.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:48:28.689 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:48:28 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/553.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/553.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/553.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:48:28 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:48:28 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:48:28 MarkDuplicates Start of doWork freeMemory: 72065608; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:48:28 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:48:28 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:48:31 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig16674:48,304
INFO 2021-03-14 19:48:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:33 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,367
INFO 2021-03-14 19:48:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:35 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig43398:15,272
INFO 2021-03-14 19:48:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:36 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig59179:6,387
INFO 2021-03-14 19:48:36 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:38 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig77227:5,921
INFO 2021-03-14 19:48:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:40 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig112678:1,107
INFO 2021-03-14 19:48:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:48:41 MarkDuplicates Read 6836851 records. 0 pairs never matched.
INFO 2021-03-14 19:48:41 MarkDuplicates After buildSortedReadEndLists freeMemory: 1060942640; totalMemory: 1509949440; maxMemory: 4294967296
INFO 2021-03-14 19:48:41 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:48:42 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:48:42 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:48:42 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:48:43 MarkDuplicates After generateDuplicateIndexes freeMemory: 1451728504; totalMemory: 2584739840; maxMemory: 4294967296
INFO 2021-03-14 19:48:43 MarkDuplicates Marking 1978470 records as duplicates.
INFO 2021-03-14 19:48:43 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:48:43 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:48:55 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:48:55 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:48:55 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:48:55 MarkDuplicates Before output close freeMemory: 1099596920; totalMemory: 2584739840; maxMemory: 4294967296
INFO 2021-03-14 19:48:55 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:48:55 MarkDuplicates After output close freeMemory: 266098984; totalMemory: 374341632; maxMemory: 4294967296
[Sun Mar 14 19:48:55 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.45 minutes.
Runtime.totalMemory()=374341632
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/553.sorted.bam O=../qc-processing/bowtie/deduplicated/553.dedup.bam M=../qc-processing/bowtie/deduplicated/553.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:48:57 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/554.sorted.bam -O ../qc-processing/bowtie/deduplicated/554.dedup.bam -M ../qc-processing/bowtie/deduplicated/554.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:48:57.142 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:48:57 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/554.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/554.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/554.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:48:57 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:48:57 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:48:57 MarkDuplicates Start of doWork freeMemory: 72063040; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:48:57 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:48:57 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:49:00 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig3374:5,260
INFO 2021-03-14 19:49:00 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:02 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:04s. Time for last 1,000,000: 1s. Last read position: Contig19567:313
INFO 2021-03-14 19:49:02 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:03 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig28227:13,555
INFO 2021-03-14 19:49:03 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:05 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig35213:853
INFO 2021-03-14 19:49:05 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:06 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig43340:17,694
INFO 2021-03-14 19:49:06 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:08 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig55470:2,632
INFO 2021-03-14 19:49:08 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:10 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 1s. Last read position: Contig60316:8,758
INFO 2021-03-14 19:49:10 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:11 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig69234:9,024
INFO 2021-03-14 19:49:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:13 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig75641:16,987
INFO 2021-03-14 19:49:13 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:14 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig97129:507
INFO 2021-03-14 19:49:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:16 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig118770:6,331
INFO 2021-03-14 19:49:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:18 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig206328:1,384
INFO 2021-03-14 19:49:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:18 MarkDuplicates Read 12330905 records. 0 pairs never matched.
INFO 2021-03-14 19:49:18 MarkDuplicates After buildSortedReadEndLists freeMemory: 882899224; totalMemory: 1509949440; maxMemory: 4294967296
INFO 2021-03-14 19:49:18 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:49:19 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:49:19 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:49:20 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:49:20 MarkDuplicates After generateDuplicateIndexes freeMemory: 1451729408; totalMemory: 2584739840; maxMemory: 4294967296
INFO 2021-03-14 19:49:20 MarkDuplicates Marking 3764674 records as duplicates.
INFO 2021-03-14 19:49:20 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:49:21 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:49:40 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:49:40 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:49:40 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:49:40 MarkDuplicates Before output close freeMemory: 267132624; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:49:40 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:49:40 MarkDuplicates After output close freeMemory: 262954888; totalMemory: 371195904; maxMemory: 4294967296
[Sun Mar 14 19:49:40 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.73 minutes.
Runtime.totalMemory()=371195904
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/554.sorted.bam O=../qc-processing/bowtie/deduplicated/554.dedup.bam M=../qc-processing/bowtie/deduplicated/554.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:49:42 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/561.sorted.bam -O ../qc-processing/bowtie/deduplicated/561.dedup.bam -M ../qc-processing/bowtie/deduplicated/561.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:49:42.549 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:49:42 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/561.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/561.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/561.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:49:42 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:49:42 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:49:42 MarkDuplicates Start of doWork freeMemory: 72059032; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:49:42 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:49:42 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:49:45 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig18306:4,271
INFO 2021-03-14 19:49:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:47 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig35176:38,138
INFO 2021-03-14 19:49:47 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:48 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig47182:16,906
INFO 2021-03-14 19:49:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:50 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig68908:722
INFO 2021-03-14 19:49:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:52 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,750
INFO 2021-03-14 19:49:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:53 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig208931:6,140
INFO 2021-03-14 19:49:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:49:53 MarkDuplicates Read 6112441 records. 0 pairs never matched.
INFO 2021-03-14 19:49:53 MarkDuplicates After buildSortedReadEndLists freeMemory: 1104628744; totalMemory: 1579155456; maxMemory: 4294967296
INFO 2021-03-14 19:49:53 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:49:54 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:49:54 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:49:55 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:49:55 MarkDuplicates After generateDuplicateIndexes freeMemory: 1520931088; totalMemory: 2653945856; maxMemory: 4294967296
INFO 2021-03-14 19:49:55 MarkDuplicates Marking 3655331 records as duplicates.
INFO 2021-03-14 19:49:55 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:49:55 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:50:04 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:50:04 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:50:04 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:50:04 MarkDuplicates Before output close freeMemory: 258380848; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:50:04 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:50:04 MarkDuplicates After output close freeMemory: 258755880; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:50:04 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.37 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/561.sorted.bam O=../qc-processing/bowtie/deduplicated/561.dedup.bam M=../qc-processing/bowtie/deduplicated/561.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:50:06 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/562.sorted.bam -O ../qc-processing/bowtie/deduplicated/562.dedup.bam -M ../qc-processing/bowtie/deduplicated/562.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:50:06.443 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:50:06 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/562.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/562.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/562.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:50:06 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:50:06 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:50:06 MarkDuplicates Start of doWork freeMemory: 72059880; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:50:06 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:50:06 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:50:09 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig4449:10,529
INFO 2021-03-14 19:50:09 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:11 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:12 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig27102:353
INFO 2021-03-14 19:50:12 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:14 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,671
INFO 2021-03-14 19:50:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:15 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig33895:2,039
INFO 2021-03-14 19:50:15 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:17 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig41285:7,002
INFO 2021-03-14 19:50:17 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:19 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig48849:2,198
INFO 2021-03-14 19:50:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:20 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig56693:7,512
INFO 2021-03-14 19:50:20 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:22 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig56910:920
INFO 2021-03-14 19:50:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:23 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig68037:746
INFO 2021-03-14 19:50:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:25 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,784
INFO 2021-03-14 19:50:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:27 MarkDuplicates Read 12,000,000 records. Elapsed time: 00:00:19s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,891
INFO 2021-03-14 19:50:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:29 MarkDuplicates Read 13,000,000 records. Elapsed time: 00:00:21s. Time for last 1,000,000: 1s. Last read position: Contig80528:7,151
INFO 2021-03-14 19:50:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:30 MarkDuplicates Read 14,000,000 records. Elapsed time: 00:00:23s. Time for last 1,000,000: 1s. Last read position: Contig85486:1,944
INFO 2021-03-14 19:50:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:32 MarkDuplicates Read 15,000,000 records. Elapsed time: 00:00:25s. Time for last 1,000,000: 1s. Last read position: Contig98047:1,310
INFO 2021-03-14 19:50:32 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:34 MarkDuplicates Read 16,000,000 records. Elapsed time: 00:00:26s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,231
INFO 2021-03-14 19:50:34 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:35 MarkDuplicates Read 17,000,000 records. Elapsed time: 00:00:28s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:50:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:37 MarkDuplicates Read 18,000,000 records. Elapsed time: 00:00:29s. Time for last 1,000,000: 1s. Last read position: Contig103886:807
INFO 2021-03-14 19:50:37 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:38 MarkDuplicates Read 19,000,000 records. Elapsed time: 00:00:31s. Time for last 1,000,000: 1s. Last read position: Contig103886:900
INFO 2021-03-14 19:50:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:40 MarkDuplicates Read 20,000,000 records. Elapsed time: 00:00:33s. Time for last 1,000,000: 1s. Last read position: Contig144382:1,153
INFO 2021-03-14 19:50:40 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:42 MarkDuplicates Read 21,000,000 records. Elapsed time: 00:00:34s. Time for last 1,000,000: 1s. Last read position: Contig686969:1,553
INFO 2021-03-14 19:50:42 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:50:42 MarkDuplicates Read 21195148 records. 0 pairs never matched.
INFO 2021-03-14 19:50:42 MarkDuplicates After buildSortedReadEndLists freeMemory: 658930704; totalMemory: 1561329664; maxMemory: 4294967296
INFO 2021-03-14 19:50:42 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:50:43 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:50:43 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:50:45 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:50:45 MarkDuplicates After generateDuplicateIndexes freeMemory: 1503106608; totalMemory: 2636120064; maxMemory: 4294967296
INFO 2021-03-14 19:50:45 MarkDuplicates Marking 8814091 records as duplicates.
INFO 2021-03-14 19:50:45 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:50:46 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:51:11 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:00:24s. Time for last 10,000,000: 24s. Last read position: Contig103849:3,274
INFO 2021-03-14 19:51:16 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:51:16 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:51:16 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:51:17 MarkDuplicates Before output close freeMemory: 267424936; totalMemory: 377487360; maxMemory: 4294967296
INFO 2021-03-14 19:51:17 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:51:17 MarkDuplicates After output close freeMemory: 258757000; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:51:17 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 1.18 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/562.sorted.bam O=../qc-processing/bowtie/deduplicated/562.dedup.bam M=../qc-processing/bowtie/deduplicated/562.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:51:18 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/563.sorted.bam -O ../qc-processing/bowtie/deduplicated/563.dedup.bam -M ../qc-processing/bowtie/deduplicated/563.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:51:18.876 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:51:19 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/563.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/563.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/563.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:51:19 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:51:19 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:51:19 MarkDuplicates Start of doWork freeMemory: 72063624; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:51:19 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:51:19 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:51:22 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig15963:2,716
INFO 2021-03-14 19:51:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:23 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig31901:3,022
INFO 2021-03-14 19:51:23 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:25 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig41518:3,079
INFO 2021-03-14 19:51:25 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:26 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig58345:594
INFO 2021-03-14 19:51:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:28 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig77299:5,429
INFO 2021-03-14 19:51:28 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:30 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig116145:3,044
INFO 2021-03-14 19:51:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:31 MarkDuplicates Read 6763523 records. 0 pairs never matched.
INFO 2021-03-14 19:51:31 MarkDuplicates After buildSortedReadEndLists freeMemory: 1185592592; totalMemory: 1697644544; maxMemory: 4294967296
INFO 2021-03-14 19:51:31 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:51:32 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:51:32 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:51:33 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:51:33 MarkDuplicates After generateDuplicateIndexes freeMemory: 1639425256; totalMemory: 2772434944; maxMemory: 4294967296
INFO 2021-03-14 19:51:33 MarkDuplicates Marking 3422885 records as duplicates.
INFO 2021-03-14 19:51:33 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:51:34 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:51:43 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:51:43 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:51:43 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:51:43 MarkDuplicates Before output close freeMemory: 258756336; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:51:43 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:51:43 MarkDuplicates After output close freeMemory: 258761200; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:51:43 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.42 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/563.sorted.bam O=../qc-processing/bowtie/deduplicated/563.dedup.bam M=../qc-processing/bowtie/deduplicated/563.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:51:45 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/564.sorted.bam -O ../qc-processing/bowtie/deduplicated/564.dedup.bam -M ../qc-processing/bowtie/deduplicated/564.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:51:45.472 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:51:45 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/564.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/564.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/564.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:51:45 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:51:45 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:51:45 MarkDuplicates Start of doWork freeMemory: 72064864; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:51:45 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:51:45 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:51:48 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig18799:2,181
INFO 2021-03-14 19:51:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:50 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig34117:26,367
INFO 2021-03-14 19:51:50 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:52 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig50592:2,032
INFO 2021-03-14 19:51:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:53 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:07s. Time for last 1,000,000: 1s. Last read position: Contig68908:2,780
INFO 2021-03-14 19:51:53 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:55 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig98974:6,007
INFO 2021-03-14 19:51:55 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:56 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig213985:2,816
INFO 2021-03-14 19:51:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:51:57 MarkDuplicates Read 6128374 records. 0 pairs never matched.
INFO 2021-03-14 19:51:57 MarkDuplicates After buildSortedReadEndLists freeMemory: 991234648; totalMemory: 1409286144; maxMemory: 4294967296
INFO 2021-03-14 19:51:57 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:51:57 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:51:57 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:51:58 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:51:58 MarkDuplicates After generateDuplicateIndexes freeMemory: 1351067920; totalMemory: 2484076544; maxMemory: 4294967296
INFO 2021-03-14 19:51:58 MarkDuplicates Marking 1780947 records as duplicates.
INFO 2021-03-14 19:51:58 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:51:59 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:52:09 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:52:09 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:52:09 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:52:09 MarkDuplicates Before output close freeMemory: 258229080; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:52:09 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:52:09 MarkDuplicates After output close freeMemory: 258760912; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:52:09 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.41 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/564.sorted.bam O=../qc-processing/bowtie/deduplicated/564.dedup.bam M=../qc-processing/bowtie/deduplicated/564.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:52:11 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/565.sorted.bam -O ../qc-processing/bowtie/deduplicated/565.dedup.bam -M ../qc-processing/bowtie/deduplicated/565.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:52:11.688 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:52:11 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/565.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/565.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/565.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:52:11 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:52:11 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:52:11 MarkDuplicates Start of doWork freeMemory: 64724848; totalMemory: 91226112; maxMemory: 4294967296
INFO 2021-03-14 19:52:11 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:52:11 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:52:14 MarkDuplicates Read 1,000,000 records. Elapsed time: 00:00:02s. Time for last 1,000,000: 2s. Last read position: Contig19099:2,138
INFO 2021-03-14 19:52:14 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:16 MarkDuplicates Read 2,000,000 records. Elapsed time: 00:00:03s. Time for last 1,000,000: 1s. Last read position: Contig30879:27,427
INFO 2021-03-14 19:52:16 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:18 MarkDuplicates Read 3,000,000 records. Elapsed time: 00:00:05s. Time for last 1,000,000: 1s. Last read position: Contig36557:11,341
INFO 2021-03-14 19:52:18 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:19 MarkDuplicates Read 4,000,000 records. Elapsed time: 00:00:06s. Time for last 1,000,000: 1s. Last read position: Contig56693:2,539
INFO 2021-03-14 19:52:19 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:21 MarkDuplicates Read 5,000,000 records. Elapsed time: 00:00:08s. Time for last 1,000,000: 1s. Last read position: Contig61521:7,844
INFO 2021-03-14 19:52:21 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:22 MarkDuplicates Read 6,000,000 records. Elapsed time: 00:00:10s. Time for last 1,000,000: 1s. Last read position: Contig68037:1,092
INFO 2021-03-14 19:52:22 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:24 MarkDuplicates Read 7,000,000 records. Elapsed time: 00:00:11s. Time for last 1,000,000: 1s. Last read position: Contig70638:18,891
INFO 2021-03-14 19:52:24 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:26 MarkDuplicates Read 8,000,000 records. Elapsed time: 00:00:13s. Time for last 1,000,000: 1s. Last read position: Contig82803:4,335
INFO 2021-03-14 19:52:26 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:27 MarkDuplicates Read 9,000,000 records. Elapsed time: 00:00:15s. Time for last 1,000,000: 1s. Last read position: Contig96898:5,336
INFO 2021-03-14 19:52:27 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:29 MarkDuplicates Read 10,000,000 records. Elapsed time: 00:00:16s. Time for last 1,000,000: 1s. Last read position: Contig103849:3,489
INFO 2021-03-14 19:52:29 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:30 MarkDuplicates Read 11,000,000 records. Elapsed time: 00:00:18s. Time for last 1,000,000: 1s. Last read position: Contig206781:4,052
INFO 2021-03-14 19:52:30 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2021-03-14 19:52:31 MarkDuplicates Read 11393656 records. 0 pairs never matched.
INFO 2021-03-14 19:52:31 MarkDuplicates After buildSortedReadEndLists freeMemory: 1017065280; totalMemory: 1481637888; maxMemory: 4294967296
INFO 2021-03-14 19:52:31 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:52:31 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:52:31 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:52:32 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:52:32 MarkDuplicates After generateDuplicateIndexes freeMemory: 1423418832; totalMemory: 2556428288; maxMemory: 4294967296
INFO 2021-03-14 19:52:32 MarkDuplicates Marking 3348207 records as duplicates.
INFO 2021-03-14 19:52:32 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:52:33 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:52:52 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:52:52 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:52:52 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:52:52 MarkDuplicates Before output close freeMemory: 264352560; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:52:52 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:52:52 MarkDuplicates After output close freeMemory: 258760424; totalMemory: 367001600; maxMemory: 4294967296
[Sun Mar 14 19:52:52 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.68 minutes.
Runtime.totalMemory()=367001600
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/565.sorted.bam O=../qc-processing/bowtie/deduplicated/565.dedup.bam M=../qc-processing/bowtie/deduplicated/565.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:52:54 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/571.sorted.bam -O ../qc-processing/bowtie/deduplicated/571.dedup.bam -M ../qc-processing/bowtie/deduplicated/571.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:52:54.324 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:52:54 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/571.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/571.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/571.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:52:54 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:52:54 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:52:54 MarkDuplicates Start of doWork freeMemory: 72065608; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:52:54 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:52:54 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:52:57 MarkDuplicates Read 611951 records. 0 pairs never matched.
INFO 2021-03-14 19:52:57 MarkDuplicates After buildSortedReadEndLists freeMemory: 480866520; totalMemory: 681574400; maxMemory: 4294967296
INFO 2021-03-14 19:52:57 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:52:57 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:52:57 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:52:57 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:52:57 MarkDuplicates After generateDuplicateIndexes freeMemory: 761768440; totalMemory: 1894776832; maxMemory: 4294967296
INFO 2021-03-14 19:52:57 MarkDuplicates Marking 204170 records as duplicates.
INFO 2021-03-14 19:52:57 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:52:58 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:52:59 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:52:59 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:52:59 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:53:00 MarkDuplicates Before output close freeMemory: 257578576; totalMemory: 367001600; maxMemory: 4294967296
INFO 2021-03-14 19:53:00 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:53:00 MarkDuplicates After output close freeMemory: 255616104; totalMemory: 363855872; maxMemory: 4294967296
[Sun Mar 14 19:53:00 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.10 minutes.
Runtime.totalMemory()=363855872
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/571.sorted.bam O=../qc-processing/bowtie/deduplicated/571.dedup.bam M=../qc-processing/bowtie/deduplicated/571.dup_metrics.txt REMOVE_DUPLICATES=true
INFO 2021-03-14 19:53:01 MarkDuplicates
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** MarkDuplicates -I ../qc-processing/bowtie/mapped-sorted/Undetermined.sorted.bam -O ../qc-processing/bowtie/deduplicated/Und.dedup.bam -M ../qc-processing/bowtie/deduplicated/Und.dup_metrics.txt -REMOVE_DUPLICATES true
**********
18:53:01.865 INFO NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib
[Sun Mar 14 19:53:01 PDT 2021] MarkDuplicates INPUT=[../qc-processing/bowtie/mapped-sorted/Undetermined.sorted.bam] OUTPUT=../qc-processing/bowtie/deduplicated/Und.dedup.bam METRICS_FILE=../qc-processing/bowtie/deduplicated/Und.dup_metrics.txt REMOVE_DUPLICATES=true MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag CLEAR_DT=true DUPLEX_UMI=false ADD_PG_TAG_TO_READS=true ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 MAX_OPTICAL_DUPLICATE_SET_SIZE=300000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
Mar 14, 2021 7:53:02 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
[Sun Mar 14 19:53:02 PDT 2021] Executing as laura@lauras-MacBook-Pro-2.local on Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.1.9.0
INFO 2021-03-14 19:53:02 MarkDuplicates Start of doWork freeMemory: 72062944; totalMemory: 98566144; maxMemory: 4294967296
INFO 2021-03-14 19:53:02 MarkDuplicates Reading input file and constructing read end information.
INFO 2021-03-14 19:53:02 MarkDuplicates Will retain up to 15561475 data points before spilling to disk.
INFO 2021-03-14 19:53:03 MarkDuplicates Read 209837 records. 0 pairs never matched.
INFO 2021-03-14 19:53:03 MarkDuplicates After buildSortedReadEndLists freeMemory: 462475616; totalMemory: 654311424; maxMemory: 4294967296
INFO 2021-03-14 19:53:03 MarkDuplicates Will retain up to 134217728 duplicate indices before spilling to disk.
INFO 2021-03-14 19:53:04 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2021-03-14 19:53:04 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2021-03-14 19:53:04 MarkDuplicates Sorting list of duplicate records.
INFO 2021-03-14 19:53:04 MarkDuplicates After generateDuplicateIndexes freeMemory: 759669224; totalMemory: 1892679680; maxMemory: 4294967296
INFO 2021-03-14 19:53:04 MarkDuplicates Marking 33979 records as duplicates.
INFO 2021-03-14 19:53:04 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2021-03-14 19:53:05 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2021-03-14 19:53:05 MarkDuplicates Writing complete. Closing input iterator.
INFO 2021-03-14 19:53:05 MarkDuplicates Duplicate Index cleanup.
INFO 2021-03-14 19:53:05 MarkDuplicates Getting Memory Stats.
INFO 2021-03-14 19:53:06 MarkDuplicates Before output close freeMemory: 264266392; totalMemory: 374341632; maxMemory: 4294967296
INFO 2021-03-14 19:53:06 MarkDuplicates Closed outputs. Getting more Memory Stats.
INFO 2021-03-14 19:53:06 MarkDuplicates After output close freeMemory: 140924104; totalMemory: 199229440; maxMemory: 4294967296
[Sun Mar 14 19:53:06 PDT 2021] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.07 minutes.
Runtime.totalMemory()=199229440
Tool returned:
0
Using GATK jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /Applications/bioinformatics/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar MarkDuplicates I=../qc-processing/bowtie/mapped-sorted/Undetermined.sorted.bam O=../qc-processing/bowtie/deduplicated/Und.dedup.bam M=../qc-processing/bowtie/deduplicated/Und.dup_metrics.txt REMOVE_DUPLICATES=true