%%bash
date
Thu Jun 22 14:26:20 PDT 2017
%%bash
system_profiler SPSoftwareDataType
Software:
System Software Overview:
System Version: Mac OS X 10.7.5 (11G63)
Kernel Version: Darwin 11.4.2
Boot Volume: SSD2
Boot Mode: Normal
Computer Name: greenbird (2)
User Name: Sam (Sam)
Secure Virtual Memory: Enabled
64-bit Kernel and Extensions: No
Time since boot: 38 days 3:04
%%bash
#Uses grep to exclude lines that display serial number and hardware UUID
system_profiler SPHardwareDataType | grep -v [SH][ea]
Model Name: Mac Pro
Model Identifier: MacPro1,1
Processor Name: Dual-Core Intel Xeon
Processor Speed: 3 GHz
Number of Processors: 2
Total Number of Cores: 4
L2 Cache (per Processor): 4 MB
Memory: 14 GB
Bus Speed: 1.33 GHz
Boot ROM Version: MP11.005C.B08
SMC Version (system): 1.7f10
The goal of this notebook is to copy the PacBio fastq.gz files for the oly genome sequencing project to the /owl/nightingales/O_lurida folder to be in compliance with our data management plan. Since these files do not have unique names (they're all named the same thing, but are stored in different subdirectories), they need to be renamed. Additionally, to confirm that the files were copied and renamed correctly, an md5 checkusm verification needs to take place.
%%bash
cd /Volumes/owl/nightingales/O_lurida/20170323_pacbio/
%%bash
time find . -maxdepth 2 -name "*.fastq.gz" -exec md5 {} + > md5checksums_fastq.gz.md5
real 0m0.011s user 0m0.001s sys 0m0.005s
Well, that didn't work. I think it didn't change directories. Should have caught this after executing cell #1 above, since there was no output listed. Let's see where I am...
%%bash
pwd
/Users/Sam/GitRepos/LabDocs/jupyter_nbs/sam
Yep, didn't change directories. I think it's because I used the bash shell magics for the cd command. Annoying!
cd /Volumes/owl/nightingales/O_lurida/20170323_pacbio/
/Volumes/owl/nightingales/O_lurida/20170323_pacbio
%%bash
time find . -maxdepth 2 -name "*.fastq.gz" -exec md5 {} + > md5checksums_fastq.gz.md5
real 3m56.425s user 0m27.842s sys 0m43.239s
%%bash
cat md5checksums_fastq.gz.md5
MD5 (./170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1/filtered_subreads.fastq.gz) = cc30d6e17499a36960dab140991af9b7 MD5 (./170228_PCB-CC_AL_20kb_P6v2_C01_1/filtered_subreads.fastq.gz) = c0f549d4867c2d45f0a4c5ebb65a7163 MD5 (./170228_PCB-CC_AL_20kb_P6v2_D01_1/filtered_subreads.fastq.gz) = 601c182bceec111f286b1a75f9276613 MD5 (./170228_PCB-CC_AL_20kb_P6v2_E01_1/filtered_subreads.fastq.gz) = 1d8c89d18d976d82f977f53f11657c96 MD5 (./170307_PCB-CC_AL_20kb_P6v2_C01_1/filtered_subreads.fastq.gz) = c000bb09dd10ea85028cd73ba1f8ce51 MD5 (./170307_PCB-CC_AL_20kb_P6v2_C02_1/filtered_subreads.fastq.gz) = 7f387f334adad2c11f71c4216b0d8fe2 MD5 (./170314_PCB-CC_20kb_P6v2_A01_1/filtered_subreads.fastq.gz) = c21ce30b76f57485a3cbe8712a623bd5 MD5 (./170314_PCB-CC_20kb_P6v2_A02_1/filtered_subreads.fastq.gz) = 20b0b0743fe0214bde319d2d40a40067 MD5 (./170314_PCB-CC_20kb_P6v2_A03_1/filtered_subreads.fastq.gz) = 511866d0fc60fe3e8bb6dea4ce87a64f MD5 (./170314_PCB-CC_20kb_P6v2_A04_1/filtered_subreads.fastq.gz) = 206b30a4b231541a85d4707f0b907153
%%bash
# Find all xml files and store results in array
xml_array=($(find . -maxdepth 2 -name "*.xml"))
# Print contents of array, with some formatting for easier reading.
echo "Contents of xml_array:"
printf '%s\n' "${xml_array[@]}"
echo ""
echo "-------------"
# Find all fastq.gz files and store results in array
fastq_array=($(find . -maxdepth 2 -name "*.fastq.gz"))
echo "Contents of xml_array:"
printf '%s\n' "${fastq_array[@]}"
echo ""
echo "-------------"
# Use parameter expansion to remove path from each component in fastq_array.
# Store results in new array.
fastq_nopath_array=($(echo "${fastq_array[@]##*/}"))
# Print contents of array, with some formatting for easier reading.
echo "Contents of fastq_nopath_array:"
printf '%s\n' "${fastq_nopath_array[@]}"
echo ""
echo "-------------"
# Use parameter expansion to remove path from each component in xml_array.
# Store results in new array.
xml_nopath_array=($(echo "${xml_array[@]##*/}"))
# Print contents of array, with some formatting for easier reading.
echo "Contents of xml_nopath_array:"
printf '%s\n' "${xml_nopath_array[@]}"
echo ""
echo "-------------"
# Use parameter expansion to remove the suffix (.xml) from each component in xml_nopath_array.
# Store results in new array.
xml_nosuffix_array=($(echo "${xml_nopath_array[@]%%.*}"))
# Print contents of array, with some formatting for easier reading.
echo "Contents of xml_nosuffix_array:"
printf '%s\n' "${xml_nosuffix_array[@]}"
echo ""
echo "-------------"
# Loop through each index (i.e. the number corresponding to each element in the array).
# Using the paths to each fastq.gz stored in the fastq_array, copy the fastq.gz file to the O_lurida nigthtingales folder &
# use array elements to provide new name to copied file.
# List the newly copied/named file to verify it got copied/renamed.
# Create md5 checksums for each newly copied/renamed file and append to checksums.md5 file.
# Use grep to verify info was written to checksums.md5 file
for item in "${!fastq_array[@]}"; do
cp "${fastq_array[$item]}" /Volumes/owl/nightingales/O_lurida/"${xml_nosuffix_array[$item]}_${fastq_nopath_array[$item]}"
ls /Volumes/owl/nightingales/O_lurida/"${xml_nosuffix_array[$item]}_${fastq_nopath_array[$item]}"
md5 /Volumes/owl/nightingales/O_lurida/"${xml_nosuffix_array[$item]}_${fastq_nopath_array[$item]}" >> \
/Volumes/owl/nightingales/O_lurida/checksums.md5
grep "${xml_nosuffix_array[$item]}_${fastq_nopath_array[$item]}" /Volumes/owl/nightingales/O_lurida/checksums.md5
done
Contents of xml_array: ./170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1/m170211_224036_42134_c101073082550000001823236402101737_s1_X0.metadata.xml ./170228_PCB-CC_AL_20kb_P6v2_C01_1/m170301_100013_42134_c101174162550000001823269408211761_s1_p0.metadata.xml ./170228_PCB-CC_AL_20kb_P6v2_D01_1/m170301_162825_42134_c101174162550000001823269408211762_s1_p0.metadata.xml ./170228_PCB-CC_AL_20kb_P6v2_E01_1/m170301_225711_42134_c101174162550000001823269408211763_s1_p0.metadata.xml ./170307_PCB-CC_AL_20kb_P6v2_C01_1/m170308_163922_42134_c101174252550000001823269408211742_s1_p0.metadata.xml ./170307_PCB-CC_AL_20kb_P6v2_C02_1/m170308_230815_42134_c101174252550000001823269408211743_s1_p0.metadata.xml ./170314_PCB-CC_20kb_P6v2_A01_1/m170315_001112_42134_c101169372550000001823273008151717_s1_p0.metadata.xml ./170314_PCB-CC_20kb_P6v2_A02_1/m170315_063041_42134_c101169382550000001823273008151700_s1_p0.metadata.xml ./170314_PCB-CC_20kb_P6v2_A03_1/m170315_124938_42134_c101169382550000001823273008151701_s1_p0.metadata.xml ./170314_PCB-CC_20kb_P6v2_A04_1/m170315_190851_42134_c101169382550000001823273008151702_s1_p0.metadata.xml ------------- Contents of xml_array: ./170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1/filtered_subreads.fastq.gz ./170228_PCB-CC_AL_20kb_P6v2_C01_1/filtered_subreads.fastq.gz ./170228_PCB-CC_AL_20kb_P6v2_D01_1/filtered_subreads.fastq.gz ./170228_PCB-CC_AL_20kb_P6v2_E01_1/filtered_subreads.fastq.gz ./170307_PCB-CC_AL_20kb_P6v2_C01_1/filtered_subreads.fastq.gz ./170307_PCB-CC_AL_20kb_P6v2_C02_1/filtered_subreads.fastq.gz ./170314_PCB-CC_20kb_P6v2_A01_1/filtered_subreads.fastq.gz ./170314_PCB-CC_20kb_P6v2_A02_1/filtered_subreads.fastq.gz ./170314_PCB-CC_20kb_P6v2_A03_1/filtered_subreads.fastq.gz ./170314_PCB-CC_20kb_P6v2_A04_1/filtered_subreads.fastq.gz ------------- Contents of fastq_nopath_array: filtered_subreads.fastq.gz filtered_subreads.fastq.gz filtered_subreads.fastq.gz filtered_subreads.fastq.gz filtered_subreads.fastq.gz filtered_subreads.fastq.gz filtered_subreads.fastq.gz filtered_subreads.fastq.gz filtered_subreads.fastq.gz filtered_subreads.fastq.gz ------------- Contents of xml_nopath_array: m170211_224036_42134_c101073082550000001823236402101737_s1_X0.metadata.xml m170301_100013_42134_c101174162550000001823269408211761_s1_p0.metadata.xml m170301_162825_42134_c101174162550000001823269408211762_s1_p0.metadata.xml m170301_225711_42134_c101174162550000001823269408211763_s1_p0.metadata.xml m170308_163922_42134_c101174252550000001823269408211742_s1_p0.metadata.xml m170308_230815_42134_c101174252550000001823269408211743_s1_p0.metadata.xml m170315_001112_42134_c101169372550000001823273008151717_s1_p0.metadata.xml m170315_063041_42134_c101169382550000001823273008151700_s1_p0.metadata.xml m170315_124938_42134_c101169382550000001823273008151701_s1_p0.metadata.xml m170315_190851_42134_c101169382550000001823273008151702_s1_p0.metadata.xml ------------- Contents of xml_nosuffix_array: m170211_224036_42134_c101073082550000001823236402101737_s1_X0 m170301_100013_42134_c101174162550000001823269408211761_s1_p0 m170301_162825_42134_c101174162550000001823269408211762_s1_p0 m170301_225711_42134_c101174162550000001823269408211763_s1_p0 m170308_163922_42134_c101174252550000001823269408211742_s1_p0 m170308_230815_42134_c101174252550000001823269408211743_s1_p0 m170315_001112_42134_c101169372550000001823273008151717_s1_p0 m170315_063041_42134_c101169382550000001823273008151700_s1_p0 m170315_124938_42134_c101169382550000001823273008151701_s1_p0 m170315_190851_42134_c101169382550000001823273008151702_s1_p0 ------------- /Volumes/owl/nightingales/O_lurida/m170211_224036_42134_c101073082550000001823236402101737_s1_X0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170211_224036_42134_c101073082550000001823236402101737_s1_X0_filtered_subreads.fastq.gz) = cc30d6e17499a36960dab140991af9b7 /Volumes/owl/nightingales/O_lurida/m170301_100013_42134_c101174162550000001823269408211761_s1_p0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170301_100013_42134_c101174162550000001823269408211761_s1_p0_filtered_subreads.fastq.gz) = c0f549d4867c2d45f0a4c5ebb65a7163 /Volumes/owl/nightingales/O_lurida/m170301_162825_42134_c101174162550000001823269408211762_s1_p0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170301_162825_42134_c101174162550000001823269408211762_s1_p0_filtered_subreads.fastq.gz) = 601c182bceec111f286b1a75f9276613 /Volumes/owl/nightingales/O_lurida/m170301_225711_42134_c101174162550000001823269408211763_s1_p0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170301_225711_42134_c101174162550000001823269408211763_s1_p0_filtered_subreads.fastq.gz) = 1d8c89d18d976d82f977f53f11657c96 /Volumes/owl/nightingales/O_lurida/m170308_163922_42134_c101174252550000001823269408211742_s1_p0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170308_163922_42134_c101174252550000001823269408211742_s1_p0_filtered_subreads.fastq.gz) = c000bb09dd10ea85028cd73ba1f8ce51 /Volumes/owl/nightingales/O_lurida/m170308_230815_42134_c101174252550000001823269408211743_s1_p0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170308_230815_42134_c101174252550000001823269408211743_s1_p0_filtered_subreads.fastq.gz) = 7f387f334adad2c11f71c4216b0d8fe2 /Volumes/owl/nightingales/O_lurida/m170315_001112_42134_c101169372550000001823273008151717_s1_p0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170315_001112_42134_c101169372550000001823273008151717_s1_p0_filtered_subreads.fastq.gz) = c21ce30b76f57485a3cbe8712a623bd5 /Volumes/owl/nightingales/O_lurida/m170315_063041_42134_c101169382550000001823273008151700_s1_p0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170315_063041_42134_c101169382550000001823273008151700_s1_p0_filtered_subreads.fastq.gz) = 20b0b0743fe0214bde319d2d40a40067 /Volumes/owl/nightingales/O_lurida/m170315_124938_42134_c101169382550000001823273008151701_s1_p0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170315_124938_42134_c101169382550000001823273008151701_s1_p0_filtered_subreads.fastq.gz) = 511866d0fc60fe3e8bb6dea4ce87a64f /Volumes/owl/nightingales/O_lurida/m170315_190851_42134_c101169382550000001823273008151702_s1_p0_filtered_subreads.fastq.gz MD5 (/Volumes/owl/nightingales/O_lurida/m170315_190851_42134_c101169382550000001823273008151702_s1_p0_filtered_subreads.fastq.gz) = 206b30a4b231541a85d4707f0b907153
Wow! I'm impressed with myself! It nearly all came out as intended!! Forgot to change the second echo statement to read "Contents of fastq_array:". Other than that, it all worked. Feels good!
Quick test of using grep and awk to isolate just the checksum values...
grep "filtered_subreads.fastq.gz" /Volumes/owl/nightingales/O_lurida/checksums.md5 | awk '{print "$4"}'
File "<ipython-input-8-0e82f528bf91>", line 1 grep "filtered_subreads.fastq.gz" /Volumes/owl/nightingales/O_lurida/checksums.md5 | awk '{print "$4"}' ^ SyntaxError: invalid syntax
%%bash
grep "filtered_subreads.fastq.gz" /Volumes/owl/nightingales/O_lurida/checksums.md5 | awk '{print "$4"}'
$4 $4 $4 $4 $4 $4 $4 $4 $4 $4
%%bash
grep "filtered_subreads.fastq.gz" /Volumes/owl/nightingales/O_lurida/checksums.md5 | awk '{print $4}'
cc30d6e17499a36960dab140991af9b7 c0f549d4867c2d45f0a4c5ebb65a7163 601c182bceec111f286b1a75f9276613 1d8c89d18d976d82f977f53f11657c96 c000bb09dd10ea85028cd73ba1f8ce51 7f387f334adad2c11f71c4216b0d8fe2 c21ce30b76f57485a3cbe8712a623bd5 20b0b0743fe0214bde319d2d40a40067 511866d0fc60fe3e8bb6dea4ce87a64f 206b30a4b231541a85d4707f0b907153
Well, I just looked through some previously used code and realized I didn't need to perform the above grep/awk test.
%%bash
original_md5=($(awk '/filtered_subreads.fastq.gz/{print $4}' /Volumes/owl/nightingales/O_lurida/20170323_pacbio/md5checksums_fastq.gz.md5))
current_md5=($(awk '/filtered_subreads.fastq.gz/{print $4}' /Volumes/owl/nightingales/O_lurida/checksums.md5
for ((i=0;i<=$count;++i))
do
printf "%s\n" "${original_md5[$i]}"
printf "%s\n\n" "${current_md5[$i]}"
done
bash: line 2: unexpected EOF while looking for matching `)' bash: line 8: syntax error: unexpected end of file
%%bash
original_md5=($(awk '/filtered_subreads.fastq.gz/{print $4}' /Volumes/owl/nightingales/O_lurida/20170323_pacbio/md5checksums_fastq.gz.md5))
current_md5=($(awk '/filtered_subreads.fastq.gz/{print $4}' /Volumes/owl/nightingales/O_lurida/checksums.md5))
count=$(( ${#original_md5[@]} - 1 ))
for ((i=0;i<=$count;++i))
do
printf "%s\n" "${original_md5[$i]}"
printf "%s\n\n" "${current_md5[$i]}"
done
cc30d6e17499a36960dab140991af9b7 cc30d6e17499a36960dab140991af9b7 c0f549d4867c2d45f0a4c5ebb65a7163 c0f549d4867c2d45f0a4c5ebb65a7163 601c182bceec111f286b1a75f9276613 601c182bceec111f286b1a75f9276613 1d8c89d18d976d82f977f53f11657c96 1d8c89d18d976d82f977f53f11657c96 c000bb09dd10ea85028cd73ba1f8ce51 c000bb09dd10ea85028cd73ba1f8ce51 7f387f334adad2c11f71c4216b0d8fe2 7f387f334adad2c11f71c4216b0d8fe2 c21ce30b76f57485a3cbe8712a623bd5 c21ce30b76f57485a3cbe8712a623bd5 20b0b0743fe0214bde319d2d40a40067 20b0b0743fe0214bde319d2d40a40067 511866d0fc60fe3e8bb6dea4ce87a64f 511866d0fc60fe3e8bb6dea4ce87a64f 206b30a4b231541a85d4707f0b907153 206b30a4b231541a85d4707f0b907153
Great! Visual inspection indicates the MD5 checkums have not changed during the copying/renaming process!
original_md5=() - This is an empty array called "original_md5".
$() - This is an empty command substitution. The stdout of commands within the parentheses are stored.
awk '/filtered_subreads.fastq.gz/{print $4}' md5_file - Awk looks for any lines from the input file (md5_file) with "filtered_subreads.fastq.gz" in them. If a line contains "filtered_subreads.fastq.gz", awk prints the fourth field (i.e. the fourth column).
Summary - The output from each result printed by awk is saved in an auto-incrementing fashion in the array called "original_md5".
count=$(()) - A variable called "count". This is a combination of empty command substitution and bash arithmeetic. Double parentheses are required for bash arithmetic.
${#current_md5[@]} - 1 - This prints the number of indices (#) in the array called "original_md5" and subtracts 1 from that number. Subtraction of one is necessary because bash is a zero-based language (e.g. the array starts at index 0).
Summary - The length of the array minus one is saved to the variable called "count".
((i=0;i<=$count;++i)) - Sets variable "i" to 0. Then, the loop evaluates whether or not the value of "i" is less than/equal to the value in the variable "count". If that condition is met, the loop increases the value stored in "i" by 1 and continues through the loop.
printf "%s\n" "${original_md5[$i]}" - Prints the value at the array index designated by the value currently stored in "i" (the printing is specified by the "%s", which means string). This is followed by printing a new line (\n).
Summary - This prints the value at each position within the array and uses printf to improve legibility of output.
cd ..
/Volumes/owl/nightingales/O_lurida
%%bash
time tar -zcf 20170323_pacbio.tar.gz /Volumes/owl/nightingales/O_lurida/20170323_pacbio/
tar: Removing leading '/' from member names tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/post_control_regions.fofn: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/filtered_subread_summary.csv: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/filtered_subreads.fasta: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/post_control_regions.chunk003of003: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/control_results_by_movie.csv: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/data.items.json: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/post_control_regions.chunk001of003: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/filtered_regions.fofn: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/post_control_regions.chunk002of003: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/filtered_regions: Couldn't visit directory: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/workflow: Couldn't visit directory: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/log: Couldn't visit directory: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected tar: Write errorWrite errorWrite errorFailed to clean up compressor tar: Error exit delayed from previous errors. real 360m51.774s user 201m23.761s sys 17m51.178s
Well... That's a bummer. I think I actually need to run this again, as I'm not entirely sure how to tell if the tarball completed properly.
pwd
Traceback (most recent call last):
File "/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py", line 970, in get_records
return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
File "/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py", line 233, in wrapped
return f(*args, **kwargs)
File "/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py", line 267, in _fixed_getinnerframes
records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 1049, in getinnerframes
framelist.append((tb.tb_frame,) + getframeinfo(tb, context))
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 1009, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 454, in getsourcefile
if hasattr(getmodule(object, filename), '__loader__'):
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 483, in getmodule
file = getabsfile(object, _filename)
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 467, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File "/Users/Sam/Applications/anaconda/lib/python2.7/posixpath.py", line 364, in abspath
cwd = os.getcwd()
OSError: [Errno 2] No such file or directory
ERROR: Internal Python error in the inspect module. Below is the traceback from this internal error. Unfortunately, your original traceback can not be constructed.
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_code(self, code_obj, result) 3081 if result is not None: 3082 result.error_in_exec = sys.exc_info()[1] -> 3083 self.showtraceback() 3084 else: 3085 outflag = 0 /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in showtraceback(self, exc_tuple, filename, tb_offset, exception_only) 1878 except Exception: 1879 stb = self.InteractiveTB.structured_traceback(etype, -> 1880 value, tb, tb_offset=tb_offset) 1881 1882 self._showtraceback(etype, value, stb) /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context) 1240 self.tb = tb 1241 return FormattedTB.structured_traceback( -> 1242 self, etype, value, tb, tb_offset, number_of_lines_of_context) 1243 1244 /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context) 1148 # Verbose modes need a full traceback 1149 return VerboseTB.structured_traceback( -> 1150 self, etype, value, tb, tb_offset, number_of_lines_of_context 1151 ) 1152 else: /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context) 1000 1001 formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context, -> 1002 tb_offset) 1003 1004 colors = self.Colors # just a shorthand + quicker name lookup /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in format_exception_as_a_whole(self, etype, evalue, etb, number_of_lines_of_context, tb_offset) 949 records = self.get_records(etb, number_of_lines_of_context, tb_offset) 950 --> 951 frames = self.format_records(records) 952 if records is None: 953 return "" /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in format_records(self, records) 722 723 abspath = os.path.abspath --> 724 for frame, file, lnum, func, lines, index in records: 725 #print '*** record:',file,lnum,func,lines,index # dbg 726 if not file: TypeError: 'NoneType' object is not iterable
%%bash
pwd
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory pwd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
cd /Volumes/
Traceback (most recent call last):
File "/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py", line 970, in get_records
return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
File "/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py", line 233, in wrapped
return f(*args, **kwargs)
File "/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py", line 267, in _fixed_getinnerframes
records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 1049, in getinnerframes
framelist.append((tb.tb_frame,) + getframeinfo(tb, context))
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 1009, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 454, in getsourcefile
if hasattr(getmodule(object, filename), '__loader__'):
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 483, in getmodule
file = getabsfile(object, _filename)
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 467, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File "/Users/Sam/Applications/anaconda/lib/python2.7/posixpath.py", line 364, in abspath
cwd = os.getcwd()
OSError: [Errno 2] No such file or directory
ERROR: Internal Python error in the inspect module. Below is the traceback from this internal error. Unfortunately, your original traceback can not be constructed.
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_code(self, code_obj, result) 3081 if result is not None: 3082 result.error_in_exec = sys.exc_info()[1] -> 3083 self.showtraceback() 3084 else: 3085 outflag = 0 /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in showtraceback(self, exc_tuple, filename, tb_offset, exception_only) 1878 except Exception: 1879 stb = self.InteractiveTB.structured_traceback(etype, -> 1880 value, tb, tb_offset=tb_offset) 1881 1882 self._showtraceback(etype, value, stb) /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context) 1240 self.tb = tb 1241 return FormattedTB.structured_traceback( -> 1242 self, etype, value, tb, tb_offset, number_of_lines_of_context) 1243 1244 /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context) 1148 # Verbose modes need a full traceback 1149 return VerboseTB.structured_traceback( -> 1150 self, etype, value, tb, tb_offset, number_of_lines_of_context 1151 ) 1152 else: /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context) 1000 1001 formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context, -> 1002 tb_offset) 1003 1004 colors = self.Colors # just a shorthand + quicker name lookup /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in format_exception_as_a_whole(self, etype, evalue, etb, number_of_lines_of_context, tb_offset) 949 records = self.get_records(etb, number_of_lines_of_context, tb_offset) 950 --> 951 frames = self.format_records(records) 952 if records is None: 953 return "" /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in format_records(self, records) 722 723 abspath = os.path.abspath --> 724 for frame, file, lnum, func, lines, index in records: 725 #print '*** record:',file,lnum,func,lines,index # dbg 726 if not file: TypeError: 'NoneType' object is not iterable
cd ~/Downloads/
Traceback (most recent call last):
File "/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py", line 970, in get_records
return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
File "/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py", line 233, in wrapped
return f(*args, **kwargs)
File "/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py", line 267, in _fixed_getinnerframes
records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 1049, in getinnerframes
framelist.append((tb.tb_frame,) + getframeinfo(tb, context))
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 1009, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 454, in getsourcefile
if hasattr(getmodule(object, filename), '__loader__'):
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 483, in getmodule
file = getabsfile(object, _filename)
File "/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py", line 467, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File "/Users/Sam/Applications/anaconda/lib/python2.7/posixpath.py", line 364, in abspath
cwd = os.getcwd()
OSError: [Errno 2] No such file or directory
ERROR: Internal Python error in the inspect module. Below is the traceback from this internal error. Unfortunately, your original traceback can not be constructed.
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_code(self, code_obj, result) 3081 if result is not None: 3082 result.error_in_exec = sys.exc_info()[1] -> 3083 self.showtraceback() 3084 else: 3085 outflag = 0 /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in showtraceback(self, exc_tuple, filename, tb_offset, exception_only) 1878 except Exception: 1879 stb = self.InteractiveTB.structured_traceback(etype, -> 1880 value, tb, tb_offset=tb_offset) 1881 1882 self._showtraceback(etype, value, stb) /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context) 1240 self.tb = tb 1241 return FormattedTB.structured_traceback( -> 1242 self, etype, value, tb, tb_offset, number_of_lines_of_context) 1243 1244 /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context) 1148 # Verbose modes need a full traceback 1149 return VerboseTB.structured_traceback( -> 1150 self, etype, value, tb, tb_offset, number_of_lines_of_context 1151 ) 1152 else: /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context) 1000 1001 formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context, -> 1002 tb_offset) 1003 1004 colors = self.Colors # just a shorthand + quicker name lookup /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in format_exception_as_a_whole(self, etype, evalue, etb, number_of_lines_of_context, tb_offset) 949 records = self.get_records(etb, number_of_lines_of_context, tb_offset) 950 --> 951 frames = self.format_records(records) 952 if records is None: 953 return "" /Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in format_records(self, records) 722 723 abspath = os.path.abspath --> 724 for frame, file, lnum, func, lines, index in records: 725 #print '*** record:',file,lnum,func,lines,index # dbg 726 if not file: TypeError: 'NoneType' object is not iterable
Ok... It seems like the notebook is just totally jacked up. Guess I'll restart it. Ugh.
%%bash
date
Thu Jun 29 06:20:30 PDT 2017
cd /Volumes/owl/nightingales/O_lurida/
/Volumes/owl/nightingales/O_lurida
ls -lr 20170323_pacbio*
-rw-rw-rw- 1 Sam staff 116176977920 Jun 28 17:31 20170323_pacbio.tar.gz 20170323_pacbio: total 48 -rw-rw-rw-@ 1 Sam staff 401 Jun 28 11:26 readme.txt -rw-rw-rw- 1 Sam staff 1022 Jun 28 08:53 md5checksums_fastq.gz.md5 -rw-rw-rw-@ 1 Sam staff 7112 Jun 28 08:20 all_md5.txt drwxrwxrwx 1 Sam staff 264 Apr 5 10:01 170314_PCB-CC_20kb_P6v2_A04_1/ drwxrwxrwx 1 Sam staff 264 Apr 3 07:27 170314_PCB-CC_20kb_P6v2_A03_1/ drwxrwxrwx 1 Sam staff 264 Apr 3 08:07 170314_PCB-CC_20kb_P6v2_A02_1/ drwxrwxrwx 1 Sam staff 264 Mar 23 18:47 170314_PCB-CC_20kb_P6v2_A01_1/ drwxrwxrwx 1 Sam staff 264 Apr 5 10:04 170307_PCB-CC_AL_20kb_P6v2_C02_1/ drwxrwxrwx 1 Sam staff 264 Mar 23 17:32 170307_PCB-CC_AL_20kb_P6v2_C01_1/ drwxrwxrwx 1 Sam staff 264 Mar 23 17:00 170228_PCB-CC_AL_20kb_P6v2_E01_1/ drwxrwxrwx 1 Sam staff 264 Mar 23 16:32 170228_PCB-CC_AL_20kb_P6v2_D01_1/ drwxrwxrwx 1 Sam staff 264 Apr 3 08:01 170228_PCB-CC_AL_20kb_P6v2_C01_1/ drwxrwxrwx 1 Sam staff 264 Apr 3 08:01 170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1/
ls -lh 20170323*
-rw-rw-rw- 1 Sam staff 108G Jun 28 17:31 20170323_pacbio.tar.gz 20170323_pacbio: total 48 drwxrwxrwx 1 Sam staff 264B Apr 3 08:01 170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1/ drwxrwxrwx 1 Sam staff 264B Apr 3 08:01 170228_PCB-CC_AL_20kb_P6v2_C01_1/ drwxrwxrwx 1 Sam staff 264B Mar 23 16:32 170228_PCB-CC_AL_20kb_P6v2_D01_1/ drwxrwxrwx 1 Sam staff 264B Mar 23 17:00 170228_PCB-CC_AL_20kb_P6v2_E01_1/ drwxrwxrwx 1 Sam staff 264B Mar 23 17:32 170307_PCB-CC_AL_20kb_P6v2_C01_1/ drwxrwxrwx 1 Sam staff 264B Apr 5 10:04 170307_PCB-CC_AL_20kb_P6v2_C02_1/ drwxrwxrwx 1 Sam staff 264B Mar 23 18:47 170314_PCB-CC_20kb_P6v2_A01_1/ drwxrwxrwx 1 Sam staff 264B Apr 3 08:07 170314_PCB-CC_20kb_P6v2_A02_1/ drwxrwxrwx 1 Sam staff 264B Apr 3 07:27 170314_PCB-CC_20kb_P6v2_A03_1/ drwxrwxrwx 1 Sam staff 264B Apr 5 10:01 170314_PCB-CC_20kb_P6v2_A04_1/ -rw-rw-rw-@ 1 Sam staff 6.9K Jun 28 08:20 all_md5.txt -rw-rw-rw- 1 Sam staff 1.0K Jun 28 08:53 md5checksums_fastq.gz.md5 -rw-rw-rw-@ 1 Sam staff 401B Jun 28 11:26 readme.txt
Well, I still have no way of knowing whether or not that tarball is legit. Guess I'll delete it and re-run the compression command. Sigh...
%%bash
time rm 20170323_pacbio.tar.gz
real 0m1.360s user 0m0.001s sys 0m0.004s
ls -lh 20170323*
total 48 drwxrwxrwx 1 Sam staff 264B Apr 3 08:01 170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1/ drwxrwxrwx 1 Sam staff 264B Apr 3 08:01 170228_PCB-CC_AL_20kb_P6v2_C01_1/ drwxrwxrwx 1 Sam staff 264B Mar 23 16:32 170228_PCB-CC_AL_20kb_P6v2_D01_1/ drwxrwxrwx 1 Sam staff 264B Mar 23 17:00 170228_PCB-CC_AL_20kb_P6v2_E01_1/ drwxrwxrwx 1 Sam staff 264B Mar 23 17:32 170307_PCB-CC_AL_20kb_P6v2_C01_1/ drwxrwxrwx 1 Sam staff 264B Apr 5 10:04 170307_PCB-CC_AL_20kb_P6v2_C02_1/ drwxrwxrwx 1 Sam staff 264B Mar 23 18:47 170314_PCB-CC_20kb_P6v2_A01_1/ drwxrwxrwx 1 Sam staff 264B Apr 3 08:07 170314_PCB-CC_20kb_P6v2_A02_1/ drwxrwxrwx 1 Sam staff 264B Apr 3 07:27 170314_PCB-CC_20kb_P6v2_A03_1/ drwxrwxrwx 1 Sam staff 264B Apr 5 10:01 170314_PCB-CC_20kb_P6v2_A04_1/ -rw-rw-rw-@ 1 Sam staff 6.9K Jun 28 08:20 all_md5.txt -rw-rw-rw- 1 Sam staff 1.0K Jun 28 08:53 md5checksums_fastq.gz.md5 -rw-rw-rw-@ 1 Sam staff 401B Jun 28 11:26 readme.txt
%%bash
time tar -zcf 20170323_pacbio.tar.gz /Volumes/owl/nightingales/O_lurida/20170323_pacbio/
tar: Removing leading '/' from member names tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.fofn: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_subread_summary.csv: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_subreads.fasta: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk003of003: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/control_results_by_movie.csv: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/data.items.json: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk001of003: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_regions.fofn: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk002of003: Cannot stat: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected tar: Write errorWrite errorWrite errorFailed to clean up compressor tar: Error exit delayed from previous errors. real 213m55.180s user 120m32.603s sys 10m36.751s
Dude! Lost the connection to Owl. AGAIN! What is going on??!! I vaguely remember Sean mentioning some data transfer issues involving Owl and Hyak (mox). I wonder if this is related. Will discuss with Steven to see if he has experienced any weird network issues this week.
Will have to consider another approach to generating the tarball (likely ssh in as admin and run tar directly on Owl - not over the network.
In the meantime, I'm going to delete this incomplete tarball.
I'll remount Owl outside of this notebook and then proceed.
$$bash
date
File "<ipython-input-1-f44d0c6507fa>", line 1 $$bash ^ SyntaxError: invalid syntax
%%bash
date
Thu Jun 29 14:37:21 PDT 2017
ls -lh /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz
-rw-rw-rw- 1 Sam staff 68G Jun 29 10:01 /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz
%%bash
time rm /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz
real 0m1.684s user 0m0.001s sys 0m0.004s
ls -lh /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz
ls: /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz: No such file or directory
Well, I'm just going to try to run this again. Third time's the charm?
cd /Volumes/web/nightingales/O_lurida/
/Volumes/web/nightingales/O_lurida
%%bash
time tar -zcf 20170323_pacbio.tar.gz /Volumes/owl/nightingales/O_lurida/20170323_pacbio/
tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio: Cannot stat: No such file or directory tar: Error exit delayed from previous errors. real 0m0.074s user 0m0.001s sys 0m0.008s
%%bash
time tar -zcf 20170323_pacbio.tar.gz /Volumes/web/nightingales/O_lurida/20170323_pacbio/
tar: Removing leading '/' from member names tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.fofn: Cannot stat: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_subread_summary.csv: Cannot stat: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_subreads.fasta: Cannot stat: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk003of003: Cannot stat: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/control_results_by_movie.csv: Cannot stat: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/data.items.json: Cannot stat: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk001of003: Cannot stat: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_regions.fofn: Cannot stat: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk002of003: Cannot stat: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected tar: Write errorWrite errorWrite errorFailed to clean up compressor tar: Error exit delayed from previous errors. real 216m49.377s user 125m13.016s sys 9m52.843s
OK, this is annoying. The problem is caused by the connection to Owl getting lost/unmounted (see below). This is an issue that Sean had previously experienced as well. He had some conversations with the people who run Hyak (mox), they did some quick tests and experienced a similar problem with moving data to/from Owl. According to Sean, either the Hyak (mox) people and/or UW IT indicated that there's a wiring issue in FTR that's causing this problem and it's not a problem related to Owl.
ls Volumes/web/nightingales/O_lurida
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory ls: Volumes/web/nightingales/O_lurida: No such file or directory
I've reconnected to Owl (via Finder) in order to remove the incomplete tarball.
%%bash
date
Mon Jul 3 05:57:38 PDT 2017
ls /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz
ls: /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz: No such file or directory