Example - finding patterns in vcf

1 920760 rs80259304 T C . PASS AA=T;AC=18;AN=120;DP=190;GP=1:930897;BN=131 GT:DP:CB 0/1:1:SM 0/0:4/SM...

• Find a sample:

0/0 0/1 1/1 ...

"[01]/[01]" (or "\d/\d")

\s[01]/[01]:

Example - finding patterns in vcf

1 920760 rs80259304 T C . PASS AA=T;AC=18;AN=120;DP=190;GP=1:930897;BN=131 GT:DP:CB 0/1:1:SM 0/0:4/SM...

• Find all lines containing more than one homozygous sample.

... 1/1:... ... 1/1:... ...

.*1/1.*1/1.*

.*\s1/1:.*\s1/1:.*

Exercise 1¶

• . matches any character (once)
• ? repeat previous pattern 0 or 1 times
• * repeat previous pattern 0 or more times
• + repeat previous pattern 1 or more times
• \w matches any letter or number, and the underscore
• \d matches any digit
• \D matches any non-digit
• \s matches any whitespace (spaces, tabs, ...)
• \S matches any non-whitespace
• [abc] matches a single character defined in this set {a, b, c}
• [^abc] matches a single character that is not a, b or c
• [a-z] matches any (lowercased) letter from the english alphabet
• .* matches anything

→ Notebook Day_5_Exercise_1 (~30 minutes)

Regular expressions in Python¶

Searching¶

Case insensitiveness¶

The match object¶

result.group(): Return the string matched by the expression

result.start(): Return the starting position of the match

result.end(): Return the ending position of the match

result.span(): Return both (start, end)

Zero or more...?¶

The * is greedy.

Finding all the matching patterns¶

How to find a full stop?¶

More operations¶

• \ escaping a character
• ^ beginning of the string
• $ end of string
• | boolean or

^hello$

Substitution¶

Finally, we can fix our spelling mistakes!¶

Overview¶

• Construct regular expressions

  p = re.compile()```
  

• Searching

  p.search(text)
  

• Substitution

  p.sub(replacement, text)
  

Typical code structure:

p = re.compile( ... )
m = p.search('string goes here')
if m:
    print('Match found: ', m.group())
else:
    print('No match')

Regular expressions¶

• A powerful tool to search and modify text

• There is much more to read in the docs

• Note: regex comes in different flavours. If you use it outside Python, there might be small variations in the syntax.

Exercise 2¶

• . matches any character (once)
• ? repeat previous pattern 0 or 1 times
• * repeat previous pattern 0 or more times
• + repeat previous pattern 1 or more times
• \w matches any letter or number, and the underscore
• \d matches any digit
• \D matches any non-digit
• \s matches any whitespace (spaces, tabs, ...)
• \S matches any non-whitespace
• [abc] matches a single character defined in this set {a, b, c}
• [^abc] matches a single character that is not a, b or c
• [a-z] matches any (lowercased) letter from the english alphabet
• .* matches anything
• \ escaping a character
• ^ beginning of the string
• $ end of string
• | boolean or

Read more: full documentation https://docs.python.org/3.6/library/re.html

Sum up!

Processing files - looping through the lines¶

for line in open('myfile.txt', 'r'):
    do_stuff(line)

Store values¶

iterations = 0
information = []

for line in open('myfile.txt', 'r'):
    iterations += 1
    information += do_stuff(line)

Values¶

• Base types:

  str "hello"```

  int 5```

  float 5.2```

  bool True```

• Collections:

  list ["a", "b", "c"]```

  dict {"a": "alligator", "b": "bear", "c": "cat"}```

  tuple ("this", "that")```

  set {"drama", "sci-fi"}```

Assign values

iterations = 0
score = 5.2

Modify values and compare¶

+, -, *,... # mathematical```

and, or, not # logical ```

==, != # comparisons```

<, >, <=, >= # comparisons```

in # membership

Strings¶

Raw text

• Common manipulations:

  s.strip() # remove unwanted spacing```

  s.split() # split line into columns```

  s.upper(), s.lower() # change the case```

• Regular expressions help you find and replace strings.

  p = re.compile('A.A.A') p.search(dnastring)

  - ```py
  p = re.compile('T')
  p.sub('U', dnastring)
  

Collections¶

Can contain strings, integer, booleans...

• Mutable: you can add, remove, change values

  • Lists:

  mylist.append('value')
  

  - Dicts:
  ```py
    mydict['key'] = 'value'```

  - Sets:
```py
    myset.add('value')```

Collections¶

• Test for membership:

value in myobj

• Check size:

len(myobj)

Lists¶

• Ordered!

todolist = ["work", "sleep", "eat", "work"]

todolist.sort()
todolist.reverse()
todolist[2]
todolist[-1]
todolist[2:6]

Dictionaries¶

• Keys have values

mydict = {"a": "alligator", "b": "bear", "c": "cat"}
counter = {"cats": 55, "dogs": 8}

mydict["a"]
mydict.keys()
mydict.values()

Sets¶

• Bag of values

  • No order

  • No duplicates

  • Fast membership checks

  • Logical set operations (union, difference, intersection...)

myset = {"drama", "sci-fi"}
|
myset.add("comedy")

myset.remove("drama")

Strings¶

• Works like a list of characters

  s += "more words" # add content```

  s[4] # get character at index 4```

  'e' in s # check for membership```

  len(s) # check size```

• But are immutable

  s[2] = 'i'```

  ---

  Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  TypeError: 'str' object does not support item assignment
  

Tuples¶

• A group (usually two) of values that belong together

tup = (max_lenght, sequence)

  - An ordered sequence (like lists)

  - ```py
length = tup[0]  # get content at index 0

• Immutable

Deciding what to do¶

if count > 10:
   print('big')
elif count > 5:
   print('medium')
else:
   print('small')

Deciding what to do - if statement¶

Program flow - for loops¶

information = []

for line in open('myfile.txt', 'r'):
    if is_comment(line):
       use_comment(line)
    else:
       information = read_data(line)

Program flow - while loops¶

keep_going = True
information = []
index = 0

while keep_going:
    current_line = lines[index]
    information += read_line(current_line)
    index += 1
    if check_something(current_line):
        keep_going = False

Different types of loops¶

For loop

is a control flow statement that performs operations over a known amount of steps.

While loop

is a control flow statement that allows code to be executed repeatedly based on a given Boolean condition.

Which one to use?

For loops - standard for iterations over lists and other iterable objects

While loops - more flexible and can iterate an unspecified number of times

Controlling loops¶

• break - stop the loop
• continue - go on to the next iteration

Watch out!

While loops may be infinite!

Input/Output¶

• In:

  • Read files: fh = open(filename, 'r')

    • for line in fh:
    • fh.read()
    • fh.readlines()

  • Read information from command line: sys.argv[1:]

• Out:

  • Write files: fh = open(filename, 'w')
    • fh.write(text)
  • Printing: print('my_information')

Input/Output¶

• Open files should be closed:
  • fh.close()

Formatting¶

{[field_name] [: format_spec]}

Format_spec:

filling alignment width precision type
   -       >        10    .2       f

print('|{:30}|{:^10}|{:^10.2f}|'.format(movie, votes, total/votes))

Code structure¶

• Functions
• Modules

Functions¶

• A named piece of code that performs a certain task.

• Is given a number of input arguments
  • to be used (are in scope) within the function body
• Returns a result (maybe None)

Functions - keyword arguments¶

def prettyprinter(name, value, delim=":", end=None):
    out = "The " + name + " is " + delim + " " + value
    if end:
        out += end
    return out

• used to set default values (often None)
• can be skipped in function calls
• improve readability

Files (modules)¶

• A (larger) piece of code containing functions, classes...
• Can be imported

import mymodule

• Can be run as a script

> python3 mymodule.py

Using your code¶

Any longer pieces of code that have been used and will be re-used should be saved

• Save it as a file .py

• To run it:

python3 mycode.py

• Import it:

import mycode

Documentation and comments¶

""" This is a doc-string explaining what the purpose of this function/module is."""```

This is a comment that helps understanding the code¶

• Comments will help you

• Undocumented code rarely gets used

• Try to keep your code readable: use informative variable and function names

Why programming?¶

Endless possibilities!

• reverse complement DNA
• custom filtering of VCF files
• plotting of results
• all excel stuff!

Why programming?¶

• Computers are fast
• Computers don't get bored
• Computers don't get sloppy

• Create reproducable results
• Extract large amount of information

Final advice¶

• Stop to think before you start coding
  • use pseudocode
  • use top-down programming
  • use paper and pen
  • take breaks

• You know the basics - don't be afraid to try
• You will get faster

Final advice¶

• Getting help
  • ask colleauges
  • talk about your problem (get a rubber duck)
  • search the web
  • take breaks!
  • NBIS drop-ins