%load_ext watermark
%watermark -d -u -v
last updated: 2018-06-09 CPython 3.6.4 IPython 6.2.1
More information about the watermark
magic command extension.
The C3 class resolution algorithm for multiple class inheritance
Assignment operators and lists - simple-add vs. add-AND operators
Python reuses objects for small integers - always use "==" for equality, "is" for identity
Shallow vs. deep copies if list contains other structures and objects
Don't use mutable objects as default arguments for functions!
Python's LEGB scope resolution and the keywords global
and nonlocal
The consequences of modifying a list when looping through it
Function annotations - What are those ->
's in my Python code?
Only the first clause of generators is evaluated immediately
Interning of compile-time constants vs. run-time expressions
If we are dealing with multiple inheritance, according to the newer C3 class resolution algorithm, the following applies:
Assuming that child class C inherits from two parent classes A and B, "class A should be checked before class B".
If you want to learn more, please read the original blog post by Guido van Rossum.
(Original source: http://gistroll.com/rolls/21/horizontal_assessments/new)
class A(object):
def foo(self):
print("class A")
class B(object):
def foo(self):
print("class B")
class C(A, B):
pass
C().foo()
class A
So what actually happened above was that class C
looked in the scope of the parent class A
for the method .foo()
first (and found it)!
I received an email containing a suggestion which uses a more nested example to illustrate Guido van Rossum's point a little bit better:
class A(object):
def foo(self):
print("class A")
class B(A):
pass
class C(A):
def foo(self):
print("class C")
class D(B,C):
pass
D().foo()
class C
Here, class D
searches in B
first, which in turn inherits from A
(note that class C
also inherits from A
, but has its own .foo()
method) so that we come up with the search order: D, B, C, A
.
Python list
s are mutable objects as we all know. So, if we are using the +=
operator on list
s, we extend the list
by directly modifying the object.
However, if we use the assignment via my_list = my_list + ...
, we create a new list object, which can be demonstrated by the following code:
a_list = []
print('ID:', id(a_list))
a_list += [1]
print('ID (+=):', id(a_list))
a_list = a_list + [2]
print('ID (list = list + ...):', id(a_list))
ID: 4486856904 ID (+=): 4486856904 ID (list = list + ...): 4486959368
Just for reference, the .append()
and .extends()
methods are modifying the list
object in place, just as expected.
a_list = []
print(a_list, '\nID (initial):',id(a_list), '\n')
a_list.append(1)
print(a_list, '\nID (append):',id(a_list), '\n')
a_list.extend([2])
print(a_list, '\nID (extend):',id(a_list))
[] ID (initial): 4486857224 [1] ID (append): 4486857224 [1, 2] ID (extend): 4486857224
True
and False
in the datetime module¶"It often comes as a big surprise for programmers to find (sometimes by way of a hard-to-reproduce bug) that, unlike any other time value, midnight (i.e. datetime.time(0,0,0)
) is False. A long discussion on the python-ideas mailing list shows that, while surprising, that behavior is desirable — at least in some quarters."
Please note that Python version <= 3.4.5 evaluated the first statement bool(datetime.time(0,0,0))
as False
, which was regarded counter-intuitive, since "12am" refers to "midnight."
(Original source: http://lwn.net/SubscriberLink/590299/bf73fe823974acea/)
from platform import python_version
import datetime
print("Current python version: ", python_version())
print('"datetime.time(0,0,0)" (Midnight) ->', bool(datetime.time(0,0,0))) # Python version <= 3.4.5 evaluates this statement to False
print('"datetime.time(1,0,0)" (1 am) ->', bool(datetime.time(1,0,0)))
Current python version: 3.6.4 "datetime.time(0,0,0)" (Midnight) -> True "datetime.time(1,0,0)" (1 am) -> True
This oddity occurs, because Python keeps an array of small integer objects (i.e., integers between -5 and 256, see the doc).
a = 1
b = 1
print('a is b', bool(a is b))
True
c = 999
d = 999
print('c is d', bool(c is d))
a is b True c is d False
(I received a comment that this is in fact a CPython artifact and must not necessarily be true* in all implementations of Python!*)
So the take home message is: always use "==" for equality, "is" for identity!
Here is a nice article explaining it by comparing "boxes" (C language) with "name tags" (Python).
This example demonstrates that this applies indeed for integers in the range in -5 to 256:
print('256 is 257-1', 256 is 257-1)
print('257 is 258-1', 257 is 258 - 1)
print('-5 is -6+1', -5 is -6+1)
print('-7 is -6-1', -7 is -6-1)
256 is 257-1 True 257 is 258-1 False -5 is -6+1 True -7 is -6-1 False
==
) vs. identity (is
):¶a = 'hello world!'
b = 'hello world!'
print('a is b,', a is b)
print('a == b,', a == b)
a is b, False a == b, True
We would think that identity would always imply equality, but this is not always true, as we can see in the next example:
a = float('nan')
print('a is a,', a is a)
print('a == a,', a == a)
a is a, True a == a, False
Shallow copy:
If we use the assignment operator to assign one list to another list, we just create a new name reference to the original list. If we want to create a new list object, we have to make a copy of the original list. This can be done via a_list[:]
or a_list.copy()
.
list1 = [1,2]
list2 = list1 # reference
list3 = list1[:] # shallow copy
list4 = list1.copy() # shallow copy
print('IDs:\nlist1: {}\nlist2: {}\nlist3: {}\nlist4: {}\n'
.format(id(list1), id(list2), id(list3), id(list4)))
list2[0] = 3
print('list1:', list1)
list3[0] = 4
list4[1] = 4
print('\nlist1:', list1)
print('list2:', list2)
print('list3:', list3)
print('list4:', list4)
IDs: list1: 4486860424 list2: 4486860424 list3: 4486818632 list4: 4486818568 list1: [3, 2] list1: [3, 2] list2: [3, 2] list3: [4, 2] list4: [1, 4]
Deep copy
As we have seen above, a shallow copy works fine if we want to create a new list with contents of the original list which we want to modify independently.
However, if we are dealing with compound objects (e.g., lists that contain other lists, read here for more information) it becomes a little trickier.
In the case of compound objects, a shallow copy would create a new compound object, but it would just insert the references to the contained objects into the new compound object. In contrast, a deep copy would go "deeper" and create also new objects
for the objects found in the original compound object.
If you follow the code, the concept should become more clear:
from copy import deepcopy
list1 = [[1],[2]]
list2 = list1.copy() # shallow copy
list3 = deepcopy(list1) # deep copy
print('IDs:\nlist1: {}\nlist2: {}\nlist3: {}\n'
.format(id(list1), id(list2), id(list3)))
list2[0][0] = 3
print('list1:', list1)
list3[0][0] = 5
print('\nlist1:', list1)
print('list2:', list2)
print('list3:', list3)
IDs: list1: 4486818824 list2: 4486886024 list3: 4486888200 list1: [[3], [2]] list1: [[3], [2]] list2: [[3], [2]] list3: [[5], [2]]
True
values from logical and
s and or
s¶Logical or
:
a or b == a if a else b
or
expressions are True
, Python will select the first value (e.g., select "a"
in "a" or "b"
), and the second one in and
expressions.This is also called short-circuiting - we already know that the logical or
must be True
if the first value is True
and therefore can omit the evaluation of the second value.
Logical and
:
a and b == b if a else a
and
expressions are True
, Python will select the second value, since for a logical and
, both values must be true.result = (2 or 3) * (5 and 7)
print('2 * 7 =', result)
2 * 7 = 14
Don't use mutable objects (e.g., dictionaries, lists, sets, etc.) as default arguments for functions! You might expect that a new list is created every time when we call the function without providing an argument for the default parameter, but this is not the case: Python will create the mutable object (default parameter) the first time the function is defined - not when it is called, see the following code:
(Original source: http://docs.python-guide.org/en/latest/writing/gotchas/
def append_to_list(value, def_list=[]):
def_list.append(value)
return def_list
my_list = append_to_list(1)
print(my_list)
my_other_list = append_to_list(2)
print(my_other_list)
[1] [1, 2]
Another good example showing that demonstrates that default arguments are created when the function is created (and not when it is called!):
import time
def report_arg(my_default=time.time()):
print(my_default)
report_arg()
time.sleep(5)
report_arg()
1528560045.3962939 1528560045.3962939
Be aware of what is happening when combining in
checks with generators, since they won't evaluate from the beginning once a position is "consumed".
gen = (i for i in range(5))
print('2 in gen,', 2 in gen)
print('3 in gen,', 3 in gen)
print('1 in gen,', 1 in gen)
2 in gen, True 3 in gen, True 1 in gen, False
Although this defeats the purpose of a generator (in most cases), we can convert a generator into a list to circumvent the problem.
gen = (i for i in range(5))
a_list = list(gen)
print('2 in l,', 2 in a_list)
print('3 in l,', 3 in a_list)
print('1 in l,', 1 in a_list)
2 in l, True 3 in l, True 1 in l, True
bool
is a subclass of int
¶Chicken or egg? In the history of Python (Python 2.2 to be specific) truth values were implemented via 1 and 0 (similar to the old C). In order to avoid syntax errors in old (but perfectly working) Python code, bool
was added as a subclass of int
in Python 2.3.
Original source: http://www.peterbe.com/plog/bool-is-int
print('isinstance(True, int):', isinstance(True, int))
print('True + True:', True + True)
print('3*True + True:', 3*True + True)
print('3*True - False:', 3*True - False)
isinstance(True, int): True True + True: 2 3*True + True: 4 3*True - False: 3
Remember the section about the consuming generators? This example is somewhat related, but the result might still come as unexpected.
(Original source: http://openhome.cc/eGossip/Blog/UnderstandingLambdaClosure3.html)
In the first example below, we call a lambda
function in a list comprehension, and the value i
will be dereferenced every time we call lambda
within the scope. Since the list comprehension has already been constructed and evaluated when we for-loop
through the list, the closure-variable will be set to the last value 4.
my_list = [lambda: i for i in range(5)]
for l in my_list:
print(l())
4 4 4 4 4
However, by using a generator expression, we can make use of its stepwise evaluation (note that the returned variable still stems from the same closure, but the value changes as we iterate over the generator).
my_gen = (lambda: n for n in range(5))
for l in my_gen:
print(l())
0 1 2 3 4
And if you are really keen on using lists, there is a nifty trick that circumvents this problem as a reader nicely pointed out in the comments: We can simply pass the loop variable i
as a default argument to the lambdas.
my_list = [lambda x=i: x for i in range(5)]
for l in my_list:
print(l())
0 1 2 3 4
global
and nonlocal
¶There is nothing particularly surprising about Python's LEGB scope resolution (Local -> Enclosed -> Global -> Built-in), but it is still useful to take a look at some examples!
global
vs. local
¶According to the LEGB rule, Python will first look for a variable in the local scope. So if we set the variable x = 1
local
ly in the function's scope, it won't have an effect on the global
x
.
x = 0
def in_func():
x = 1
print('in_func:', x)
in_func()
print('global:', x)
in_func: 1 global: 0
If we want to modify the global
x via a function, we can simply use the global
keyword to import the variable into the function's scope:
x = 0
def in_func():
global x
x = 1
print('in_func:', x)
in_func()
print('global:', x)
in_func: 1 global: 1
local
vs. enclosed
¶Now, let us take a look at local
vs. enclosed
. Here, we set the variable x = 1
in the outer
function and set x = 1
in the enclosed function inner
. Since inner
looks in the local scope first, it won't modify outer
's x
.
def outer():
x = 1
print('outer before:', x)
def inner():
x = 2
print("inner:", x)
inner()
print("outer after:", x)
outer()
outer before: 1 inner: 2 outer after: 1
Here is where the nonlocal
keyword comes in handy - it allows us to modify the x
variable in the enclosed
scope:
def outer():
x = 1
print('outer before:', x)
def inner():
nonlocal x
x = 2
print("inner:", x)
inner()
print("outer after:", x)
outer()
outer before: 1 inner: 2 outer after: 2
As we all know, tuples are immutable objects in Python, right!? But what happens if they contain mutable objects?
First, let us have a look at the expected behavior: a TypeError
is raised if we try to modify immutable types in a tuple:
tup = (1,)
tup[0] += 1
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-27-ddf241f0ac4f> in <module>() 1 tup = (1,) ----> 2 tup[0] += 1 TypeError: 'tuple' object does not support item assignment
TypeError
at the same time.¶tup = ([],)
print('tup before: ', tup)
tup[0] += [1]
tup before: ([],)
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-28-8baa7ba7b4d3> in <module>() 1 tup = ([],) 2 print('tup before: ', tup) ----> 3 tup[0] += [1] TypeError: 'tuple' object does not support item assignment
print('tup after: ', tup)
tup = ([],)
print('tup before: ', tup)
tup[0].extend([1])
print('tup after: ', tup)
tup before: ([],) tup after: ([1],)
tup = ([],)
print('tup before: ', tup)
tup[0].append(1)
print('tup after: ', tup)
tup before: ([],) tup after: ([1],)
A. Jesse Jiryu Davis has a nice explanation for this phenomenon (Original source: http://emptysqua.re/blog/python-increment-is-weird-part-ii/)
If we try to extend the list via +=
"then the statement executes STORE_SUBSCR
, which calls the C function PyObject_SetItem
, which checks if the object supports item assignment. In our case the object is a tuple, so PyObject_SetItem
throws the TypeError
. Mystery solved."
immutable
status of tuples. Tuples are famous for being immutable. However, how comes that this code works?¶my_tup = (1,)
my_tup += (4,)
my_tup = my_tup + (5,)
print(my_tup)
(1, 4, 5)
What happens "behind" the curtains is that the tuple is not modified, but a new object is generated every time, which will inherit the old "name tag":
my_tup = (1,)
print(id(my_tup))
my_tup += (4,)
print(id(my_tup))
my_tup = my_tup + (5,)
print(id(my_tup))
4486707912 4485211784 4486955152
"List comprehensions are fast, but generators are faster!?" - No, not really (or significantly, see the benchmarks below). So what's the reason to prefer one over the other?
import timeit
def plainlist(n=100000):
my_list = []
for i in range(n):
if i % 5 == 0:
my_list.append(i)
return my_list
def listcompr(n=100000):
my_list = [i for i in range(n) if i % 5 == 0]
return my_list
def generator(n=100000):
my_gen = (i for i in range(n) if i % 5 == 0)
return my_gen
def generator_yield(n=100000):
for i in range(n):
if i % 5 == 0:
yield i
def test_plainlist(plain_list):
for i in plain_list():
pass
def test_listcompr(listcompr):
for i in listcompr():
pass
def test_generator(generator):
for i in generator():
pass
def test_generator_yield(generator_yield):
for i in generator_yield():
pass
print('plain_list: ', end='')
%timeit test_plainlist(plainlist)
print('\nlistcompr: ', end='')
%timeit test_listcompr(listcompr)
print('\ngenerator: ', end='')
%timeit test_generator(generator)
print('\ngenerator_yield: ', end='')
%timeit test_generator_yield(generator_yield)
plain_list: 10.8 ms ± 793 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) listcompr: 10 ms ± 830 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) generator: 11.4 ms ± 1 ms per loop (mean ± std. dev. of 7 runs, 100 loops each) generator_yield: 12.3 ms ± 1.82 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
Who has not stumbled across this quote "we are all consenting adults here" in the Python community, yet? Unlike in other languages like C++ (sorry, there are many more, but that's one I am most familiar with), we can't really protect class methods from being used outside the class (i.e., by the API user).
All we can do is indicate methods as private to make clear that they are not to be used outside the class, but it really is up to the class user, since "we are all consenting adults here"!
So, when we want to mark a class method as private, we can put a single underscore in front of it.
If we additionally want to avoid name clashes with other classes that might use the same method names, we can prefix the name with a double-underscore to invoke the name mangling.
This doesn't prevent the class users to access this class member though, but they have to know the trick and also know that it is at their own risk...
Let the following example illustrate what I mean:
class my_class():
def public_method(self):
print('Hello public world!')
def __private_method(self):
print('Hello private world!')
def call_private_method_in_class(self):
self.__private_method()
my_instance = my_class()
my_instance.public_method()
my_instance._my_class__private_method()
my_instance.call_private_method_in_class()
Hello public world! Hello private world! Hello private world!
It can be really dangerous to modify a list when iterating through it - this is a very common pitfall that can cause unintended behavior!
Look at the following examples, and for a fun exercise: try to figure out what is going on before you skip to the solution!
a = [1, 2, 3, 4, 5]
for i in a:
if not i % 2:
a.remove(i)
print(a)
[1, 3, 5]
b = [2, 4, 5, 6]
for i in b:
if not i % 2:
b.remove(i)
print(b)
[4, 5]
b = [2, 4, 5, 6]
for index, item in enumerate(b):
print(index, item)
if not item % 2:
b.remove(item)
print(b)
0 2 1 5 2 6 [4, 5]
Be careful, dynamic binding is convenient, but can also quickly become dangerous!
print('first list:')
for i in range(3):
print(i)
print('\nsecond list:')
for j in range(3):
print(i) # I (intentionally) made typo here!
first list: 0 1 2 second list: 2 2 2
As we have all encountered it 1 (x10000) time(s) in our lives, the infamous IndexError
:
my_list = [1, 2, 3, 4, 5]
print(my_list[5])
--------------------------------------------------------------------------- IndexError Traceback (most recent call last) <ipython-input-40-2f6b582502c3> in <module>() 1 my_list = [1, 2, 3, 4, 5] ----> 2 print(my_list[5]) IndexError: list index out of range
But suprisingly, it is not raised when we are doing list slicing, which can be a real pain when debugging:
my_list = [1, 2, 3, 4, 5]
print(my_list[5:])
UnboundLocalErrors
¶Usually, it is no problem to access global variables in the local scope of a function:
def my_func():
print(var)
var = 'global'
my_func()
And is also no problem to use the same variable name in the local scope without affecting the local counterpart:
def my_func():
var = 'locally changed'
var = 'global'
my_func()
print(var)
But we have to be careful if we use a variable name that occurs in the global scope, and we want to access it in the local function scope if we want to reuse this name:
def my_func():
print(var) # want to access global variable
var = 'locally changed' # but Python thinks we forgot to define the local variable!
var = 'global'
my_func()
In this case, we have to use the global
keyword!
def my_func():
global var
print(var) # want to access global variable
var = 'locally changed' # changes the gobal variable
var = 'global'
my_func()
print(var)
Let's assume a scenario where we want to duplicate sublist
s of values stored in another list. If we want to create an independent sublist
object, using the arithmetic multiplication operator could lead to rather unexpected (or undesired) results:
my_list1 = [[1, 2, 3]] * 2
print('initially ---> ', my_list1)
# modify the 1st element of the 2nd sublist
my_list1[1][0] = 'a'
print("after my_list1[1][0] = 'a' ---> ", my_list1)
my_list2 = [[1, 2, 3] for i in range(2)]
print('initially: ---> ', my_list2)
# modify the 1st element of the 2nd sublist
my_list2[1][0] = 'a'
print("after my_list2[1][0] = 'a': ---> ", my_list2)
for a, b in zip(my_list1, my_list2):
print('id my_list1: {}, id my_list2: {}'.format(id(a), id(b)))
There are some good articles already that are summarizing the differences between Python 2 and 3, e.g.,
etc.
But it might be still worthwhile, especially for Python newcomers, to take a look at some of those! (Note: the the code was executed in Python 3.4.0 and Python 2.7.5 and copied from interactive shell sessions.)
#############
# Python 2 #
#############
>>> type(unicode('is like a python3 str()'))
<type 'unicode'>
>>> type(b'byte type does not exist')
<type 'str'>
>>> 'they are really' + b' the same'
'they are really the same'
>>> type(bytearray(b'bytearray oddly does exist though'))
<type 'bytearray'>
#############
# Python 3
#############
>>> print('strings are now utf-8 \u03BCnico\u0394é!')
strings are now utf-8 μnicoΔé!
>>> type(b' and we have byte types for storing data')
<class 'bytes'>
>>> type(bytearray(b'but also bytearrays for those who prefer them over strings'))
<class 'bytearray'>
>>> 'string' + b'bytes for data'
Traceback (most recent call last):s
File "<stdin>", line 1, in <module>
TypeError: Can't convert 'bytes' object to str implicitly
Very trivial, but this change makes sense, Python 3 now only accepts print
s with proper parentheses - just like the other function calls ...
# Python 2
>>> print 'Hello, World!'
Hello, World!
>>> print('Hello, World!')
Hello, World!
# Python 3
>>> print('Hello, World!')
Hello, World!
>>> print 'Hello, World!'
File "<stdin>", line 1
print 'Hello, World!'
^
SyntaxError: invalid syntax
And if we want to print the output of 2 consecutive print functions on the same line, you would use a comma in Python 2, and a end=""
in Python 3:
# Python 2
>>> print "line 1", ; print 'same line'
line 1 same line
# Python 3
>>> print("line 1", end="") ; print (" same line")
line 1 same line
This is a pretty dangerous thing if you are porting code, or executing Python 3 code in Python 2 since the change in integer-division behavior can often go unnoticed.
So, I still tend to use a float(3)/2
or 3/2.0
instead of a 3/2
in my Python 3 scripts to save the Python 2 guys some trouble ... (PS: and vice versa, you can from __future__ import division
in your Python 2 scripts).
# Python 2
>>> 3 / 2
1
>>> 3 // 2
1
>>> 3 / 2.0
1.5
>>> 3 // 2.0
1.0
# Python 3
>>> 3 / 2
1.5
>>> 3 // 2
1
>>> 3 / 2.0
1.5
>>> 3 // 2.0
1.0
xrange()
¶xrange()
was pretty popular in Python 2.x if you wanted to create an iterable object. The behavior was quite similar to a generator ('lazy evaluation'), but you could iterate over it infinitely. The advantage was that it was generally faster than range()
(e.g., in a for-loop) - not if you had to iterate over the list multiple times, since the generation happens every time from scratch!
In Python 3, the range()
was implemented like the xrange()
function so that a dedicated xrange()
function does not exist anymore.
# Python 2
>>> python -m timeit 'for i in range(1000000):' ' pass'
10 loops, best of 3: 66 msec per loop
> python -m timeit 'for i in xrange(1000000):' ' pass'
10 loops, best of 3: 27.8 msec per loop
# Python 3
>>> python3 -m timeit 'for i in range(1000000):' ' pass'
10 loops, best of 3: 51.1 msec per loop
>>> python3 -m timeit 'for i in xrange(1000000):' ' pass'
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/timeit.py", line 292, in main
x = t.timeit(number)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/timeit.py", line 178, in timeit
timing = self.inner(it, self.timer)
File "<timeit-src>", line 6, in inner
for i in xrange(1000000):
NameError: name 'xrange' is not defined
Where Python 2 accepts both notations, the 'old' and the 'new' way, Python 3 chokes (and raises a SyntaxError
in turn) if we don't enclose the exception argument in parentheses:
# Python 2
>>> raise IOError, "file error"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: file error
>>> raise IOError("file error")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: file error
# Python 3
>>> raise IOError, "file error"
File "<stdin>", line 1
raise IOError, "file error"
^
SyntaxError: invalid syntax
>>> raise IOError("file error")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: file error
Also the handling of exceptions has slightly changed in Python 3. Now, we have to use the as
keyword!
# Python 2
>>> try:
... blabla
... except NameError, err:
... print err, '--> our error msg'
...
name 'blabla' is not defined --> our error msg
# Python 3
>>> try:
... blabla
... except NameError as err:
... print(err, '--> our error msg')
...
name 'blabla' is not defined --> our error msg
next()
function and .next()
method¶Where you can use both function and method in Python 2.7.5, the next()
function is all that remains in Python 3!
# Python 2
>>> my_generator = (letter for letter in 'abcdefg')
>>> my_generator.next()
'a'
>>> next(my_generator)
'b'
# Python 3
>>> my_generator = (letter for letter in 'abcdefg')
>>> next(my_generator)
'a'
>>> my_generator.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'generator' object has no attribute 'next'
This goes back to a change that was made in Python 3.x and is described in What’s New In Python 3.0 as follows:
"List comprehensions no longer support the syntactic form [... for var in item1, item2, ...]
. Use [... for var in (item1, item2, ...)]
instead. Also note that list comprehensions have different semantics: they are closer to syntactic sugar for a generator expression inside a list()
constructor, and in particular the loop control variables are no longer leaked into the surrounding scope."
>>> from platform import python_version
>>> print 'This code cell was executed in Python', python_version()
'This code cell was executed in Python 2.7.6'
>>> i = 1
>>> print [i for i in range(5)]
'[0, 1, 2, 3, 4]'
>>> print i, '-> i in global'
'4 -> i in global'
%%python3
from platform import python_version
print('This code cell was executed in Python', python_version())
i = 1
print([i for i in range(5)])
print(i, '-> i in global')
This code cell was executed in Python 3.6.4 [0, 1, 2, 3, 4] 1 -> i in global
>>> from platform import python_version
>>> print 'This code cell was executed in Python', python_version()
'This code cell was executed in Python 2.7.6'
>>> print [1, 2] > 'foo'
'False'
>>> print (1, 2) > 'foo'
'True'
>>> print [1, 2] > (1, 2)
'False'
Couldn't find program: 'python2'
from platform import python_version
print('This code cell was executed in Python', python_version())
print([1, 2] > 'foo')
print((1, 2) > 'foo')
print([1, 2] > (1, 2))
This code cell was executed in Python 3.6.4
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-67-40a145eac137> in <module>() 2 print('This code cell was executed in Python', python_version()) 3 ----> 4 print([1, 2] > 'foo') 5 print((1, 2) > 'foo') 6 print([1, 2] > (1, 2)) TypeError: '>' not supported between instances of 'list' and 'str'
->
's in my Python code?¶Have you ever seen any Python code that used colons inside the parantheses of a function definition?
def foo1(x: 'insert x here', y: 'insert x^2 here'):
print('Hello, World')
return
And what about the fancy arrow here?
def foo2(x, y) -> 'Hi!':
print('Hello, World')
return
Q: Is this valid Python syntax?
A: Yes!
Q: So, what happens if I just call the function?
A: Nothing!
Here is the proof!
foo1(1,2)
Hello, World
foo2(1,2)
Hello, World
So, those are function annotations ...
You probably will never make use of them (or at least very rarely). Usually, we write good function documentations below the function as a docstring - or at least this is how I would do it (okay this case is a little bit extreme, I have to admit):
def is_palindrome(a):
"""
Case-and punctuation insensitive check if a string is a palindrom.
Keyword arguments:
a (str): The string to be checked if it is a palindrome.
Returns `True` if input string is a palindrome, else False.
"""
stripped_str = [l for l in my_str.lower() if l.isalpha()]
return stripped_str == stripped_str[::-1]
However, function annotations can be useful to indicate that work is still in progress in some cases. But they are optional and I see them very, very rarely.
As it is stated in PEP3107:
Function annotations, both for parameters and return values, are completely optional.
Function annotations are nothing more than a way of associating arbitrary Python expressions with various parts of a function at compile-time.
The nice thing about function annotations is their __annotations__
attribute, which is a dictionary of all the parameters and/or the return
value you annotated.
foo1.__annotations__
{'x': 'insert x here', 'y': 'insert x^2 here'}
foo2.__annotations__
{'return': 'Hi!'}
When are they useful?
Function annotations can be useful for a couple of things
...
finally
blocks¶Python's try-except-finally
blocks are very handy for catching and handling errors. The finally
block is always executed whether an exception
has been raised or not as illustrated in the following example.
def try_finally1():
try:
print('in try:')
print('do some stuff')
float('abc')
except ValueError:
print('an error occurred')
else:
print('no error occurred')
finally:
print('always execute finally')
try_finally1()
def try_finally2():
try:
print("do some stuff in try block")
return "return from try block"
finally:
print("do some stuff in finally block")
return "always execute finally"
print(try_finally2())
do some stuff in try block do some stuff in finally block always execute finally
I am not yet sure in which context this can be useful, but it is a nice fun fact to know that we can assign types as values to variables.
a_var = str
a_var(123)
'123'
from random import choice
a, b, c = float, int, str
for i in range(5):
j = choice([a,b,c])(i)
print(j, type(j))
0.0 <class 'float'> 1.0 <class 'float'> 2 <class 'int'> 3.0 <class 'float'> 4.0 <class 'float'>
The main reason why we love to use generators in certain cases (i.e., when we are dealing with large numbers of computations) is that it only computes the next value when it is needed, which is also known as "lazy" evaluation. However, the first clause of an generator is already checked upon it's creation, as the following example demonstrates:
gen_fails = (i for i in 1/0)
--------------------------------------------------------------------------- ZeroDivisionError Traceback (most recent call last) <ipython-input-24-1207434291a1> in <module>() ----> 1 gen_fails = (i for i in 1/0) ZeroDivisionError: division by zero
Certainly, this is a nice feature, since it notifies us about syntax erros immediately. However, this is (unfortunately) not the case if we have multiple cases in our generator.
gen_succeeds = (i for i in range(5) for j in 1/0)
print('But obviously fails when we iterate ...')
for i in gen_succeeds:
print(i)
But obviously fails when we iterate ...
--------------------------------------------------------------------------- ZeroDivisionError Traceback (most recent call last) <ipython-input-26-b681b1d9a91c> in <module>() 1 print('But obviously fails when we iterate ...') ----> 2 for i in gen_succeeds: 3 print(i) <ipython-input-25-31ec63d21a60> in <genexpr>(.0) ----> 1 gen_succeeds = (i for i in range(5) for j in 1/0) ZeroDivisionError: division by zero
*args
and **kwargs
¶Python has a very convenient "keyword argument unpacking syntax" (often referred to as "splat"-operators). This is particularly useful, if we want to define a function that can take a arbitrary number of input arguments.
def a_func(*args):
print('type of args:', type(args))
print('args contents:', args)
print('1st argument:', args[0])
a_func(0, 1, 'a', 'b', 'c')
type of args: <class 'tuple'> args contents: (0, 1, 'a', 'b', 'c') 1st argument: 0
def b_func(**kwargs):
print('type of kwargs:', type(kwargs))
print('kwargs contents: ', kwargs)
print('value of argument a:', kwargs['a'])
b_func(a=1, b=2, c=3, d=4)
type of kwargs: <class 'dict'> kwargs contents: {'a': 1, 'b': 2, 'c': 3, 'd': 4} value of argument a: 1
Another useful application of the "unpacking"-operator is the unpacking of lists and other other iterables.
val1, *vals = [1, 2, 3, 4, 5]
print('val1:', val1)
print('vals:', vals)
val1: 1 vals: [2, 3, 4, 5]
Usually, it is the __init__
method when we think of instanciating a new object from a class. However, it is the static method __new__
(it is not a class method!) that creates and returns a new instance before __init__()
is called.
More specifically, this is what is returned:
return super(<currentclass>, cls).__new__(subcls, *args, **kwargs)
For more information about the __new__
method, please see the documentation.
As a little experiment, let us screw with __new__
so that it returns None
and see if __init__
will be executed:
class a_class(object):
def __new__(clss, *args, **kwargs):
print('excecuted __new__')
return None
def __init__(self, an_arg):
print('excecuted __init__')
self.an_arg = an_arg
a_object = a_class(1)
print('Type of a_object:', type(a_object))
excecuted __new__ Type of a_object: <class 'NoneType'>
As we can see in the code above, __init__
requires the returned instance from __new__
in order to called. So, here we just created a NoneType
object.
Let us override the __new__
, now and let us confirm that __init__
is called now to instantiate the new object":
class a_class(object):
def __new__(cls, *args, **kwargs):
print('excecuted __new__')
inst = super(a_class, cls).__new__(cls)
return inst
def __init__(self, an_arg):
print('excecuted __init__')
self.an_arg = an_arg
a_object = a_class(1)
print('Type of a_object:', type(a_object))
print('a_object.an_arg: ', a_object.an_arg)
excecuted __new__ excecuted __init__ Type of a_object: <class '__main__.a_class'> a_object.an_arg: 1
for i in range(5):
if i == 1:
print('in for')
else:
print('in else')
print('after for-loop')
in for in else after for-loop
for i in range(5):
if i == 1:
break
else:
print('in else')
print('after for-loop')
after for-loop
I would claim that the conditional else
is every programmer's daily bread and butter. However, there is a second flavor of else
-clauses in Python, which I will call "completion else" (for reason that will become clear later).
But first, let us take a look at our "traditional" conditional else that we all are familiar with.
# conditional else
a_list = [1,2]
if a_list[0] == 1:
print('Hello, World!')
else:
print('Bye, World!')
Hello, World!
# conditional else
a_list = [1,2]
if a_list[0] == 2:
print('Hello, World!')
else:
print('Bye, World!')
Bye, World!
Why am I showing those simple examples? I think they are good to highlight some of the key points: It is either the code under the if
clause that is executed, or the code under the else
block, but not both.
If the condition of the if
clause evaluates to True
, the if
-block is exectured, and if it evaluated to False
, it is the else
block.
try:
print('first element:', a_list[0])
except IndexError:
print('raised IndexError')
else:
print('no error in try-block')
first element: 1 no error in try-block
try:
print('third element:', a_list[2])
except IndexError:
print('raised IndexError')
else:
print('no error in try-block')
raised IndexError
In the code above, we can see that the code under the else
-clause is only executed if the try-block
was executed without encountering an error, i.e., if the try
-block is "complete".
The same rule applies to the "completion" else
in while- and for-loops, which you can confirm in the following samples below.
i = 0
while i < 2:
print(i)
i += 1
else:
print('in else')
0 1 in else
i = 0
while i < 2:
print(i)
i += 1
break
else:
print('completed while-loop')
0
for i in range(2):
print(i)
else:
print('completed for-loop')
0 1 completed for-loop
for i in range(2):
print(i)
break
else:
print('completed for-loop')
0
This might not be particularly useful, but it is nonetheless interesting: Python's interpreter is interning compile-time constants but not run-time expressions (note that this is implementation-specific).
(Original source: Stackoverflow)
Let us have a look at the simple example below. Here we are creating 3 variables and assign the value "Hello" to them in different ways before we test them for identity.
hello1 = 'Hello'
hello2 = 'Hell' + 'o'
hello3 = 'Hell'
hello3 = hello3 + 'o'
print('hello1 is hello2:', hello1 is hello2)
print('hello1 is hello3:', hello1 is hello3)
hello1 is hello2: True hello1 is hello3: False
Now, how does it come that the first expression evaluates to true, but the second does not? To answer this question, we need to take a closer look at the underlying byte codes:
import dis
def hello1_func():
s = 'Hello'
return s
dis.dis(hello1_func)
3 0 LOAD_CONST 1 ('Hello') 2 STORE_FAST 0 (s) 4 4 LOAD_FAST 0 (s) 6 RETURN_VALUE
def hello2_func():
s = 'Hell' + 'o'
return s
dis.dis(hello2_func)
2 0 LOAD_CONST 3 ('Hello') 2 STORE_FAST 0 (s) 3 4 LOAD_FAST 0 (s) 6 RETURN_VALUE
def hello3_func():
s = 'Hell'
s = s + 'o'
return s
dis.dis(hello3_func)
2 0 LOAD_CONST 1 ('Hell') 2 STORE_FAST 0 (s) 3 4 LOAD_FAST 0 (s) 6 LOAD_CONST 2 ('o') 8 BINARY_ADD 10 STORE_FAST 0 (s) 4 12 LOAD_FAST 0 (s) 14 RETURN_VALUE
print(hello1_func() is hello2_func())
print(hello1_func() is hello3_func())
True False
Finally, to show that this hypothesis is the answer to this rather unexpected observation, let us intern
the value manually:
import sys
print(hello1_func() is sys.intern(hello3_func()))
True