Table of Contents
Python Distilled Python Basics
Return to Python Distilled, Python books
Python: Python Variables, Python Data Types, Python Control Structures, Python Loops, Python Functions, Python Modules, Python Packages, Python File Handling, Python Errors and Exceptions, Python Classes and Objects, Python Inheritance, Python Polymorphism, Python Encapsulation, Python Abstraction, Python Lists, Python Dictionaries, Python Tuples, Python Sets, Python String Manipulation, Python Regular Expressions, Python Comprehensions, Python Lambda Functions, Python Map, Filter, and Reduce, Python Decorators, Python Generators, Python Context Managers, Python Concurrency with Threads, Python Asynchronous Programming, Python Multiprocessing, Python Networking, Python Database Interaction, Python Debugging, Python Testing and Unit Testing, Python Virtual Environments, Python Package Management, Python Data Analysis, Python Data Visualization, Python Web Scraping, Python Web Development with Flask/Django, Python API Interaction, Python GUI Programming, Python Game Development, Python Security and Cryptography, Python Blockchain Programming, Python Machine Learning, Python Deep Learning, Python Natural Language Processing, Python Computer Vision, Python Robotics, Python Scientific Computing, Python Data Engineering, Python Cloud Computing, Python DevOps Tools, Python Performance Optimization, Python Design Patterns, Python Type Hints, Python Version Control with Git, Python Documentation, Python Internationalization and Localization, Python Accessibility, Python Configurations and Environments, Python Continuous Integration/Continuous Deployment, Python Algorithm Design, Python Problem Solving, Python Code Readability, Python Software Architecture, Python Refactoring, Python Integration with Other Languages, Python Microservices Architecture, Python Serverless Computing, Python Big Data Analysis, Python Internet of Things (IoT), Python Geospatial Analysis, Python Quantum Computing, Python Bioinformatics, Python Ethical Hacking, Python Artificial Intelligence, Python Augmented Reality and Virtual Reality, Python Blockchain Applications, Python Chatbots, Python Voice Assistants, Python Edge Computing, Python Graph Algorithms, Python Social Network Analysis, Python Time Series Analysis, Python Image Processing, Python Audio Processing, Python Video Processing, Python 3D Programming, Python Parallel Computing, Python Event-Driven Programming, Python Reactive Programming.
Variables, Data Types, Control Structures, Loops, Functions, Modules, Packages, File Handling, Errors and Exceptions, Classes and Objects, Inheritance, Polymorphism, Encapsulation, Abstraction, Lists, Dictionaries, Tuples, Sets, String Manipulation, Regular Expressions, Comprehensions, Lambda Functions, Map, Filter, and Reduce, Decorators, Generators, Context Managers, Concurrency with Threads, Asynchronous Programming, Multiprocessing, Networking, Database Interaction, Debugging, Testing and Unit Testing, Virtual Environments, Package Management, Data Analysis, Data Visualization, Web Scraping, Web Development with Flask/Django, API Interaction, GUI Programming, Game Development, Security and Cryptography, Blockchain Programming, Machine Learning, Deep Learning, Natural Language Processing, Computer Vision, Robotics, Scientific Computing, Data Engineering, Cloud Computing, DevOps Tools, Performance Optimization, Design Patterns, Type Hints, Version Control with Git, Documentation, Internationalization and Localization, Accessibility, Configurations and Environments, Continuous Integration/Continuous Deployment, Algorithm Design, Problem Solving, Code Readability, Software Architecture, Refactoring, Integration with Other Languages, Microservices Architecture, Serverless Computing, Big Data Analysis, Internet of Things (IoT), Geospatial Analysis, Quantum Computing, Bioinformatics, Ethical Hacking, Artificial Intelligence, Augmented Reality and Virtual Reality, Blockchain Applications, Chatbots, Voice Assistants, Edge Computing, Graph Algorithms, Social Network Analysis, Time Series Analysis, Image Processing, Audio Processing, Video Processing, 3D Programming, Parallel Computing, Event-Driven Programming, Reactive Programming.
Python Glossary, Python Fundamentals, Python Inventor: Python Language Designer: Guido van Rossum on 20 February 1991; PEPs, Python Scripting, Python Keywords, Python Built-In Data Types, Python Data Structures - Python Algorithms, Python Syntax, Python OOP - Python Design Patterns, Python Module Index, pymotw.com, Python Package Manager (pip-PyPI), Python Virtualization (Conda, Miniconda, Virtualenv, Pipenv, Poetry), Python Interpreter, CPython, Python REPL, Python IDEs (PyCharm, Jupyter Notebook), Python Development Tools, Python Linter, Pythonista-Python User, Python Uses, List of Python Software, Python Popularity, Python Compiler, Python Transpiler, Python DevOps - Python SRE, Python Data Science - Python DataOps, Python Machine Learning, Python Deep Learning, Functional Python, Python Concurrency - Python GIL - Python Async (Asyncio), Python Standard Library, Python Testing (Pytest), Python Libraries (Flask), Python Frameworks (Django), Python History, Python Bibliography, Manning Python Series, Python Official Glossary - Python Glossary, Python Topics, Python Courses, Python Research, Python GitHub, Written in Python, Python Awesome List, Python Versions. (navbar_python - see also navbar_python_libaries, navbar_python_standard_library, navbar_python_virtual_environments, navbar_numpy, navbar_datascience)
Fair Use Source: B094CMKN2J, PyDis
This chapter gives an overview of the core of the Python language. It covers variables, data types, expressions, control flow, functions, classes, and input/output. The chapter concludes with a discussion of modules, script writing, packages, and a few tips on organizing larger programs. This chapter is not trying to provide comprehensive coverage of every feature, nor does it concern itself with all of the tooling that might surround a larger Python project. However, experienced]] programmers should be able to extrapolate from the material here to write more advanced programs. Newcomers are en[[couraged to try the examples in a simple environment, such as a terminal window and a basic text [[editor.
Python programs are executed by an interpreter. There are many different environments in which the Python [[interpreter might run—an IDE, a browser, or a terminal window. However, underneath all that, the core of the interpreter is a text-based application that can be started by typing python in a command shell such as bash. Since Python 2 and Python 3 might both be installed on the same machine, you might need to type python2 or python3 to pick a version. This book assumes Python 3.8 or newer.
When the interpreter starts, a prompt appears where you can type programs into a so-called “read-evaluation-print loop” (or REPL). For example, in the following output, the interpreter displays its copyright message and presents the user with the »> prompt, at which the user types a familiar “Hello World” program:
Python 3.8.0 (default, Feb 3 2019, 05:53:21)
GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type “help”, “copyright”, “cr[[edits” or “license” for more information.
print('Hello World')
Certain environments may display a different prompt. The following output is from ipython (an alternate shell for Python):
Python 3.8.0 (default, Feb 4, 2019, 07:39:16)
Type 'copyright', 'cr[[edits' or 'license' for more information
IPython 6.5.0 – An enhanced Interactive Python. Type '?' for help.
In [1]: print('Hello World')
In [2]:
Regardless of the exact form of output you see, the underlying principle is the same. You type a command, it runs, and you immediately see the output.
Python’s interactive mode is one of its most useful features because you can type any valid statement and immediately see the result. This is useful for debugging and experimentation. Many people, including the author, use interactive Python as their desktop calculator. For example:
6000 + 4523.50 + 134.25
10657.75
_ + 8192.75
18850.5
When you use Python interactively, the variable _ holds the result of the last operation. This is useful if you want to use that result in subsequent statements. This variable only gets defined when working interactively, so don’t use it in saved programs.
You can exit the interactive interpreter by typing quit() or the EOF (end of file]]) character. On UNIX, EOF is Ctrl+D; on Windows, it’s Ctrl+Z.
If you want to create a program that you can run repeatedly, put statements in a text file. For example:
print('Hello World')
Python source files are UTF-8-en[[coded]] text files that normally have a .py suf[[fix. The # character denotes a comment that extends to the end of the line. International (Unicode) characters can be freely used in the source [[code as long as you use the UTF-8 encoding (this is the default in most editors, but it never hurts to check your editor settings if you’re unsure).
To execute the hello.py file, provide the filename to the interpreter as follows:
shell %
print('Hello World')
On UNIX, if you give this file execute permissions (for example, by chmod +x hello.py), you can run the program by typing hello.py into your shell.
On Windows, you can double-click on a .py file or type the name of the program into the Run command on the Windows Start menu to launch it. The #! line, if given, is used to pick the interpreter version (Python 2 versus 3). Execution of a program might take place in a console window that disappears immediately after the program completes— often before you can read its output. For debugging, it’s better to run the program within a Python development environment.
The interpreter runs statements in order until it reaches the end of the input file. At that point, the program terminates and Python exits.
1.3 Primitives, Variables, and Expressions
Python provides a collection of primitive types such as integers, floats, and strings:
42 # int
4.2 # float
A variable is a name that refers to a value. A value represents an object of some type:
x = 42
Sometimes you might see a type explicitly attached to a name. For example:
x: int = 42
The type is merely a hint to improve code readability. It can be used by third-party code-checking tools. Otherwise, it is completely ignored. It does not prevent you from assigning]] a different kind of value later.
An expression is a combination of primitives, names, and operators that produces a value:
2 + 3 * 4 # → 14
The following program uses variables and expressions to perform a compound-interest calculation:
principal = 1000 # Initial a[[mount
rate = 0.05 # Interest rate
numyears = 5 # Number of years
year = 1
- (1 + rate)
year += 1
When executed, it produces the following output:
1 1050.0
2 1102.5
3 1157.625
4 1215.5062500000001
5 1276.2815625000003
The while statement tests the conditional expression that immediately follows. If the tested condition is true, the body of the while statement executes. The condition is then retested and the body executed again until the condition becomes false. The body of the loop is denoted by indentation. Thus, the three statements following while in interest.py execute on each iteration. Python doesn’t specify the a[[mount of required indentation, as long as it’s consistent within a block. It is most common to use four spaces per indentation level.
One problem with the interest.py program is that the output isn’t very pretty. To make it better, you could right-align the columns and limit the precision of principal to two digits. Change the print() function to use a so-called f-string like this:
print(f'{year:>3d} {principal:0.2f}')
In the f-string, variable names and expressions can be evaluated by enclosing them in curly]] braces. Optionally, each substitution can have a formatting specifier attached to it. '>3d' means a three-digit decimal number, right aligned. '0.2f' means a floating-point number with two decimal places of accuracy. More information about these formatting codes can be found in Chapter 9.
Now the output of the program looks like this:
1 1050.00
2 1102.50
3 1157.62
4 1215.51
5 1276.28
1.4 Arithmetic Operators
Python has a standard set of mathematical operators, shown in Table 1.1. These operators have the same meaning they do in most other programming languages.
Table 1.1 Arithmetic Operators

The division operator (/) produces a floating-point number when applied to integers. Therefore, 7/4 is 1.75. The truncating division operator //, also known as floor division, truncates the result to an integer and works with both integers and floating-point numbers. The modulo operator returns the remainder of the division x // y. For example, 7 % 4 is 3. For floating-point numbers, the modulo operator returns the floating-point remainder of x // y, which is x – (x // y) * y.
In addition, the built-in functions in Table 1.2 provide a few more commonly used numerical]] operations.
Table 1.2 Common Mathematic Functions

The round() function implements “banker’s rounding]].” If the value being rounded is equally close to two multiples, it is rounded to the nearest even multiple (for example, 0.5 is rounded to 0.0, and 1.5 is rounded to 2.0).
Integers provide a few additional]] operators to support bit manipulation, shown in Table 1.3.
Table 1.3 Bit Manipulation Operators

One would commonly use these with binary integers. For example:
a = 0b11001001
mask = 0b11110000
x = (a & mask) » 4 # x = 0b1100 (12)
In this example, 0b11001001 is how you write an integer value in binary. You could have written it as decimal 201 or hexadecimal 0xc9, but if you’re fiddling with bits, binary makes it easier to visualize what you’re doing.
The semantics of the bitwise [[operators assumes that the integers use a two’s complement binary representation and that the sign bit is infinitely extended to the left. Some care is required if you are working with raw bit patterns that are intended to map to native integers on the hardware. This is because Python does not truncate the bits or allow values to overflow—instead, the result will grow arbitrarily large in magnitude. It’s up to you to make sure the result is properly sized or truncated if needed.
To compare numbers, use the comparison operators in Table 1.4.
Table 1.4 Comparison Operators

The result of a comparison is a Boolean value True or False.
The and, or, and not operators (not to be confused with the bit-manipulation operators above) can form more complex Boolean expressions. The behavior of these operators is as shown in Table 1.5.
Table 1.5 Logical [[Operators

A value is considered false if it is literally False, None, numerical]]ly zero, or empty. Otherwise, it’s considered true.
It is common to write an expression that updates a value. For example:
x = x + 1
y = y * n
For these, you can write the following shortened operation instead:
x += 1
y *= n
This shortened form of update can be used with any of the +, -, *, **, /, //, %, &, |, ^, «, » operators. Python does not have increment (++) or decrement (–) operators found in some other languages.
1.5 Conditionals and Control Flow
The while, if and else statements are used for [[looping and conditional code execution. Here’s an example:
if a < b:
else:
The bodies of the if and else clauses are denoted by indentation. The else clause is optional. To create an empty clause, use the pass statement, as follows:
if a < b:
- Do nothing
else:
To handle multiple-test cases, use the elif statement:
elif suf[[fix == '.png':
else:
raise RuntimeError(f'Unknown content type]] {suf[[fix!r}')
If you are assigning]] a value in combination with a test, use a conditional expression:
maxval = a if a > b else b
This is the same as the longer:
if a > b:
maxval = a
else:
maxval = b
Sometimes, you may see the assignment of a variable and a conditional combined together using the := operator. This is known as an assignment expression (or more colloquially as the “walrus operator” because := looks like a walrus tipped over on its side—presumably playing dead). For example:
x = 0
while (x := x + 1) < 10: # Prints 1, 2, 3, …, 9
print(x)
The parenthese]]s used to surround an assignment expression are always required.
The break statement]] can be used to abort a loop early. It only applies to the innermost loop. For example:
x = 0
while x < 10:
if x == 5:
print(x)
x += 1
print('Done')
The continue statement]] skips the rest of the loop body and goes back to the top of the loop. For example:
x = 0
while x < 10:
x += 1
if x == 5:
print(x)
print('Done')
1.6 Text Strings
To define a string literal]], enclose it in single, double, or triple quotes as follows:
a = 'Hello World'
c = Computer says no.
d = “”“Computer still says no.”“”
The same type of quote used to start a string must be used to terminate it. Triple-quoted strings capture all the text until the terminating triple quote—as opposed to single-and double-quoted strings which must be specified on one logical line. Triple-quoted strings are useful when the contents of a string literal]] span multiple lines of text:
print(Content-type]]: text/html
<h1> Hello World </h1>
Click <a href=“http://www.python.org”>here</a>.
)
Immediately adjacent string literal]]s are concatenated into a single string. Thus, the above example could also be written as:
'\n'
'<h1> Hello World </h1>\n'
'Clock <a href=“http://www.python.org”>here</a>\n'
)
If the opening quotation mark of a string is prefaced by an f, escaped expressions within a string are evaluated. For example, in earlier examples, the following statement was used to output values of a calculation:
print(f'{year:>3d} {principal:0.2f}')
Although this is only using simple variable names, any valid expression can appear. For example:
…
print(f'{base_year + year:>4d} {principal:0.2f}')
As an alternative to f-strings, the format() method and % operator are also sometimes used to format strings. For example:
print('{0:>3d} {1:0.2f}'.format(year, principal))
print('%3d %0.2f' % (year, principal))
More information about string formatting is found in Chapter 9.
Strings are stored as sequences of Unicode characters indexed by integers, starting at zero. Negative indices index from the end of the string. The length of a string s is computed using len(s). To extract a single character, use the indexing operator s[i] where i is the index.
a = 'Hello World'
b = a[4] # b = 'o'
c = a[-1] # c = 'd'
To extract a substring, use the slicing operator s[i:j]. It extracts all characters from s whose index k is in the range i ⇐ k < j. If either index is omitted, the beginning or end of the string is assumed, respectively:
c = a[:5] # c = 'Hello'
d = a[6:] # d = 'World'
e = a[3:8] # e = 'lo Wo'
f = a[-5:] # f = 'World'
Strings have a variety of methods for manipulating their contents. For example, the replace() method performs a simple text replacement:
g = a.replace('Hello', 'Hello Cruel') # f = 'Hello Cruel World'
Table 1.6 shows a few common string methods. Here and elsewhere, arguments enclosed in square brackets are optional.
Table 1.6 Common String Methods

Strings are concatenated with the plus (+) operator:
g = a + 'ly' # g = 'Hello Worldly'
Python never implicitly interprets the contents of a string as numerical]] data. Thus, + always concatenates strings:
x = '37'
y = '42'
z = x + y # z = '3742' (String Concatenation)
To perform mathematical calculations, a string first has to be converted into a numeric value using a function such as int() or float(). For example:
z = int(x) + int(y) # z = 79 (Integer Addition)
Non-string values can be converted into a string representation by using the str(), repr(), or format() functions. Here’s an example:
s = 'The value of x is ' + str(x)
s = 'The value of x is ' + repr(x)
s = 'The value of x is ' + format(x, '4d')
Although str() and repr() both create strings, their output is often different. str() produces the output that you get when you use the print() function, whereas repr() creates a string that you type into a program to exactly represent the value of an object. For example:
print(str(s))
print(repr(s))
When debugging, use repr(s) to produce output because it shows you more information about a value and its type.
The format() function is used to convert a single value to a string with a specific formatting applied. For example:
x = 12.34567
format(x, '0.2f')
'12.35'
The format code given to format() is the same code you would use with f-strings when producing formatted output. For example, the above code could be replaced by the following:
f'{x:0.2f}'
'12.35'
The following program opens a file and reads its contents line by line as text strings:
with open('data.txt') as file:
print(line, end=) # end= omits the extra newline
The open() function returns a new file object. The with statement that precedes it declares a block of statements (or context) where the file (file) is going to be used. Once control leaves this block, the file is automatically closed. If you don’t use the with statement, the code would need to look like this:
print(line, end=) # end= omits the extra newline
It’s easy to forget the extra step of calling close() so it’s better to use the with statement and have the file closed for you.
The for [[loop iterates line-by-line over the file until no more data is available.
If you want to read the file in its entirety as a string, use the read() method like this:
with open('data.txt') as file:
If you want to read a large file in chunks, give a size hint to the read() method as follows:
with open('data.txt') as file:
while (chunk := file.read(10000)):
The := operator used in this example assigns to a variable and returns its value so that it can be tested by the while loop to break out. When the end of a file is reached, read() returns an empty string. An alternate way to write the above function is using break:
with open('data.txt') as file:
if not chunk:
To make the output of a program go to a file, supply a file argument to the print() function:
with open('out.txt', 'wt') as out:
- (1 + rate)
print(f'{year:>3d} {principal:0.2f}', file=out)
year += 1
In addition, file objects support a write() method that can be used to write string data. For example, the print() function in the previous example could have been written this way:
out.write(f'{year:3d} {principal:0.2f}\n')
By default, files contain text en[[coded]] as UTF-8. If you’re working with a different text encoding, use the extra encoding argument when opening the file. For example:
with open('data.txt', encoding='latin-1') as file:
Sometimes you might want to read data typed]] interactively in the console. To do that, use the input() function. For example:
name = input('Enter your name : ')
The input() function returns all of the typed text up to the terminating newline, which is not included.
1.8 Lists
Lists are an ordered collection of arbitrary objects. Create a list by enclosing values in square brackets:
names = [ 'Dave', 'Paula', 'Thomas', 'Lewis' ]
Lists are indexed by integers, starting with zero. Use the indexing operator to access and modify individual items of the list:
a = names[2] # Returns the third item of the list, 'Thomas'
names[2] = 'Tom' # Changes the third item to 'Tom'
print(names[-1]) # Print the last item ('Lewis')
To append new items to the end of a list, use the append() method:
To insert an item in the list at a specific position, use the insert() method:
To iterate over the items in a list, use a for [[loop:
You can extract or reassign a portion of a list by using the slicing operator:
b = names[0:2] # b → ['Dave', 'Paula']
c = names[2:] # c → ['Aya', 'Tom', 'Lewis', 'Alex']
names[1] = 'Becky' # Replaces 'Paula' with 'Becky'
names[0:2] = ['Dave', 'Mark', 'Jeff'] # Replace the first two items
- with ['Dave','Mark','Jeff']
Use the plus (+) operator to concatenate lists:
a = ['x','y'] + ['z','z','y'] # Result is ['x','y','z','z','y']
An empty list is created in one of two ways:
names = list() # An empty list
Specifying [] for an empty list is more idiomatic. list is the name of the class associated with the list type. It’s more common to see it used when performing conversions of data to a list. For example:
letters = list('Dave') # letters = ['D', 'a', 'v', 'e']
Most of the time, all of the items in a list are of the same type (for example, a list of numbers or a list of strings). However, lists can contain any mix of Python objects, including other lists, as in the following example:
a = [1, 'Dave', 3.14, ['Mark', 7, 9, [100, 101]], 10]
Items contained in nested lists are accessed by applying more than one indexing operation:
a[1] # Returns 'Dave'
a[3][2] # Returns 9
a[3][3][1] # Returns 101
The following program pcost.py illustrates how to read data into a list and perform a simple calculation. In this example, lines are assumed to contain comma-separated]] values. The program computes the sum of the product of two columns.
- For example:
- SYM,123,456.78
raise SystemExit(f'Usage: {sys.argv[0]} filename')
rows = []
with open(sys.argv[1], 'rt') as file:
- [
- ['SYM', '123', '456.78']
- …
- ]
total = sum(int(row[1]) * float(row[2]) for row in rows ])
print(f'Total cost: {total:0.2f}')
The first line of this program uses the import statement to load the sys module from the Python library. This module is used to obtain command-line arguments which are found in the list sys.argv. The initial check makes sure that a filename has been provided. If not, a SystemExit exception is raised with a helpful error message. In this message, sys.argv[0] inserts the name of the program that’s running.
The open() function uses the filename that was specified on the command line. The for line in file loop is reading the file line by line. Each line is split into a small list using the comma character as a de[[limiter. This list is appended to rows. The final result, rows, is a list of lists—remember that a list can contain anything including other lists.
The expression [ int(row[1]) * float(row[2]) for row in rows ] constructs a new list by looping over all of the lists in rows and computing the product of the second and third items. This useful technique for constructing a list is known as a list comprehension. The same computation could have been expressed more verbosely as follows:
values = []
values.append(int(row[1]) * float(row[2]))
As a general rule, list comprehensions are a preferred technique for performing simple calculations. The built-in sum() function computes the sum for all items in a sequence.
1.9 Tuples
To create simple data structures, you can pack a collection of values into an immutable object known as a tuple. You create a tuple by enclosing a group of values in parenthese]]s:
holding = ('GOOG', 100, 490.10)
address = ('www.python.org', 80)
For completeness, 0- and 1-element tuples can be defined, but have special syntax:
a = () # 0-tuple (empty tuple)
b = (item,) # 1-tuple (note the trailing comma)
The values in a tuple can be extracted by numerical]] index just like a list. However, it is more common to unpack tuples into a set of variables, like this:
Although tuples support most of the same operations as lists (such as indexing, slicing, and concatenation), the elements of a tuple cannot be changed after creation—that is, you cannot replace, delete, or append new elements to an existing tuple. A tuple is best viewed as a single immutable object that consists of several parts, not as a collection of distinct objects like a list.
Tuples and lists are often used together to represent data. For example, this program shows how you might read a file containing columns of data separated]] by commas:
portfolio = []
holding = (name, shares, price)
The resulting portfolio list created by this program looks like a two-dimensional array of rows and columns. Each row is represented by a tuple and can be accessed as follows:
portfolio[0]
('AA', 100, 32.2)
portfolio[1]
('IBM', 50, 91.1)
Individual items of data can be accessed like this:
portfolio[1][1]
50
portfolio[1][2]
91.1
Here’s how to loop over all of the records and unpack fields into a set of variables:
total = 0.0
for name, shares, price in portfolio:
Alternatively, you could use a list comprehension:
total = sum(shares * price for _, shares, price in portfolio])
When ite[[rating over tuples, the variable _ can be used to indicate a discarded value. In the above calculation, it means we’re ignoring the first item (the name).
1.10 Sets
A set is an un[[ordered collection of unique objects. Sets are used to find distinct values or to manage problems related to membership. To create a set, enclose a collection of values in curly]] braces or give an existing collection of items to set(). For example:
names1 = { 'IBM', 'MSFT', 'AA' }
names2 = set(['IBM', 'MSFT', 'HPE', 'IBM', 'CAT'])
The elements of a set are typically restricted to immutable objects. For example, you can make a set of numbers, strings, or tuples. However, you can’t make a set containing lists. Most common objects will probably work with a set, however—when in doubt, try it.
Unlike lists and tuples, sets are un[[ordered and cannot be indexed by numbers. Moreover, the elements of a set are never duplicated. For example, if you inspect the value of names2 from the preceding code, you get the following:
names2
Notice that 'IBM' only appears once. Also, the order of items can’t be predicted; the output may vary from what’s shown. The order might even change between interpreter runs on the same computer.
If working with existing data, you can also create a set using a set comprehension. For example, this statement turns all of the stock names from the data in the previous section into a set:
names = { s[0] for s in portfolio }
To create an empty set, use set() with no arguments:
r = set() # Initially empty set
Sets support a standard collection of operations including union, intersection, difference, and symmetric difference. Here’s an example:
a = t | s # Union {'MSFT', 'CAT', 'HPE', 'AA', 'IBM'}
b = t & s # Intersection {'IBM', 'MSFT'}
c = t - s # Difference { 'CAT', 'HPE' }
d = s - t # Difference { 'AA' }
e = t ^ s # Symmetric difference { 'CAT', 'HPE', 'AA' }
The difference operation s - t gives items in s that aren’t in t. The symmetric difference s ^ t gives items that are in either s or t but not in both.
New items can be added to a set using add() or update():
t.add('DIS') # Add a single item
s.update({'JJ', 'GE', 'ACME'}) # Adds multiple items to s
An item can be removed using remove() or discard():
t.remove('IBM') # Remove 'IBM' or raise KeyError if absent.
s.discard('SCOX') # Remove 'SCOX' if it exists.
The difference between remove() and discard() is that discard() doesn’t raise an exception if the item isn’t present.
1.11 Dictionaries
A dictionary is a mapping between keys and values. You create a dictionary by enclosing the key-value pairs, each separated]] by a colon, in curly]] braces ({ }), like this:
s = {
'name' : 'GOOG',
'shares' : 100,
'price' : 490.10
}
To access members of a dictionary, use the indexing operator as follows:
cost = s['shares'] * s['price']
Inserting]] or modifying]] objects works like this:
s['shares'] = 75
A dictionary is a useful way to define an object that consists of named fields. However, dictionaries are also commonly used as a mapping for performing fast lookups on un[[ordered data. For example, here’s a dictionary of stock prices:
prices = {
'GOOG' : 490.1,
'AAPL' : 123.5,
'IBM' : 91.5,
'MSFT' : 52.13
}
Given such a dictionary, you can look up a price:
Dictionary membership is tested with the in operator:
else:
p = 0.0
This particular sequence of steps can also be performed more compactly using the get() method:
p = prices.get('IBM', 0.0) # prices['IBM'] if it exists, else 0.0
Use the del statement to remove an element of a dictionary:
del prices['GOOG']
Although strings are the most common type of key, you can use many other Python objects, including numbers and tuples. For example, tuples are often used to construct composite or multipart keys:
prices = { }
prices[('IBM', '2015-02-03')] = 91.23
prices['IBM', '2015-02-04'] = 91.42 # Parens omitted
Any kind of object can be placed into a dictionary, including other dictionaries. However, mutable data structures such as lists, sets, and dictionaries cannot be used as keys.
Dictionaries are often used as building blocks for various algorithms and data-handling problems. One such problem is tabulation. For example, here’s how you could count the total number of shares for each stock name in earlier data:
portfolio = [
('ACME', 50, 92.34),
('IBM', 75, 102.25),
('PHP', 40, 74.50),
('IBM', 50, 124.75)
]
total_shares = { s[0]: 0 for s in portfolio }
for name, shares, _ in portfolio:
In this example, { s[0]: 0 for s in portfolio } is an example of a dictionary comprehension. It creates a dictionary of key-value pairs from another collection of data. In this case, it’s making an initial dictionary mapping stock names to 0. The for [[loop that follows iterates over the dictionary and adds up all of the held shares for each stock symbol.
Many common data processing tasks such as this one have already been implemented by library modules. For example, the collections module has a Counter object that can be used for this task:
from collections import Counter
for name, shares, _ in portfolio:
An empty dictionary is created in one of two ways:
prices = dict() # An empty dict
It is more idiomatic to use {} for an empty dictionary—although caution is required since it might look like you are trying to create an empty set (use set() instead). dict() is commonly used to create dictionaries from key-value values. For example:
pairs = [('IBM', 125), ('ACME', 50), ('PHP', 40)]
d = dict(pairs)
To obtain a list of dictionary keys, convert a dictionary to a list:
syms = list(prices) # syms = ['AAPL', 'MSFT', 'IBM', 'GOOG']
Alternatively, you can obtain the keys using dict.keys():
The difference between these two methods is that keys() returns a special “keys view” that is attached to the dictionary and actively reflects changes made to the dictionary. For example:
d = { 'x': 2, 'y':3 }
k = d.keys()
k
dict_keys(['x', 'y'])
d['z'] = 4
k
The keys always appear in the same order as the items were initially inserted into the dictionary. The list conversion above will preserve this order. This can be useful when dicts are used to represent key-value data read from files and other data sources. The dictionary will preserve the input order. This might help readability and debugging. It’s also nice if you want to write the data back to a file. Prior to Python 3.6, however, this ordering was not guaranteed, so you cannot rely upon it if compatibility with older versions of Python is required. Order is also not guaranteed if multiple deletions and insertions have taken place.
To obtain the values stored in a dictionary, use the dict.values() method. To obtain key-value pairs, use dict.items(). For example, here’s how to iterate over the entire contents of a dictionary as key-value pairs:
for sym, price in prices.items():
The most widely used looping construct is the for statement that iterates over a collection of items. One common form of iteration is to loop over all the members of a sequence—such as a string, list, or tuple. Here’s an example:
for n in [1, 2, 3, 4, 5, 6, 7, 8, 9]:
print(f'2 to the {n} power is {2**n}')
In this example, the variable n will be assigned successive items from the list [1, 2, 3, 4, …, 9] on each iteration. Since looping over ranges of integers is quite common, there is a shortcut:
for n in range(1, 10):
print(f'2 to the {n} power is {2**n}')
The range(i, j [,step) function creates an object that represents a range of integers with values from i up to, but not including, j. If the starting value is omitted, it’s taken to be zero. An optional stride can also be given as a third argument. Here are some examples:
a = range(5) # a = 0, 1, 2, 3, 4
b = range(1, 8) # b = 1, 2, 3, 4, 5, 6, 7
c = range(0, 14, 3) # c = 0, 3, 6, 9, 12
d = range(8, 1, -1) # d = 8, 7, 6, 5, 4, 3, 2
The object created by range() computes the values it represents on [[demand when lookups are requested. Thus, it’s efficient to use even with a large range of numbers.
The for statement is not limited to sequences of integers. It can be used to iterate over many kinds of objects including strings, lists, dictionaries, and files. Here’s an example:
message = 'Hello World'
for c in message:
names = ['Dave', 'Mark', 'Ann', 'Phil']
prices = { 'GOOG' : 490.10, 'IBM' : 91.50, 'AAPL' : 123.15 }
The for [[loop is one of Python’s most powerful language features because you can create custom iterator objects and generator functions that supply it with sequences of values. More details about iterators and generators can be found later in Chapter 6.
1.13 Functions
Use the def statement to define a function:
def remainder(a, b):
q = a // b # // is truncating division.
r = a - q * b
return r
To invoke a function, use its name followed by its arguments in parenthese]]s, for example result = remainder(37, 15).
It is common practice for a function to include a documentation string as the first statement. This string feeds the help() command and may be used by IDEs and other development tools to assist the programmer. For example:
def remainder(a, b):
Computes the remainder of dividing a by b
q = a // b
r = a - q * b
return r
If the inputs and outputs of a function aren’t clear from their names, they might be an[[notated with types:
def remainder(a: int, b: int) → int:
Computes the remainder of dividing a by b
q = a // b
r = a - q * b
return r
Such annotations are merely informational and are not actually enforced]] at runtime. Someone could still call the above function with non-integer values, such as result = remainder(37.5, 3.2).
Use a tuple to return multiple values from a function:
def divide(a, b):
q = a // b # If a and b are integers, q is integer
r = a - q * b
return (q, r)
When multiple values are returned in a tuple, they can be unpacked into separate variables like this:
quotient, remainder = divide(1456, 33)
To assign a default value to a function parameter, use assignment:
def connect(hostname, port, timeout=300):
…
When default values are given in a function definition, they can be omitted from subsequent function calls. An omitted argument will take on the supplied default value. Here’s an example:
connect('www.python.org', 80, 500)
Default arguments are often used for optional features. If there are many such arguments, readability can suffer. It’s therefore recommended to specify such arguments using keyword arguments. For example:
connect('www.python.org', 80, timeout=500)
If you know the names of the arguments, all of them can be named when calling a function. When named, the order in which they are listed doesn’t matter. For example, this is fine:
connect(port=80, hostname='www.python.org')
When variables are created or assigned inside a function, their scope is local. That is, the variable is only defined inside the body of the function and is destroyed when the function returns. Functions can also access variables defined outside of a function as long as they are defined in the same file. For example:
if debug:
…
Scoping rules are described in more detail in Chapter 5.
1.14 Exceptions
If an error occurs in your program, an exception is raised and a traceback message appears:
Traceback (most recent call last):
File “readport.py”, line 9, in <module>
ValueError: in[[valid literal for int() with base 10: 'N/A'
The traceback message indicates the type of error that occurred, along with its location. Normally, errors cause a program to terminate. However, you can catch and handle exceptions using try and except statements, like this:
portfolio = []
with open('portfolio.csv') as file:
try:
holding = (name, shares, price)
In this code, if a ValueError occurs, details concerning the cause of the error are placed in err and control passes to the code in the except block. If some other kind of exception is raised, the program crashes as usual. If no errors occur, the code in the except block is ignored. When an exception is handled, program execution resumes with the statement that immediately follows the last except block. The program does not return to the location where the exception occurred.
The raise statement is used to signal an exception. You need to give the name of an exception. For instance, here’s how to raise RuntimeError, a built-in exception:
raise RuntimeError('Computer says no')
Proper management of system resources such as locks, files, and network connections is often tricky when combined with exception [[handling. Sometimes there are actions that must be performed no matter what happens. For this, use try-finally]]. Here is an example involving a lock that must be released to avoid deadlock:
…
try:
…
…
finally]]:
To simplify such programming, most objects that involve resource management also support the with statement. Here is a modified version of the above code:
with lock:
…
…
In this example, the lock object is automatically acquired when the with statement executes. When execution leaves the context of the with block, the lock is automatically released. This is done regardless of what happens inside the with block. For example, if an exception occurs, the lock is released when control leaves the context of the block.
The with statement is normally only compatible with objects related to system resources or the execution environment—such as files, connections, and locks. However, user-defined objects can have their own custom processing, as described further in Chapter 3.
1.15 Program Termination
A program terminates when no more statements exist to execute in the input program or when an uncaught SystemExit exception is raised. If you want to force a program to quit, here’s how to do it:
raise SystemExit() # Exit with no error message
raise SystemExit(“Something is wrong”) # Exit with error
On exit, the interpreter makes a best effort to garbage-collect all active objects. However, if you need to perform a specific cleanup action (remove files, close a connection), you can register it with the atexit module as follows:
connection = open_connection(“deaddot.com”)
print “Going away…”
All values used in a program are objects. An object consists of internal data and methods that perform various kinds of operations involving that data. You have already used objects and methods when working with the built-in type]]s such as strings and lists. For example:
items = [37, 42] # Create a list object
items.append(73) # Call the append() method
The dir() function lists the methods available on an object. It is a useful tool for interactive experimentation when no fancy IDE is available. For example:
items = [37, 42]
['__add__', '__class__', '__contains__', '__delattr__', '__delitem__',
…
'append', 'count', 'extend', 'index', 'insert', 'pop',
When inspecting objects, you will see familiar methods such as append() and insert() listed. However, you will also see special methods whose names begin and end with a double underscore. These methods implement various operators. For example, the __add__() method is used to implement the + operator. These methods are explained in more detail in later chapters.
[37, 42, 73, 101]
The class statement is used to define new types of objects and for object-oriented programming]]. For example, the following class defines a stack with push() and pop() operations:
def __init__(self): # Initialize the stack
def __repr__(self):
return f'<{type(self).__name__} at 0x{id(self):x}, size={len(self)}>'
def __len__(self):
Inside the class definition, methods are defined using the def statement. The first argument in each method always refers to the object itself. By convention, self is the name used for this argument. All operations involving the attributes of an object must explicitly refer to the self variable. Methods with leading and trailing double underscores are special methods. For example, __init__ is used to initialize an object. In this case, __init__ creates an internal list for storing the stack data.
To use a class, write [[code such as this:
s.push('Dave') # Push some things onto it
s.push(42)
s.push([3, 4, 5])
Within the class, you will notice that the methods use an internal _items variable. Python does not have any mechanism for hiding or protecting data. However, there is a programming convention wherein names preceded by a single underscore are taken to be “private.” In this example, _items should be treated (by you) as internal implementation and not used outside the Stack class itself. Be aware that there is no actual enforcement of this convention—if you want to access _items, you can do so at any time. You’ll just have to answer to your coworkers when they review your code.
The __repr__() and __len__() methods are there to make the object play nicely with the rest of the environment. Here, __len__() makes a Stack work with the built-in len() function and __repr__() changes the way that a Stack is displayed and printed. It’s a good [[idea to always define __repr__() as it can simplify debugging.
s = Stack()
s.push('Dave')
s.push(42)
len(s)
2
s
<Stack at 0x10108c1d0, size=2>
A major feature of objects is that you can add to or redefine the capabilities of existing classes via inheritance.
Suppose you wanted to add a method to swap the top two items on the stack. You might write a class like this:
MyStack is identical to Stack except that it has a new method, swap().
s = MyStack()
s.push('Dave')
s.push(42)
s.swap()
s.pop()
'Dave'
s.pop()
42
Inheritance can also be used to change the behavior of an existing method. Suppose you want to restrict the stack to only hold numeric data]]. Write a class like this:
if not isinstance(item, (int, float)):
raise Type[[Error('Expected an int or float')
In this example, the push() method has been redefined to add extra checking. The super() operation is a way to invoke the prior definition of push(). Here’s how this class would work:
s.push(42)
s.push('Dave')
Traceback (most recent call last):
…
Type[[Error: Expected an int or float
Often, inheritance is not the best solution. Suppose you wanted to define a simple stack-based 4-function calculator that worked like this:
# calculate 2 + 3 * 4
calc = calculator()
calc.mul()
14
You might look at this code, see the use of push() and pop(), and think that calculator could be defined by inheriting from Stack. Although that would work, it is probably better to define calculator as a completely separate class:
self.push(self.pop() + self.pop())
def mul(self):
self.push(self.pop() * self.pop())
def div(self):
In this implementation, a calculator contains a Stack as an internal implementation detail. This is an example of composition. The push() and pop() methods delegate to the internal Stack. The main reason for taking this approach is that you don’t really think of the calculator as a Stack. It’s a separate concept—a different kind of object. By analogy, your phone contains a central processing unit (CPU) but you don’t usually think of your phone as a type of CPU.
1.17 Modules
As your programs grow in size, you will want to break them into multiple files for easier maintenance. To do this, use the import statement. To create a module, put the relevant statements and definitions into a file with a .py suf[[fix and the same name as the module. Here’s an example:
portfolio = []
try:
holding = (name, shares, price)
To use your module in other files, use the import statement. For example, here is a module pcost.py that uses the above read_portfolio() function:
Compute the total shares*price of a portfolio
port = readport.read_portfolio(filename)
return sum(shares * price for _, shares, price in port)
The import statement creates a new namespace (or environment) and executes all the statements in the associated .py file within that namespace. To access the contents of the namespace after import, use the name of the module as a pre[[fix, as in readport.read_portfolio() in the preceding example.
If the import statement fails with an ImportError exception, you need to check a few things in your environment. First, make sure you created a file called readport.py. Next, check the directories listed on sys.path. If your file isn’t saved in one of those directories, Python won’t be able to find it.
If you want to import a module under a different name, supply the import statement with an optional as qualifier:
port = rp.read_portfolio('portfolio.dat')
To import specific definitions into the current namespace, use the from statement:
from readport import read_portfolio
port = read_portfolio('portfolio.dat')
As with objects, the dir() function lists the contents of a module. It is a useful tool for interactive experimentation.
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__',
'__name__', '__package__', '__spec__', 'read_portfolio']
…
Python provides a large standard library of modules that simplify certain programming tasks. For example, the csv module is a standard library for dealing with files of comma-separated]] values. You could use it in your program as follows:
portfolio = []
try:
holding = (name, shares, price)
Python also has a vast number of third-party modules that can be installed to solve almost any imaginable task (including the reading of CSV files). See https://pypi.org.
1.18 Script Writing
Any file can execute either as a script or as a library imported with import. To better support imports, script code is often enclosed with a conditional check against the module name:
…
def main():
portfolio = read_portfolio('portfolio.csv')
for name, shares, price in portfolio:
print(f'{name:>10s} {shares:10d} {price:10.2f}')
main()
__name__ is a built-in variable that always contains the name of the enclosing module. If a program is run as the main script with a command such as python readport.py, the __name__ variable is set to '__main__'. Otherwise, if the code is imported using a statement such as import readport, the __name__ variable is set to 'readport'.
As shown, the program is hardcoded]] to use a filename 'portfolio.csv'. Instead, you may want to prompt the user for a filename or accept the filename as a command-line argument. To do this, use the built-in input() function or the sys.argv list. For example, here is a modified version of the main() function:
def main(argv):
if len(argv) == 1:
filename = input('Enter filename: ')
elif len(argv) == 2:
filename = argv[1]
else:
raise SystemExit(f'Usage: {argv[0]} [ filename ]')
portfolio = read_portfolio(filename)
for name, shares, price in portfolio:
print(f'{name:>10s} {shares:10d} {price:10.2f}')
This program can be run in two different ways from the command line:
…
bash % python readport.py portfolio.csv
…
bash % python readport.py a b c
Usage: readport.py [ filename ]
bash %
For very simple programs, it is often enough to process arguments in sys.argv as shown. For more advanced usage, the argparse standard library module can be used.
1.19 Packages
In large programs, it’s common to organize code into packages. A package is a hierarchical collection of modules. On the file[[system, put your code as a collection of files in a directory like this:
tutorial/
…
The directory should have an __init__.py file, which may be empty. Once you’ve done this, you should be able to make nested import statements. For example:
port = tutorial.readport.read_portfolio('portfolio.dat')
If you don’t like the long names, you can shorten things using an import like this:
from tutorial.readport import read_portfolio
port = read_portfolio('portfolio.dat')
One tricky issue with packages is imports between files within the same package. In an earlier example, a pcost.py module was shown that started with an import like this:
…
If the pcost.py and readport.py files are moved into a package, this import statement breaks. To fix it, you must use a fully qualified module import:
…
Alternatively, you can use a package-relative import like this:
…
The latter form has the benefit of not hardcoding the package name. This makes it easier to later rename a package or move it around within your project.
Other subtle details concerning packages are covered later (see Chapter 8).
1.20 Structuring an Application
As you start to write more Python code, you may find yourself working on larger applications that include a mix of your own code as well as third-party dependencies. Managing all of this is a complex topic that continues to evolve. There are also many conflicting opinions about what constitutes “best practice.” However, there are a few essential facets to it that you should know.
First, it is standard practice to organize large code bases into packages (that is, directories of .py files that include the special __init__.py file). When doing this, pick a unique package name for the top-[[level directory name. The primary purpose of the package directory is to manage import statements and the namespaces of modules used while programming. You want your code isolated from everyone else’s code.
In addition to your main project source [[code, you might additional]]ly have tests, examples, scripts, and documentation. This additional]] material usually lives in a separate set of directories than the package containing your source [[code. Thus, it is common to create an enclosing top-[[level directory for your project and to put all of your work under that. For example, a fairly typical project organization might look like this:
tutorial/
…
tests/
…
examples/
…
doc/
…
Keep in mind that there is more than one way to do it. The nature of the problem you’re solving might dictate a different structure. Nevertheless, as long as your main set of source [[code files lives in a proper package (again, the directory with the __init__.py file), you should be fine.
1.21 Managing Third-Party Packages
Python has a large library of contributed packages that can be found at the Python Package Index (https://pypi.org). You may need to depend on some of these packages in your own code. To install a third-party package, use a command such as pip:
bash % python3 -m pip install somepackage
Installed packages are placed into a special site-packages directory that you can find if you inspect the value of sys.path. For example, on a UNIX machine, packages might be placed in /usr/local/lib/python3.8/site-packages. If you are ever left wondering where a package is coming from, inspect the __file__ attribute of a package after importing it in the interpreter:
'/usr/local/lib/python3.8/site-packages/pandas/__init__.py'
One potential problem with installing a package is that you might not have permission to change the locally installed version of Python. Even if you had permission, it still might not be a good [[idea. For example, many systems already have Python installed for use by various system utilities. Altering the installation of that version of Python is often a bad idea.
To make a sandbox where you can install packages and work without worrying about breaking anything, create a virtual environment]] by a command like this:
bash % python3 -m venv myproject
This will set up a dedicated Python installation for you in a directory called myproject/. Within that directory, you’ll find an interpreter executable and library where you can safely install packages. For example, if you run myproject/bin/python3, you’ll get an interpreter configured for your personal]] use. You can install packages into this interpreter without worrying about breaking any part of the default Python installation. To install a package, use pip as before but make sure to specify the correct interpreter:
bash % ./myproject/bin/python3 -m pip install somepackage
There are various tools that aim to simplify the use of pip and venv. Such matters might also be handled automagically by your IDE. As this is a fluid and ever-evolving part of Python, no further advice is given here.
1.22 Python: It Fits Your Brain
In the early days of Python, “it fits your brain” was a common motto. Even today, the core of Python is a small programming language along with a useful collection of built-in objects—lists, sets, and dictionaries. A vast array of practical problems can be solved using nothing more than the basic features presented in this chapter. This is a good thing to keep in mind as you begin your Python adventure—although there are always more complicated ways to solve a problem, there might also be a simple way to do it with the basic features Python already provides. When in doubt, you’ll probably thank your past self for doing just that.
Fair Use Source
© 1994 - 2024 Cloud Monk Losang Jinpa or Fair Use. Disclaimers
SYI LU SENG E MU CHYWE YE. NAN. WEI LA YE. WEI LA YE. SA WA HE.