Python Notes

Python file handling: open, read, write, with, CSV, zip and pickle

Himanshu Sharma · 3 March 2023 · 10 min read

Variables vanish when a program ends. Files are how data survives: a list of contacts, a report you export, a configuration file, a log. Python makes file work short, but a few details (modes, encodings, closing the file) decide whether your code is reliable. This post covers text and binary files, opening and closing, reading and writing, the with statement, CSV files, zip archives and pickling. All the examples were run in an empty folder on Python 3.9.

Types of files

Computers store two kinds of file content:

Text files hold characters: .txt, .csv, .json, .py, .html. You can open them in a text editor. Python reads them as str.
Binary files hold raw bytes: images, PDFs, audio, zip archives, executables. Python reads them as bytes.

Choosing the right mode when you open a file tells Python which one you expect.

Opening and closing a file

open(name, mode) returns a file object. The most important modes are:

Mode	Meaning	If the file does not exist	Existing content
`"r"` (default)	read text	`FileNotFoundError`	untouched
`"w"`	write text	created	erased
`"a"`	append text	created	kept; new text goes at the end
`"x"`	create, fail if it exists	created	`FileExistsError` if present
`"r+"`	read and write	`FileNotFoundError`	kept

Add b for binary: "rb", "wb", "ab". Always close what you open, because buffered text may not reach the disk until the file is closed.

f = open("notes.txt", "w", encoding="utf-8")
print(f.name, f.mode, f.closed, f.encoding)       # notes.txt w False utf-8
f.write("first line\n")
f.close()
print(f.closed)                                    # True

name, mode, closed and encoding are properties of the file object; readable(), writable() and seekable() report what it can do. Always pass encoding="utf-8" for text. Without it Python uses a default that differs between computers, which makes the same code produce different files on Windows and Linux.

Writing text

write() writes one string and returns the number of characters written. It does not add a newline. writelines() writes a list of strings, also without newlines.

with open("notes.txt", "w", encoding="utf-8") as f:
    n = f.write("alpha\n")
    f.writelines(["beta\n", "gamma\n"])
    print("wrote", n, "chars", file=f)       # print can write to a file too

Opening in "w" mode a second time would erase everything. To add to the end, use "a":

with open("notes.txt", "a", encoding="utf-8") as f:
    f.write("delta\n")

The `with` statement

You may have noticed with open(...) as f: above. A with block closes the file automatically when the block ends, even if an error happens inside it. It replaces the try/finally you would otherwise need, so use it every time.

with open("notes.txt", encoding="utf-8") as f:
    print(f.closed)         # False
print(f.closed)             # True

Reading text

You can read in several ways:

with open("notes.txt", encoding="utf-8") as f:
    print(repr(f.read(5)))        # 'alpha'
    print(repr(f.readline()))     # '\n'
    print(f.readlines())          # ['beta\n', 'gamma\n', 'wrote 6 chars\n', 'delta\n']

with open("notes.txt", encoding="utf-8") as f:
    whole = f.read()
print(len(whole), whole.count("\n"))            # 37 5

Method	Returns
`read()`	the whole rest of the file as one string
`read(n)`	the next `n` characters
`readline()`	the next line, including its `\n`
`readlines()`	a list of all remaining lines

For large files do not use read() or readlines(), because they load everything into memory. A file object is iterable, so a plain loop reads one line at a time:

with open("notes.txt", encoding="utf-8") as f:
    for number, line in enumerate(f, start=1):
        print(number, line.rstrip("\n"))

1 alpha
2 beta
3 gamma
4 wrote 6 chars
5 delta

The rstrip("\n") removes the newline that each line carries; otherwise print would add a second one.

Moving around: `tell()` and `seek()`

A file has a current position. tell() reports it and seek(n) moves it, which lets you read a file twice:

with open("notes.txt", encoding="utf-8") as f:
    f.read(6)
    print(f.tell())            # 6
    f.seek(0)
    print(f.readline(), end="")    # alpha

Handling errors

Opening can fail, and code that matters should say what happens:

try:
    open("missing.txt", encoding="utf-8")
except FileNotFoundError as e:
    print(e)        # [Errno 2] No such file or directory: 'missing.txt'

PermissionError happens when you may not read or write the file, and IsADirectoryError when the name is a folder. The exception handling post shows how to structure this.

Checking files and folders

The os.path functions and the newer pathlib module tell you what exists:

import os
from pathlib import Path
print(os.path.exists("notes.txt"), os.path.isfile("notes.txt"), os.path.getsize("notes.txt"))
# True True 37
p = Path("notes.txt")
print(p.suffix, p.stem, p.read_text(encoding="utf-8").splitlines()[0])      # .txt notes alpha
Path("report.txt").write_text("quick write\n", encoding="utf-8")
print(sorted(x.name for x in Path(".").glob("*.txt")))      # ['notes.txt', 'report.txt']

pathlib builds paths in a way that works on every operating system and gives one-line read_text and write_text. os.remove(name) deletes a file, os.rename(old, new) renames it, and os.listdir(".") lists a folder.

Binary files

Open with "rb" or "wb" and you work with bytes. This is how you copy any file exactly, such as an image:

data = bytes([0, 1, 2, 250, 255])
with open("blob.bin", "wb") as f:
    print(f.write(data))                 # 5
with open("blob.bin", "rb") as f:
    back = f.read()
print(back, list(back), back == data)    # b'\x00\x01\x02\xfa\xff' [0, 1, 2, 250, 255] True

Copying a large binary file in chunks keeps memory use small:

with open("blob.bin", "rb") as src, open("blob_copy.bin", "wb") as dst:
    while chunk := src.read(2):          # 2 bytes here; use 64 KB or more in real code
        dst.write(chunk)
print(open("blob_copy.bin", "rb").read() == data)      # True

(The := operator needs Python 3.8 or newer. The standard library also offers shutil.copyfile(src, dst) for this job.) To change an image, use a library like Pillow; to compress one in the browser, try the Image Compressor.

CSV files

CSV (comma-separated values) is the simplest table format, and spreadsheets open it. Do not split lines on commas yourself, because quoted values can contain commas. Use the csv module. Open CSV files with newline="":

import csv
rows = [["name", "city", "score"], ["Asha", "Pune", 88], ["Ravi", "Delhi, NCR", 75]]
with open("scores.csv", "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows(rows)
print(open("scores.csv", encoding="utf-8").read())

name,city,score
Asha,Pune,88
Ravi,"Delhi, NCR",75

The writer put quotes around Delhi, NCR automatically, and it ends each row with \r\n, which is why the file is a few bytes larger than the visible text. Read it back as lists, or as dictionaries keyed by the header row:

with open("scores.csv", newline="", encoding="utf-8") as f:
    for row in csv.reader(f):
        print(row)
with open("scores.csv", newline="", encoding="utf-8") as f:
    people = list(csv.DictReader(f))
print(people[1]["city"], sum(int(p["score"]) for p in people))      # Delhi, NCR 163

['name', 'city', 'score']
['Asha', 'Pune', '88']
['Ravi', 'Delhi, NCR', '75']

Every value comes back as a string, so convert numbers with int() or float(). The JSON section below shows the same data as JSON.

Zipping and unzipping

The zipfile module creates and reads .zip archives:

import zipfile
with zipfile.ZipFile("backup.zip", "w", zipfile.ZIP_DEFLATED) as z:
    z.write("notes.txt")
    z.write("scores.csv")
with zipfile.ZipFile("backup.zip") as z:
    print(z.namelist())                       # ['notes.txt', 'scores.csv']
    print([(i.filename, i.file_size) for i in z.infolist()])
    z.extract("scores.csv", "restored")      # unzip one file into a folder
    z.extractall("restored_all")             # or everything
print(sorted(os.listdir("restored")))         # ['scores.csv']

[('notes.txt', 37), ('scores.csv', 53)]

Be careful with extractall on archives from strangers: a malicious archive can contain paths that try to write outside the target folder. Only unzip files you trust.

Pickling and unpickling

Pickling turns a Python object (a list, dictionary, even your own class instance) into bytes that can be saved in a file; unpickling rebuilds the object.

import pickle
data = {"name": "Asha", "scores": [88, 92], "tags": {"a", "b"}}
with open("data.pkl", "wb") as f:
    pickle.dump(data, f)
with open("data.pkl", "rb") as f:
    again = pickle.load(f)
print(again == data, again)

The output was True followed by the dictionary; a set prints in an arbitrary order, so its exact appearance can differ between runs. Pickle is convenient but unsafe: loading a pickle from an untrusted source can run arbitrary code. Never unpickle data you did not create, and prefer json for data you share. JSON works for dictionaries, lists, strings and numbers and is readable in any language:

import json
payload = {"name": "Asha", "scores": [88, 92]}
text = json.dumps(payload, indent=2)
print(text)
print(json.loads(text) == payload)

{
  "name": "Asha",
  "scores": [
    88,
    92
  ]
}
True

The JSON Formatter shows how to inspect and validate such text without code.

Common mistakes

Opening a file with "w" to add data, which erases it. Use "a".
Forgetting to close a file instead of using with.
Leaving out encoding="utf-8" and getting garbled characters on another machine.
Using read() on a huge file instead of looping over it.
Forgetting that lines keep their \n.
Splitting CSV text by commas instead of using csv.
Opening a CSV on Windows without newline="", which creates blank lines.
Unpickling files from untrusted sources.

Practice

Write a program that reads a text file and prints the number of lines, words and characters, then writes the ten most common words to a CSV file. Add error handling for a missing file. Finally, add the original and the CSV to a zip archive and list its contents.

Continue

Files raise errors often, so read the exception handling post next. The input and output post shows console input and print(..., file=f), and the logging post writes messages to log files. Practise in the Python video course.