Python File Handling Pitfalls Cheat Sheet

When writing security scripts that ingest logs, configs, signatures, or binary payloads, small file-handling mistakes can silently break detection or produce false negatives. This cheat sheet covers the most common mistakes and the safe patterns to use instead.

1. Use with open(...) so files close automatically

with open("access.log", "r", encoding="utf-8") as f:
    data = f.read()

• Always specify encoding for text files. The default is platform-dependent.
• For binary files use "rb"/"wb".

2. Iterate over the file handle, don't use .readlines() for big logs

with open("access.log", "r", encoding="utf-8") as f:
    for line in f:
        process(line.rstrip("\n"))

• readlines() loads the entire file into memory.
• Streaming is safer for large logs and avoids crashes on huge files.

3. Open files as raw bytes only when you need the exact bytes

with open("shellcode.bin", "rb") as f:
    blob = f.read()

• Use binary mode for malware samples, firmware, images, shellcode, or PCAPs.
• Never try to decode() arbitrary binary data.

4. Strip trailing newlines from signature and word lists

with open("payload_list.txt", "r", encoding="utf-8") as f:
    signatures = [line.strip() for line in f if line.strip()]

• readlines() keeps \n, so exact matches like path in signature_list fail.
• Skip empty lines to avoid false positives.

5. Use pathlib for directory creation

from pathlib import Path

report_dir = Path("report")
report_dir.mkdir(exist_ok=True)
(report_dir / "top_ips.json").write_text(json.dumps(counter, indent=2))

• mkdir("report") raises FileExistsError if the directory already exists.
• exist_ok=True makes the call idempotent.

6. Prefer "w" over "x" for reports unless you explicitly want fail-if-exists

with open("report/top_ips.json", "w", encoding="utf-8") as f:
    json.dump(counter, f, indent=2)

• "x" raises FileExistsError if the file already exists. That is usually not what you want for generated reports.
• Use "x" only when accidental overwrites must be prevented.

7. Check exists() before reading or skip gracefully

from pathlib import Path

path = Path(filename)
if not path.exists():
    print(f"[error] {path} does not exist", file=sys.stderr)
    sys.exit(1)

• Don't let a missing file crash the script with an ugly traceback.
• Report errors to stderr with file=sys.stderr.

8. Avoid print(..., sys.stderr) as if it were a second argument

# WRONG: prints the tuple ("message", <stderr>)
print("error", sys.stderr)

# RIGHT
print("error", file=sys.stderr)

9. Use sys.exit(code) in scripts, not exit()

import sys

sys.exit(1)

• exit() is a REPL helper and may not be available in all environments.

10. Validate parsed data before indexing it

parts = request_line.split()
if len(parts) != 3:
    print(f"[warn] malformed request line: {request_line!r}", file=sys.stderr)
    continue
method, uri, version = parts

• Logs can be corrupted or attacker-controlled. Never assume fields are present.

Quick Reference Table

Related Concepts

• Defensive Input Handling for Security Scripts
• Python Essentials for Security Scripting
• Regex Fundamentals