Python Development Guide

Table of Contents

Environment Setup

Virtual Environments

Why Virtual Environments? - ✅ Isolate project dependencies - ✅ Prevent package version conflicts - ✅ Reproducible development environments - ✅ Easy sharing with requirements.txt

Creating and Using Virtual Environments

# Create virtual environment
python3 -m venv myproject

# Activate virtual environment
# Linux/Mac:
source myproject/bin/activate

# Windows:
myproject\Scripts\activate

# Your prompt changes to show active environment:
(myproject) user@computer:~$

# Deactivate when done
deactivate

Managing Packages

# Install single package
pip install requests

# Install specific version
pip install requests==2.28.0

# Install from requirements file
pip install -r requirements.txt

# Upgrade package
pip install --upgrade requests

# Uninstall package
pip uninstall requests

# List installed packages
pip list

# Show package details
pip show requests

# Create requirements file (save current packages)
pip freeze > requirements.txt

Example requirements.txt

requests==2.31.0
numpy==1.24.3
pandas==2.0.2
flask==2.3.2

Python Version Management

Understanding the Tools

Tool Purpose Manages Example Use
pyenv Python version manager Python interpreters Switch between Python 3.9, 3.10, 3.11
venv Virtual environment Package dependencies Isolate project packages

Key Difference: - pyenv manages which Python you're using - venv manages which packages are installed

pyenv - Python Version Management

Installation:

# Linux/Mac
curl https://pyenv.run | bash

# Or via package manager (Mac)
brew install pyenv

Basic Usage:

# List available Python versions
pyenv install --list

# Install specific Python version
pyenv install 3.11.0
pyenv install 3.10.5
pyenv install 3.9.13

# List installed versions
pyenv versions

# Set global default version (system-wide)
pyenv global 3.11.0

# Set local version (current directory only)
pyenv local 3.10.5

# Set version for current shell session
pyenv shell 3.9.13

# Check current Python version
python --version

Common Workflow:

# Install Python version for project
pyenv install 3.11.0

# Create project directory
mkdir myproject
cd myproject

# Set Python version for this project
pyenv local 3.11.0

# Create virtual environment with this version
python -m venv venv

# Activate virtual environment
source venv/bin/activate

# Now you have Python 3.11.0 with isolated packages!

Combined Workflow Example

Scenario: Starting new project with Python 3.11 and specific packages

# 1. Install Python 3.11 (if not already)
pyenv install 3.11.0

# 2. Create project directory
mkdir data-analysis-project
cd data-analysis-project

# 3. Set Python version for this project
pyenv local 3.11.0

# 4. Create virtual environment
python -m venv venv

# 5. Activate virtual environment
source venv/bin/activate  # Linux/Mac
# venv\Scripts\activate   # Windows

# 6. Install packages
pip install pandas numpy matplotlib

# 7. Save dependencies
pip freeze > requirements.txt

# 8. Work on project...
python main.py

# 9. Deactivate when done
deactivate

Sharing Project:

# Other developers can recreate environment:
pyenv install 3.11.0
pyenv local 3.11.0
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Core Concepts

Decorators

What is a Decorator?

Literal meaning: To enhance the appearance of something by adding ornamental elements - Painting a building - Adding lights and screens to an event - Decorating a room

In Python: A function that takes another function as an argument and returns a new function with enhanced functionality.

Basic Decorator Example

def my_decorator(func):
    """Decorator that adds functionality before and after function execution."""
    def wrapper():
        print("Something before function")
        func()  # Call original function
        print("Something after function")
    return wrapper

# Using decorator with @ syntax
@my_decorator
def say_hello():
    print("Hello!")

# Calling decorated function
say_hello()

# Output:
# Something before function
# Hello!
# Something after function

Decorator with Arguments

def repeat(times):
    """Decorator that repeats function execution."""
    def decorator(func):
        def wrapper(*args, **kwargs):
            for _ in range(times):
                result = func(*args, **kwargs)
            return result
        return wrapper
    return decorator

@repeat(times=3)
def greet(name):
    print(f"Hello, {name}!")

greet("Alice")

# Output:
# Hello, Alice!
# Hello, Alice!
# Hello, Alice!

Practical Decorators

Timing Decorator:

import time
from functools import wraps

def timer(func):
    """Measure function execution time."""
    @wraps(func)  # Preserves original function metadata
    def wrapper(*args, **kwargs):
        start = time.time()
        result = func(*args, **kwargs)
        end = time.time()
        print(f"{func.__name__} took {end - start:.4f} seconds")
        return result
    return wrapper

@timer
def slow_function():
    time.sleep(2)
    return "Done"

result = slow_function()
# Output: slow_function took 2.0012 seconds

Logging Decorator:

def log_calls(func):
    """Log function calls with arguments."""
    @wraps(func)
    def wrapper(*args, **kwargs):
        print(f"Calling {func.__name__} with args={args}, kwargs={kwargs}")
        result = func(*args, **kwargs)
        print(f"{func.__name__} returned {result}")
        return result
    return wrapper

@log_calls
def add(a, b):
    return a + b

result = add(3, 5)
# Output:
# Calling add with args=(3, 5), kwargs={}
# add returned 8

Authentication Decorator (Web frameworks):

def require_auth(func):
    """Check if user is authenticated before executing."""
    @wraps(func)
    def wrapper(*args, **kwargs):
        if not user_is_authenticated():
            raise PermissionError("Authentication required")
        return func(*args, **kwargs)
    return wrapper

@require_auth
def view_profile():
    return "User profile data"

Multiple Decorators

@decorator1
@decorator2
@decorator3
def my_function():
    pass

# Equivalent to:
my_function = decorator1(decorator2(decorator3(my_function)))

Example with Multiple Decorators:

@timer
@log_calls
def calculate(x, y):
    return x ** y

result = calculate(2, 10)
# First logs call, then times execution

File and Directory Operations

pathlib - Modern File Handling

pathlib is the modern, object-oriented approach to file operations (Python 3.4+).

Why pathlib over os.path? - ✅ More intuitive object-oriented API - ✅ Chain operations easily - ✅ Works consistently across OS (Windows, Linux, Mac) - ✅ Built-in methods for common tasks

Basic Path Operations

from pathlib import Path

# Create paths
file_path = Path("data/file.txt")
dir_path = Path("data/images")
current_dir = Path(".")
home_dir = Path.home()

# Get absolute path
abs_path = file_path.absolute()

# Get parent directory
parent = file_path.parent  # Path("data")

# Get filename
name = file_path.name  # "file.txt"
stem = file_path.stem  # "file"
suffix = file_path.suffix  # ".txt"

# Join paths
config_path = Path("config") / "settings.json"
# Same as: Path("config/settings.json")

Checking Existence and Type

# Check if exists
if file_path.exists():
    print("Path exists")

# Check if file
if file_path.is_file():
    print("Is a file")

# Check if directory
if dir_path.is_dir():
    print("Is a directory")

# Check if absolute path
if file_path.is_absolute():
    print("Is absolute")

Creating Directories

# Create single directory
dir_path = Path("new_folder")
dir_path.mkdir()

# Create nested directories
dir_path = Path("data/images/2024")
dir_path.mkdir(parents=True, exist_ok=True)
# parents=True: Create parent directories if needed
# exist_ok=True: Don't raise error if already exists

# Example: Create project structure
project = Path("myproject")
(project / "src").mkdir(parents=True, exist_ok=True)
(project / "tests").mkdir(parents=True, exist_ok=True)
(project / "docs").mkdir(parents=True, exist_ok=True)

Reading and Writing Files

# Read entire file as string
file_path = Path("data.txt")
content = file_path.read_text()

# Read as bytes
binary_content = file_path.read_bytes()

# Write string to file
file_path.write_text("Hello, World!")

# Write bytes to file
file_path.write_bytes(b"Binary data")

# Read lines into list
lines = file_path.read_text().splitlines()

# Example: Process each line
for line in file_path.read_text().splitlines():
    print(line.strip())

Listing Files and Directories

# List all items in directory
dir_path = Path("data")
for item in dir_path.iterdir():
    print(item)

# List only files
for file in dir_path.iterdir():
    if file.is_file():
        print(file.name)

# List files matching pattern (glob)
for file in dir_path.glob("*.txt"):
    print(file)

# Recursive search
for file in dir_path.rglob("*.py"):  # Find all .py files recursively
    print(file)

# List with specific extensions
image_files = list(dir_path.glob("*.jpg"))
python_files = list(dir_path.rglob("*.py"))

File Operations

# Copy file (requires shutil)
import shutil
source = Path("file.txt")
destination = Path("backup/file.txt")
shutil.copy(source, destination)

# Move/rename file
source.rename(destination)

# Delete file
file_path.unlink()  # Delete file
file_path.unlink(missing_ok=True)  # Don't error if doesn't exist

# Delete directory
dir_path.rmdir()  # Only if empty

# Delete directory with contents
import shutil
shutil.rmtree(dir_path)

Practical Examples

Example 1: Process all CSV files:

from pathlib import Path
import pandas as pd

data_dir = Path("data")

for csv_file in data_dir.glob("*.csv"):
    print(f"Processing {csv_file.name}")
    df = pd.read_csv(csv_file)
    # Process dataframe...

    # Save to new directory
    output_dir = Path("processed")
    output_dir.mkdir(exist_ok=True)
    output_file = output_dir / f"processed_{csv_file.name}"
    df.to_csv(output_file, index=False)

Example 2: Organize files by extension:

from pathlib import Path
import shutil

source_dir = Path("downloads")

for file in source_dir.iterdir():
    if file.is_file():
        # Get file extension
        ext = file.suffix.lower()

        # Create directory for this extension
        target_dir = source_dir / ext[1:]  # Remove leading dot
        target_dir.mkdir(exist_ok=True)

        # Move file
        shutil.move(file, target_dir / file.name)
        print(f"Moved {file.name} to {target_dir}")

Example 3: Find largest files:

from pathlib import Path

def find_largest_files(directory, n=10):
    """Find n largest files in directory."""
    files = [f for f in Path(directory).rglob("*") if f.is_file()]

    # Sort by size
    sorted_files = sorted(files, key=lambda f: f.stat().st_size, reverse=True)

    for file in sorted_files[:n]:
        size_mb = file.stat().st_size / (1024 * 1024)
        print(f"{file.name}: {size_mb:.2f} MB")

find_largest_files(".")

Web Application Frameworks

Library Best For Complexity Learning Curve When to Use
Streamlit Quick interactive dashboards Low 1-2 hours Data apps, ML demos, internal tools
Gradio ML model demos and sharing Low 1-2 hours Share ML models, quick prototypes
Flask Multi-page web apps, APIs Medium 1-2 days Small to medium web apps, REST APIs
FastAPI High-performance APIs Medium 2-3 days Modern APIs, microservices, async apps
Django Full-featured web applications High 1-2 weeks Large apps, admin panels, e-commerce

Detailed Comparison

Streamlit:

import streamlit as st

st.title("My Dashboard")
name = st.text_input("Enter your name")
st.write(f"Hello, {name}!")

# Run with: streamlit run app.py
  • ✅ Zero HTML/CSS needed
  • ✅ Perfect for data science
  • ❌ Limited customization

Gradio:

import gradio as gr

def greet(name):
    return f"Hello, {name}!"

demo = gr.Interface(fn=greet, inputs="text", outputs="text")
demo.launch()
  • ✅ Great for ML models
  • ✅ Easy sharing
  • ❌ Less flexible than web frameworks

Flask:

from flask import Flask, render_template

app = Flask(__name__)

@app.route('/')
def home():
    return render_template('index.html')

if __name__ == '__main__':
    app.run(debug=True)
  • ✅ Lightweight and flexible
  • ✅ Good for REST APIs
  • ❌ Requires more setup

FastAPI:

from fastapi import FastAPI

app = FastAPI()

@app.get("/items/{item_id}")
async def read_item(item_id: int):
    return {"item_id": item_id}

# Run with: uvicorn main:app --reload
  • ✅ Automatic API documentation
  • ✅ Type hints and validation
  • ✅ High performance

Django:

# Full MVC framework with ORM, admin panel, auth
# Best for large, database-driven applications
  • ✅ Batteries included (auth, admin, ORM)
  • ✅ Scalable for large apps
  • ❌ Steeper learning curve

OCR Libraries

OCR = Optical Character Recognition (extracting text from images)

Library Size Accuracy Speed Language Support Best For
pytesseract Light (~60MB) Medium Fast 100+ languages Simple documents, quick prototypes
EasyOCR Medium (~500MB) Good Medium 80+ languages General purpose, good balance
PaddleOCR Heavy (~1GB+) Best Slower 80+ languages High accuracy needs, production

Installation and Usage

pytesseract (Tesseract wrapper):

# Install tesseract engine first
# Ubuntu:
sudo apt install tesseract-ocr

# Mac:
brew install tesseract

# Windows: Download from GitHub

# Install Python package
pip install pytesseract pillow

import pytesseract
from PIL import Image

# Read image
image = Image.open('document.png')

# Extract text
text = pytesseract.image_to_string(image)
print(text)

# With language specification
text = pytesseract.image_to_string(image, lang='eng+fra')  # English + French

EasyOCR:

pip install easyocr

import easyocr

# Create reader (downloads model on first use)
reader = easyocr.Reader(['en'])  # English

# Read text from image
results = reader.readtext('document.png')

# Print detected text
for (bbox, text, confidence) in results:
    print(f"Text: {text}, Confidence: {confidence:.2f}")

PaddleOCR:

pip install paddlepaddle paddleocr

from paddleocr import PaddleOCR

# Initialize OCR
ocr = PaddleOCR(use_angle_cls=True, lang='en')

# Perform OCR
result = ocr.ocr('document.png')

# Print results
for line in result[0]:
    print(f"Text: {line[1][0]}, Confidence: {line[1][1]:.2f}")

Choosing an OCR Library

Use pytesseract when: - Simple text extraction - Quick prototyping - Limited resources - Scanned documents

Use EasyOCR when: - Need good accuracy - Working with multiple languages - Handwriting recognition - General purpose OCR

Use PaddleOCR when: - Need best accuracy - Production environment - Complex documents - Have computational resources

Other Essential Libraries

Data Science

Library Purpose
NumPy Numerical computing, arrays
Pandas Data manipulation, analysis
Matplotlib Data visualization
Scikit-learn Machine learning
TensorFlow/PyTorch Deep learning

Web Scraping

Library Purpose
Requests HTTP requests
BeautifulSoup HTML/XML parsing
Scrapy Web crawling framework
Selenium Browser automation

Utilities

Library Purpose
Pillow Image processing
OpenCV Computer vision
pytest Testing framework
Click CLI applications

Best Practices

Project Structure

myproject/
├── venv/                 # Virtual environment (don't commit)
├── src/                  # Source code
│   ├── __init__.py
│   ├── main.py
│   └── utils.py
├── tests/                # Test files
│   ├── __init__.py
│   └── test_main.py
├── docs/                 # Documentation
├── requirements.txt      # Dependencies
├── .gitignore           # Git ignore file
├── README.md            # Project description
└── setup.py             # Package setup (optional)

.gitignore for Python

# Virtual environments
venv/
env/
ENV/

# Python cache
__pycache__/
*.pyc
*.pyo
*.pyd

# IDE
.vscode/
.idea/
*.swp

# Environment variables
.env

# Build
dist/
build/
*.egg-info/

Requirements.txt Tips

# Create from current environment
pip freeze > requirements.txt

# Create minimal requirements (only direct dependencies)
pip install pipreqs
pipreqs .

# Install in development mode
pip install -e .

Quick Reference

Virtual Environment Commands

# Create
python -m venv venv

# Activate
source venv/bin/activate  # Linux/Mac
venv\Scripts\activate     # Windows

# Deactivate
deactivate

# Delete (just remove folder)
rm -rf venv

Package Management

pip install package          # Install
pip install package==1.2.3   # Specific version
pip install -r requirements.txt  # From file
pip uninstall package        # Uninstall
pip list                     # List installed
pip freeze > requirements.txt  # Save current

pathlib Quick Reference

from pathlib import Path

Path("file.txt").exists()           # Check exists
Path("folder").mkdir()              # Create directory
Path("file.txt").read_text()        # Read file
Path("file.txt").write_text("Hi")   # Write file
Path("folder").glob("*.py")         # Find files

Additional Resources

Happy Python coding! 🐍✨