Python Sitemap Generator Project

Creating an XML sitemap is one of the most effective ways to boost a website’s SEO, and thanks to Python’s powerful libraries, you can automate the entire process with just a few lines of code. In this guide we’ll walk through a complete Python sitemap generator project—from planning and crawling to generating a standards‑compliant sitemap and deploying it to your server. Whether you’re a solo developer, a digital marketer, or part of a larger web‑development team, this tutorial will give you a reusable, extensible solution that saves time and improves search‑engine visibility.

Why a Custom Python Sitemap Generator?

While many CMS platforms offer built‑in sitemap features, a custom generator gives you full control over:

• Frequency and priority settings for each URL.
• Inclusion of dynamic pages that might be missed by generic tools.
• Integration with continuous‑integration pipelines for automatic updates.
• Support for multiple sitemap formats (XML, TXT, JSON) in a single project.

All of this can be achieved with a lightweight Python script that runs on any operating system.

Project Overview

The project is divided into four logical components:

1. Crawler – discovers URLs on the target domain.
2. URL filter – removes duplicates, external links, and unwanted patterns.
3. Sitemap builder – creates a valid XML file according to the Sitemap protocol.
4. Deployment script – uploads the sitemap to the web server or cloud storage.

Setting Up the Development Environment

Required Packages

We’ll rely on three popular Python packages:

• requests – for fetching page content.
• beautifulsoup4 – for parsing HTML and extracting links.
• lxml – for building the final XML sitemap efficiently.

Install them with a single pip command:

pip install requests beautifulsoup4 lxml

Project Structure

Keeping the code organized makes future enhancements easier. A typical layout looks like this:

python-sitemap-generator/
│
├── crawler.py          # Core crawling logic
├── sitemap_builder.py  # XML generation utilities
├── utils.py            # Helper functions (URL normalization, logging)
├── config.yaml         # Project configuration (domains, exclusions, priorities)
└── main.py             # Entry point that ties everything together

Step‑by‑Step Implementation

1. Load Configuration

Store settings in config.yaml so you can adjust crawl depth, exclude patterns, and default priority without touching the code.

site:
  base_url: https://example.com
  max_depth: 3
  user_agent: "PythonSitemapBot/1.0"
  exclude:
    - "/admin"
    - "/login"
  priority:
    home: 1.0
    blog: 0.8
    product: 0.6

2. Crawl the Site

The crawler uses a breadth‑first search (BFS) algorithm to respect the max_depth limit and avoid infinite loops.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
import yaml

def load_config(path="config.yaml"):
    with open(path, "r") as f:
        return yaml.safe_load(f)

def is_valid(url, base_netloc, exclude):
    parsed = urlparse(url)
    if parsed.netloc and parsed.netloc != base_netloc:
        return False
    for pattern in exclude:
        if pattern in parsed.path:
            return False
    return True

def crawl(base_url, max_depth, exclude):
    visited = set()
    queue = [(base_url, 0)]
    base_netloc = urlparse(base_url).netloc

    while queue:
        current_url, depth = queue.pop(0)
        if depth > max_depth or current_url in visited:
            continue
        try:
            resp = requests.get(current_url, timeout=10)
            resp.raise_for_status()
        except requests.RequestException:
            continue

        visited.add(current_url)
        soup = BeautifulSoup(resp.text, "html.parser")
        for link in soup.find_all("a", href=True):
            href = urljoin(current_url, link["href"])
            if is_valid(href, base_netloc, exclude):
                queue.append((href, depth + 1))
    return visited

3. Assign Priority and Change Frequency

Google recommends setting priority and changefreq values based on the importance and update rate of each page. The following helper maps URLs to these attributes using simple pattern matching.

def get_priority(url, config):
    if url.endswith("/") or url == config["site"]["base_url"]:
        return config["site"]["priority"]["home"]
    if "/blog/" in url:
        return config["site"]["priority"]["blog"]
    if "/product/" in url:
        return config["site"]["priority"]["product"]
    return 0.5  # default fallback

def get_changefreq(url):
    if "/news/" in url:
        return "hourly"
    if "/blog/" in url:
        return "daily"
    return "weekly"

4. Build the XML Sitemap

Using lxml.etree guarantees that the output complies with the official XML schema.

from lxml import etree
from datetime import datetime

def build_sitemap(urls, config):
    urlset = etree.Element(
        "urlset",
        xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
    )
    for url in sorted(urls):
        url_el = etree.SubElement(urlset, "url")
        etree.SubElement(url_el, "loc").text = url
        etree.SubElement(url_el, "lastmod").text = datetime.utcnow().date().isoformat()
        etree.SubElement(url_el, "changefreq").text = get_changefreq(url)
        etree.SubElement(url_el, "priority").text = str(get_priority(url, config))
    return etree.tostring(urlset, pretty_print=True, xml_declaration=True, encoding="UTF-8")

5. Save and Deploy

After generating the XML, write it to sitemap.xml and optionally upload it to your server via FTP, SFTP, or a cloud storage SDK (e.g., AWS S3).

def save_sitemap(xml_bytes, path="sitemap.xml"):
    with open(path, "wb") as f:
        f.write(xml_bytes)

def upload_to_s3(bucket_name, file_path, s3_key):
    import boto3
    s3 = boto3.client("s3")
    s3.upload_file(file_path, bucket_name, s3_key, ExtraArgs={"ContentType": "application/xml"})

6. Orchestrate with main.py

if __name__ == "__main__":
    cfg = load_config()
    urls = crawl(
        base_url=cfg["site"]["base_url"],
        max_depth=cfg["site"]["max_depth"],
        exclude=cfg["site"]["exclude"]
    )
    sitemap_xml = build_sitemap(urls, cfg)
    save_sitemap(sitemap_xml)

    # Optional: upload to AWS S3
    # upload_to_s3("my-bucket", "sitemap.xml", "sitemap.xml")
    print(f"Sitemap generated with {len(urls)} URLs.")

Best Practices for SEO‑Friendly Sitemaps

• Keep the file size under 50 MB or 50,000 URLs; otherwise split into multiple sitemap files and reference them in a sitemap index.
• Use lastmod dates that reflect actual content updates—search engines use this to prioritize crawling.
• Set priority values sparingly; they are hints, not guarantees.
• Validate the generated XML with tools like Google Search Console’s Sitemap Test before publishing.
• Schedule the generator to run after each deployment or on a nightly cron job to keep the sitemap fresh.

Extending the Project

Support for Multiple Languages

If your site serves localized versions (e.g., /en/, /fr/), add xhtml:link tags to indicate language alternates. The lxml library makes it easy to insert the required namespace.

Integration with Django or Flask

Wrap the generator in a management command (python manage.py generatesitemap) for Django, or expose it as a Flask route that returns the XML on demand. This removes the need for a separate static file and ensures the sitemap always reflects the live database.

Adding a Sitemap Index

When your site exceeds the 50,000‑URL limit, generate multiple sitemap files (e.g., sitemap1.xml, sitemap2.xml) and create an index file:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

    <sitemap>

        <loc>https://example.com/sitemap1.xml</loc>

        <lastmod>2026-10-13</lastmod>

    </sitemap>

    <sitemap>

        <loc>https://example.com/sitemap2.xml</loc>

        <lastmod>202