hiseomarketing.in

Python News Aggregator Web App Project

Written by

arun

in

Imagine a single, sleek web portal that pulls the latest headlines from your favorite tech blogs, financial journals, and local news stations—all in real time, neatly organized, and searchable with a single click. Building a Python news aggregator web app not only sharpens your coding skills but also creates a valuable tool you can showcase in your portfolio. In this guide we’ll walk through the why, the what, and the how of creating a robust, SEO‑friendly news aggregator from scratch, using modern Python frameworks, APIs, and best‑practice design patterns.

Why Build a News Aggregator with Python?

Core Benefits

  • Hands‑on learning: Work with HTTP requests, JSON parsing, and database ORM—all essential Python skills.
  • Portfolio boost: A live web app demonstrates full‑stack competence to recruiters and clients.
  • Customizable content: Tailor the feed to niche interests, regional news, or industry‑specific updates.
  • SEO advantage: Properly indexed aggregator pages can rank for long‑tail keywords like “latest Python tutorials” or “daily tech news roundup”.

Key Features of a Modern News Aggregator

  • Automated fetching from RSS feeds, public APIs, and web‑scraped sources.
  • User‑defined categories, tags, and keyword filters.
  • Responsive UI with infinite scroll or pagination.
  • Article summarization and sentiment analysis (optional AI add‑on).
  • Bookmarking, sharing, and email newsletter integration.
  • SEO‑optimized URLs, meta tags, and schema markup.

Choosing the Right Python Framework

Flask vs Django

  • Flask: Lightweight, flexible, ideal for micro‑services or when you want full control over each component.
  • Django: Batteries‑included, with built‑in admin, ORM, and authentication—great for rapid development of larger projects.
  • Both support Jinja2 templating, easy integration with requests for API calls, and can be containerized with Docker.

Project Architecture Overview

High‑Level Components

  • Fetcher Service: Periodic tasks (Celery or APScheduler) that pull articles from sources.
  • Database Layer: PostgreSQL or SQLite for storing raw articles, tags, and user preferences.
  • RESTful API: Exposes endpoints for the frontend (e.g., /api/articles, /api/categories).
  • Frontend UI: HTML5/CSS3 + JavaScript (or a lightweight framework like Alpine.js) to render the feed.
  • SEO Middleware: Generates meta tags, Open Graph data, and JSON‑LD schema for each article page.

Data Flow Diagram

User Request → Flask/Django View → API Layer → Service Layer
    ↑                                 ↓
  Frontend ← JSON Response ← Database (PostgreSQL)

Step‑by‑Step Development Guide

1. Set Up the Development Environment

# Create a virtual environment
python -m venv venv
source venv/bin/activate   # On Windows use `venv\Scripts\activate`

# Install core packages
pip install flask sqlalchemy requests beautifulsoup4
pip install celery redis        # Optional for async fetching
pip install gunicorn           # For production server

2. Fetch News Sources (RSS, APIs, Scraping)

Below is a minimal example using feedparser for RSS and requests for JSON APIs.

import feedparser
import requests

def fetch_rss(url):
    feed = feedparser.parse(url)
    for entry in feed.entries:
        yield {
            'title': entry.title,
            'link': entry.link,
            'published': entry.published,
            'summary': entry.summary
        }

def fetch_json_api(endpoint, headers=None):
    resp = requests.get(endpoint, headers=headers, timeout=10)
    resp.raise_for_status()
    data = resp.json()
    for item in data['articles']:
        yield {
            'title': item['title'],
            'link': item['url'],
            'published': item['publishedAt'],
            'summary': item['description']
        }

3. Store Articles in a Database

from sqlalchemy import Column, Integer, String, Text, DateTime, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker

Base = declarative_base()

class Article(Base):
    __tablename__ = 'articles'
    id = Column(Integer, primary_key=True)
    title = Column(String(255), nullable=False)
    link = Column(String(500), nullable=False, unique=True)
    published = Column(DateTime)
    summary = Column(Text)

engine = create_engine('sqlite:///news.db')
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)

4. Build the Backend API

from flask import Flask, jsonify, request
from sqlalchemy.orm import scoped_session

app = Flask(__name__)
db_session = scoped_session(Session)

@app.route('/api/articles')
def get_articles():
    page = int(request.args.get('page', 1))
    per_page = 20
    query = db_session.query(Article).order_by(Article.published.desc())
    articles = query.offset((page-1)*per_page).limit(per_page).all()
    return jsonify([{
        'title': a.title,
        'link': a.link,
        'published': a.published.isoformat(),
        'summary': a.summary
    } for a in articles])

if __name__ == '__main__':
    app.run(debug=True)

5. Create the Frontend Interface

  • Use a simple index.html with a container element for the article list.
  • Fetch data via fetch('/api/articles?page=1') and render with vanilla JS or a tiny library.
  • Implement infinite scroll: detect when the user reaches the bottom and request the next page.
  • Add SEO‑friendly <meta> tags dynamically server‑side for each article page.

Enhancing SEO and Performance

  • Server‑Side Rendering (SSR): Render article snippets on the server to let crawlers see content without JavaScript.
  • Canonical URLs: Prevent duplicate content penalties by defining a single canonical link for each article.
  • Schema.org Markup: Use Article JSON‑LD to improve rich‑snippet eligibility.
  • Cache Headers: Set Cache-Control and ETag headers for static assets and API responses.
  • Lazy Loading Images: Reduce initial page weight and improve Core Web Vitals.

Common Pitfalls and How to Avoid Them

  • Rate‑Limiting: Respect source API limits; implement exponential back‑off and store timestamps of last fetch.
  • Duplicate Articles: Use the article URL as a unique key in the database to prevent repeats.
  • Encoding Issues: Normalize text to UTF‑8 before saving; handle malformed HTML with BeautifulSoup’s html.parser.
  • Security Risks: Sanitize user‑generated content, set CSP headers, and keep dependencies up‑to‑date.
  • Scalability: Offload heavy fetching to background workers (Celery) and consider a message broker like Redis.

Deploying Your App to Production

More posts