TL;DR
BeautifulSoup is a Python library for parsing HTML and XML into a navigable tree you can search with tags, attributes, and CSS selectors. It does not fetch pages or run JavaScript; you pair it with an HTTP client like requests. It is popular because it handles messy, broken markup gracefully and has a gentle learning curve.
What It Does and Does Not Do
BeautifulSoup is a parser, one piece of a scraper. It takes an HTML string and turns it into a tree of objects you can query. It does not download pages (that is requests or httpx), and it does not execute JavaScript (that is Playwright). Give it the HTML of a dynamic page before scripts run and it sees the empty shell, because it only parses the markup you hand it.
Basic Usage
import requests
from bs4 import BeautifulSoup
html = requests.get("https://example.com", timeout=(5, 30)).text
soup = BeautifulSoup(html, "lxml") # or "html.parser"
title = soup.find("h1").get_text(strip=True)
prices = [el.get_text(strip=True) for el in soup.select("span.price")]
links = [a["href"] for a in soup.find_all("a", href=True)]You navigate by tag (soup.find, soup.find_all), by CSS selector (soup.select), or by walking the tree (.parent, .next_sibling).
Why It Stays Popular
Two reasons. First, tolerance: real-world HTML is broken, and BeautifulSoup parses tag soup without choking, which is where its name comes from. Second, readability: its API is easy to learn, so it is the common first parser in tutorials. For speed at scale, people back it with the lxml parser or move to lxml directly, but for most jobs BeautifulSoup is fast enough and clearer to write.
The Fragility to Know About
Selector-based parsing breaks when a site changes its HTML structure or class names. A scraper built on soup.select("div.product-card") fails silently the day the site renames that class. This brittleness is inherent to selector parsing and is a reason AI-based extraction, which targets meaning rather than fixed selectors, has gained ground.
Key Takeaways
- BeautifulSoup parses HTML into a searchable tree; it does not fetch or run JS.
- Pair it with requests/httpx for fetching and lxml for speed.
- Selector-based parsing is brittle when a site's markup changes.
How ScrapeGraphAI Handles This
ScrapeGraphAI's extract endpoint replaces brittle selector parsing with AI that reads a page by meaning, so a class-name change does not break your scraper the way a hardcoded BeautifulSoup selector would.