Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python BeautifulSoup: find, find_all, and CSS Selectors

Learn when to use find, find_all, and CSS selectors in BeautifulSoup. Compare syntax, performance, and practical decision criteria.

~4 min readPython
#BeautifulSoup#web scraping#CSS selectors#Python
Illustration of a Python BeautifulSoup document tree with highlighted elements selected via find_all and CSS selectors

When parsing HTML with Python's BeautifulSoup library, you can locate elements with the find/find_all methods or with CSS selectors through select/select_one. Choosing between the two depends on the structure of your query, the kind of filtering you need, and how readable you want the scraping code to be.

How find() and find_all() Work

The find() method returns the first matching tag, while find_all() returns a ResultSet (a list subclass) of all matching tags. Both accept a tag name, attribute filters, text filters, or a combination of filters. For example:

python
from bs4 import BeautifulSoup html = """ <div class="product"> <span class="price">$19.99</span> <span class="name">Widget</span> </div> <div class="product"> <span class="price">$29.99</span> <span class="name">Gadget</span> </div> """ soup = BeautifulSoup(html, 'html.parser') first_price = soup.find('span', class_='price') all_prices = soup.find_all('span', class_='price')

find() returns a Tag object or None if no match exists. find_all() always returns a list-like object, even when empty. This distinction matters when you chain methods or check for existence.

CSS Selectors with select() and select_one()

BeautifulSoup's select() method accepts a CSS selector string and returns a list of matching elements. select_one() returns the first match or None. CSS selectors are often more expressive for complex structural queries:

python
# Equivalent to the previous example first_price = soup.select_one('span.price') all_prices = soup.select('span.price')

You can use descendant combinators, attribute selectors, and pseudo-classes such as :nth-of-type(2). This often reduces code verbosity compared to nested find_all calls.

Comparing find/find_all and CSS Selectors

Featurefind/find_allselect/select_one
SyntaxMethod calls with filtersCSS selector string
Return typeTag or list of TagsTag or list of Tags
ExpressivenessLimited to method filtersComplex CSS selector syntax
ReadabilityVerbose for nested queriesCompact for complex paths
PerformanceTraverses with Python filtersParses CSS, then traverses
Best use caseSimple lookups, dynamic filtersComplex structural queries

Both approaches traverse the parsed tree. find_all applies the Python filter you provide, while select first parses the CSS selector string. For ordinary scraping scripts the difference is usually negligible. If performance matters, measure and optimize only after confirming a bottleneck.

When to Use CSS Selectors Over find_all

Use CSS selectors when your query involves hierarchy, attribute conditions, or pseudo-classes. For example, extracting all links inside a specific section:

python
# With find_all links = [] for section in soup.find_all('div', class_='content'): links.extend(section.find_all('a')) # With select links = soup.select('div.content a')

The select version is shorter and directly expresses the intended structure. If you need to filter by attribute value, CSS selectors are often clearer:

python
# find_all with attribute filter inputs = soup.find_all('input', attrs={'type': 'text'}) # select with attribute filter inputs = soup.select('input[type="text"]')

Performance and Parsing Overhead

Both approaches traverse the parsed tree once a query is built. select also has to process the selector string, so repeated use of the same selector in a hot loop can be worth optimizing. SoupSieve, the selector engine used by BeautifulSoup, lets you compile a selector once and reuse it:

python
import soupsieve compiled = soupsieve.compile('div.product > span.price') # Later, reuse compiled on any soup object prices = compiled.select(soup)

This avoids repeated work in code that runs the same query many times. For typical scraping scripts, the difference is rarely noticeable; do not optimize unless you have profiled the code.

Common Pitfalls with Class Names and Nested Elements

A common mistake is using class instead of class_ in find_all. Python reserves class, so BeautifulSoup uses class_ as the argument name. In CSS selectors, dot notation is natural for classes.

Another pitfall is handling multiple class names. CSS selectors express class-list conditions directly:

python
# Element has both 'a' and 'b' classes soup.select('.a.b') # Element has either 'a' or 'b' class soup.select('.a, .b')

With find_all, these conditions require a custom predicate or more verbose filtering, so CSS is usually the better choice for class-list logic. Nested element queries also differ: find_all often needs explicit loops or list comprehensions, while select uses descendant combinators directly.

Practical Example: Scraping a Table with Both Approaches

Consider an HTML table with rows and cells. You want to extract all cell text from the second column.

python
html = """ <table> <tr><td>Name</td><td>Price</td></tr> <tr><td>Widget</td><td>$10</td></tr> <tr><td>Gadget</td><td>$20</td></tr> </table> """ soup = BeautifulSoup(html, 'html.parser') # Using find_all prices = [] for row in soup.find_all('tr'): cells = row.find_all('td') if len(cells) > 1: prices.append(cells[1].text) # Using select prices = [td.text for td in soup.select('tr td:nth-of-type(2)')]

The select version is more concise and directly targets the second cell without manual index checks. It also avoids issues with rows that have fewer cells.

Choosing the Right Method for Your Scraper

There is no universal winner. Use find_all when you need to filter by dynamic Python conditions that are hard to express in CSS, such as a custom predicate over attribute values. Use select when your query is structural and static, especially if you are comfortable with CSS. For complex paths, select often reads like the document structure. For simple lookups, either works; pick the one that makes the code clearer to your team.

If you are already using BeautifulSoup, both methods operate on the same Tag objects, so you can mix them freely. The key is to choose the approach that minimizes cognitive overhead for the specific query you are writing.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.