Web Scraping: What It Is, How It Works and When It Gets Hard

Web scraping is using software to copy specific data off web pages and into a structured file, usually a spreadsheet: prices, listings, directory entries, reviews, job posts. The method is always the same five steps. Fetch the page, pick out the fields, clean them, store them as rows, and repeat on a schedule so the spreadsheet stays current. A person copying and pasting does the same job slowly. A scraper does it across hundreds of pages while you work on something else. The copying is rarely the hard part. The page is.

What is web scraping good for, and what it is not

Scraping earns its place when three things are true. The data is on pages anyone can open. It repeats in the same layout across many pages, like a product grid or a directory. And the site offers no download, feed or API that would hand you the same rows directly. Typical jobs are competitor price tracking, public business directories turned into a contact sheet, job posts in one field, review counts over time, and property or event listings.

It is the wrong tool in two cases. If you need fifteen rows once, copy and paste wins: setting up a scraper takes longer than the copying. And if the site already gives you an export or an API, use that instead. An official export keeps its column names from month to month. A web page does not promise you anything about its layout, and that difference decides how much maintenance you sign up for.

Web crawling vs scraping: two jobs that get mixed up

The two words are used as if they meant the same thing. They don't. A crawler is a program that systematically browses the web to collect data from pages, and search engines use crawlers to build their indexes ^[1]. Crawling answers which pages exist. Scraping answers what is on this page. Most real projects need a little of the first and a lot of the second.

In practice the two run back to back. You crawl a category page to collect the product links, then scrape each link for the same four or five fields. If you already have the list of URLs in a sheet, you can skip crawling entirely, which removes half of what can go wrong.

Five steps every scraping job goes through

1. Fetch. Request the page the way a browser would and receive what the server sends back.
2. Extract. Point at the parts you want (the price, the title, the rating) and pull each one into a named field.
3. Clean. Strip currency signs, trim spaces, turn 1.2K reviews into a number, make dates one format.
4. Store. Write one row per item, with the source URL and the date collected in their own columns.
5. Refresh and monitor. Run it again on a schedule and check that the row count still looks right.

Step one hides a second step on many modern sites. The server sends a nearly empty page, and JavaScript fills in the content afterward. Google describes its own pipeline for such pages as three phases, crawling, rendering and indexing, with rendering queued separately ^[2]. Your scraper faces the same split. A tool that only fetches sees the empty shell. A tool that renders sees what you see.

For simple, static pages the spreadsheet you already own can do the fetching, extracting and storing with a single formula. We cover the exact functions in how to scrape a website with Google Sheets or Excel.

A weekly price sheet for forty competitor products

Take a shop that sells espresso gear and wants to watch forty products at three competing stores. Start with the spreadsheet, not the scraper. Six columns: product, store, price, in stock, URL, date checked. Fill the URL column by hand or with a short crawl of each store's category page. That list of one hundred and twenty URLs is now the whole scope of the job, and it is small enough to reason about.

Next, extract price and stock for the first twenty URLs and compare every value with the live page. This is where the real problems show up. One store prints a sale price beside a crossed-out list price. Another shows from prices for products with sizes. A third hides stock status until you pick a color. Decide the rule for each case once, write it down in the spreadsheet, and only then run all the rows.

Prices on gear like this move slowly, so a weekly run is enough. Each run adds new rows with a new date instead of replacing the old ones. After two months the spreadsheet answers a better question than what does it cost today: who discounts, when, and by how much.

Why scrapers break: the page changes, not your sheet

Content that loads after the page

Infinite scroll, load more buttons and prices that appear a second after the page opens all mean JavaScript is building the content. A plain fetch returns blanks. You need something that runs a real browser, waits, scrolls and clicks.

Logins and codes

Data behind an account needs a signed-in session. Many sites then ask for a one-time code or a CAPTCHA. A person has to step in at that moment, so plan for it instead of hoping it won't happen.

Pagination

Results spread over fifty pages are fifty fetches with a pattern in between: a page number in the URL, a next link, or a button. Miss the pattern and you collect page one fifty times.

Layout changes

Sites redesign without telling you. The field that held the price last month now holds the shipping note. Nothing crashes. The sheet simply fills with the wrong thing.

Terms, robots.txt and personal data: the respect line

A page being public does not mean its owner agreed to automated collection. Before a job runs, do three checks. Read the site's terms for what they say about automated access. Open the site's robots.txt file, which lists the rules crawlers are requested to honor; the standard itself notes that these rules are not a form of access authorization ^[3]. And look at your own column list for names, emails or phone numbers of private people. If you have no right to use that data, leave the column out.

Then there is pace. Google's own guidance says robots.txt is used mainly to avoid overloading a site with requests ^[4], which tells you what site owners worry about. Fetch slowly, run at quiet hours, and collect only the pages you need. A weekly run of one hundred and twenty pages is a visitor. Thousands of requests a minute is a problem you caused for someone else.

When an AI agent does the collecting for you

An agent changes who can do this work. Instead of writing selectors, you describe the rows you want in plain language, and the agent opens a browser, reads the pages and fills the table. Three products in our AI agents catalog approach it differently; the side-by-side view, part by part, is in web scraping tools compared by the job each one covers. The first suits a non-developer who needs the same collection to repeat on a schedule, and our guide to web scraping in Gumloop walks through the setup.

The second answers a different situation: the data sits inside an account you already use, such as a supplier portal or a members-only directory. There the question is less about scale and more about whose browser does the reading. Our article on the Manus AI Browser Operator explains what you authorize and when.

The third is for a team that already builds its own automations and wants scraping as one step inside a larger tool, with outside scraping services plugged in where the page is difficult. It asks for more assembly, and it gives you control over each step in return; our article on tool steps in Relevance AI shows what a step is.

Whichever route you take, a person checks a sample before the rows feed a pricing decision, a mailing or a system of record. Manus says in its own documentation that output can be confident but wrong. Treat every agent that way.

Start with twenty rows you can check by hand

Pick one page type and write the column list before you touch a tool. Do the three checks on the site. Then collect twenty rows by whatever means is closest, a formula, an agent or plain copy and paste, and compare each one with the live page. If all twenty hold, schedule the run and add a row count to watch. If they don't, you just found the rule your scraper was missing, and it cost you ten minutes instead of a month of bad data.