Simple, honest definitions with no needless jargon. Enough to understand why a site blocks you and what it takes to get through.
Automated extraction of data from web pages: a program fetches the HTML (or a JSON response) of a URL, then isolates the useful information (prices, titles, reviews...). It is lawful for public data, but subject to the site's terms, GDPR and copyright.
Quickstart in 5 minutes →A crawler follows a site's links to discover its pages (like Googlebot); a scraper extracts the content of a given page. You crawl to find the URLs, you scrape to read what they contain. The two usually go together.
When a site offers an official API, that's the most reliable route: structured, stable, authorized data. Scraping steps in when there is no API, when it's throttled, or when the data only appears in the page. Checking for an API before scraping saves time.
A file at a site's root (/robots.txt) that tells robots which areas not to crawl. It isn't legally binding on its own, but honoring it is good practice and a sign of responsible scraping. It doesn't replace reading the terms of service.
A cap on how many requests a single source (IP, account, key) can send in a given time window. Past the threshold, sites often return a 429 (Too Many Requests). You work around it by spacing calls out and spreading traffic across several IPs.
Parameters: rate and country →An HTTP header by which a client introduces itself (browser, version, OS). A missing, generic or incoherent user-agent is a classic bot signal. It must stay consistent with the other headers and the client's real fingerprint.
Tool: parse a User-Agent →Splitting a list of results across several pages (?page=2, offset, cursor, infinite scroll). Scraping a full list means following that mechanism to the end, handling duplicates and the end-of-list so nothing is missed and nothing loops forever.
A very common CDN and shield. Beyond caching, its Bot Management and its "Just a moment..." page (JS challenge, sometimes Turnstile) filter out automated clients. A plain HTTP client gets blocked: you need a real browser, a credible TLS fingerprint and a coherent IP.
Tool: detect a URL's protection →A specialized anti-bot protection, common on marketplaces (Vinted, Leboncoin...). It combines browser fingerprinting, behavioral analysis and a "press & hold" CAPTCHA. It rarely lets a bare HTTP client through; you need a warmed-up browser session and a clean IP.
Guide: scrape Vinted →An anti-bot solution (rebranded HUMAN) that sets a challenge cookie (_px) and scores client behavior. Like DataDome, it aims to tell a human from a script and requires a real browser with coherent signals.
Protection built into the Akamai CDN, common on large e-commerce and banking sites. It relies on cookies (_abck, ak_bmsc) and client-side signal collection. It requires a solid browser fingerprint to avoid being flagged as a bot.
An application shield (formerly Incapsula) that filters traffic via cookies (visid_incap, incap_ses) and challenge pages. You recognize it by those markers; getting past it again means a real browser and a reputable IP.
A test meant to prove a human is in control: checkboxes, image selection, or an invisible check (reCAPTCHA v3, Cloudflare Turnstile). Faced with a CAPTCHA, you avoid it upstream (good fingerprint, good IP) rather than "break" it: solving it systematically is costly and brittle.
A signature computed from how a client negotiates the encrypted connection (cipher order, extensions...). A standard HTTP library produces a different fingerprint from a real Chrome: protections use it to spot bots before even reading the request.
A set of signals collected in JavaScript to identify a browser: canvas/WebGL rendering, fonts, resolution, timezone, navigator properties... A poorly configured automated browser leaves traces (navigator.webdriver, inconsistencies) that anti-bots detect.
A trap invisible to humans but visible to robots: a hidden link or field that a naive scraper will follow or fill, giving itself away. Avoiding it means only visiting elements that are actually visible and clickable.
Detection based not on what the client is, but on what it does: browsing speed, mouse movement, inhuman regularity, not loading secondary resources. Traffic that's too perfect or too fast becomes suspicious.
A proxy whose IP address belongs to a consumer ISP (a real home line, a real subscriber). To a site, the request looks like an ordinary person's, which makes it far more credible than a datacenter IP. It's often required against strict protections.
Docs: proxy mode and exit country →A proxy hosted at a cloud/hosting provider. Fast and cheap, but its IP belongs to well-known ranges that protections block easily. Fine for lightly protected sites, not enough for strict marketplaces.
A proxy whose IP comes from a mobile network (4G/5G). Because thousands of subscribers share the same IP through the carrier's NAT, blocking it is risky for the site: these IPs are among the most tolerated, but also the most expensive.
Changing IP address on each request (or regularly) to spread traffic and dodge rate-limiting thresholds. Effective for volume, but to be avoided inside a logged-in session, where the IP must instead stay stable.
A setup where you keep the same exit IP for a whole session, rather than rotating. Essential once logged in or when a cart/form spans several pages: switching IP midway breaks the session and raises suspicion.
Docs: sessions and logins →Many sites build their content in the browser, after the fact, in JavaScript. Fetching the raw HTML then isn't enough: you have to run the page (via a real browser) for the data to appear. That's "JS rendering".
Docs: enable JS rendering →A real browser (Chrome, Firefox) driven by code, with no graphical interface, via tools like Playwright, Puppeteer or nodriver. It runs JavaScript and mimics a human, but poorly configured it exposes automation signals that anti-bots catch.
Using a service (humans or models) to answer a CAPTCHA on the fly. It's a last resort: slow, costly and unstable. Better to avoid the CAPTCHA upstream, with a credible fingerprint and IP, than to solve it in a loop.
Tokens set by the site to recognize a client from one request to the next (login, cart, passed anti-bot challenge). Keeping and sending them back correctly is essential: losing them often restarts the anti-bot check from scratch.
WyndPath handles proxies, JS rendering and anti-bot in one call. Billed on success.
Start for free →