How anti-bot systems work

When a scraper returns a 403 error or a "Just a moment…" page instead of the content you expected, it's never by chance. An anti-bot system observed the request, compared it to what it expects from a real browser, and spotted an inconsistency. These systems don't ask a single question, they stack several. Here are the layers they go through, in the order they come into play.

Layer 1: the network, before the request even lands

It all starts with the IP address. Before it looks at what you're asking for, a server looks at where you're coming from. Not all IPs are equal, and an implicit trust hierarchy has taken hold.

Mobile, then residential, then datacenter: that's the order of trust an IP is judged by.

A mobile IP (a cellular carrier's network) inspires the most trust, because thousands of subscribers share the same address, which makes blocking risky. A residential IP (a home internet box) is considered credible, because a real human uses it. A datacenter IP (a server rented from a hosting provider) is immediately suspicious: very few real visitors browse from a data center. The server identifies the origin through the network number (ASN) and reverse DNS. On top of that comes rate limiting: too many requests from the same address, at too regular a pace, and access shuts down.

Layer 2: the TLS handshake

This is the least understood layer, and often the most decisive. Before any data is exchanged over HTTPS, the client and the server negotiate encryption. The client's very first message, called the ClientHello, is sent in the clear and describes its capabilities: supported versions, cipher suites, extensions, in a precise order. And that order and content vary depending on the software library used.

You can therefore compute a fingerprint of this handshake, known as JA3 and later JA4. The problem for a naive scraper is easy to state:

A client that claims to be Chrome in its headers, but shakes hands like a Python script, contradicts itself.

No header tweak hides this contradiction, because it plays out at a lower level than HTTP. We break down this mechanism in a dedicated article on TLS fingerprinting.

Layer 3: the browser fingerprint

Once the connection is established and a page has loaded, a JavaScript script gets to work. It queries dozens of browser properties to build a stable identifier, without needing a cookie.

• Canvas and WebGL: the script asks to draw an invisible image, then reads the result. Depending on the graphics processor, the driver and the installed fonts, the rendering varies very slightly, but reproducibly.
• Fonts, resolution, time zone, language: so many small signals that, combined, become almost unique.
• Automation clues: the presence of a flag like navigator.webdriver, set by automation tools, or purely software-based graphics rendering, gives away a browser driven by a program.

The system's strength doesn't come from any single signal, but from their consistency. A browser that claims to be Chrome on Windows but whose graphics rendering looks like that of a headless Linux server, that's a story that doesn't hold up.

Layer 4: behavior

The most advanced engines don't just take a snapshot of the client, they watch it act. Mouse movements, scroll speed, click rhythm, request pace: everything is analyzed in real time.

A human is irregular by nature. Their mouse follows curves, hesitates, speeds up then slows down. A bot, on the other hand, tends toward mechanical perfection: straight-line trajectories, intervals perfect to the millisecond, or quite simply no movement at all. That very regularity is what gives it away.

Layer 5: the challenge

When doubt remains, the system imposes a test. They don't all look alike:

• The invisible score: some systems, like reCAPTCHA v3, show nothing to the user. They assign a trust score in the background and let the site decide on the threshold.
• The computational challenge: Cloudflare's "Just a moment…" page, for example, has the browser solve a series of small JavaScript problems, including a deliberately expensive computation. Negligible for a real browser, that cost becomes prohibitive when you have to pay it millions of times with a fleet of bots.
• The visual puzzle: image selection only shows up as a last resort, when the score is really low.

The major vendors, and their signatures

A handful of players dominate this market, each with its own approach. They leave recognizable traces, often in the form of cookies.

• Cloudflare operates as close to the visitor as possible, on its global delivery network. It sets a __cf_bm cookie for its bot score, and cf_clearance as proof that a challenge was passed. The latter is tied to the session, the announced browser and the IP all at once: change any of the three, and it becomes invalid.
• DataDome, a French vendor, decides in a few milliseconds based on a machine-learning model that combines fingerprint, behavior and network. Signature: the datadome cookie.
• Akamai relies on heavy telemetry collected on the client side. Its _abck cookie becomes invalid at the slightest inconsistency with the observed fingerprint.
• HUMAN (formerly PerimeterX) bets on behavioral biometrics and intelligence shared across its customers.
• Kasada serves a heavily obfuscated script that embeds its own virtual machine, and imposes a computational cost per request.
• Imperva, acquired by Thales in 2023, combines a web application firewall and a delivery network.

Why a simple proxy is never enough

This is the most common mistake. A proxy changes only one thing: the IP address. It fixes layer 1, and leaves all the others untouched. A scraper that goes through a nice residential IP but shakes hands in Python, without running the JavaScript and without any credible behavior, gets spotted at the second layer.

To cleanly access public data, every layer has to tell the same story: a real browser engine capable of running the JavaScript and the challenges, an IP consistent with the profile, and a perfect match between the TLS handshake, the headers, the browser properties and the behavior. That orchestration work is exactly what WyndPath takes on, while respecting the rules published by sites and rate limiting.