Bot Traffic and Referrer Spam: What It Looks Like and What a Tracker Can Catch

What bot traffic and referrer spam look like in a dashboard

Bots are not one thing. A search crawler, an uptime monitor, a price scraper and a vulnerability scanner differ in whether they run JavaScript, announce themselves or look like a person.

Referrer spam is a separate problem with a different motive. The goal is not to read your pages. It is to put a domain into your referrer report. In crawler spam a real bot requests your pages with a spam domain in the referrer header. In ghost referrals nothing visits your site: the hit is sent straight to the analytics endpoint with a made-up referrer. The tell for crawler spam is a referrer domain with no page linking to you.

Why script-based trackers see fewer crawlers

A server log records every HTTP request the server answers. A script-based tracker records only what a browser engine reports after it has loaded your page and run the tracking script.

Most simple crawlers fetch the HTML, read the links and move on. They do not execute JavaScript, so the tag never runs and your analytics never hears about them. The bots that do reach a script-based tracker run a full browser: headless scrapers, rendering crawlers, uptime monitors and any bot written to look like a person. That group is smaller, but it is the one most likely to be mistaken for real visitors.

So the two numbers answer different questions, and a gap between them is normal. For the mechanics of cookieless collection, see cookieless tracking explained. One more way a visit never reaches the dashboard: the tracker reads the browser's Global Privacy Control and Do Not Track signals and sends nothing when either is on.

What GhostMetrics filters: two layers, and what neither does

Layer one runs at ingest and decides what is stored; layer two runs when numbers are read and decides what is counted (next section). The ingest filter lives in the Worker that receives the tracker's beacons, and every event type goes through it.

What it drops

A request is dropped if any one of three things is true:

1. It carries an internal QA marker. Our own test tooling uses it. This only matters to us.
2. It has no User-Agent.
3. Its User-Agent matches the pattern list. The match is a case-insensitive substring.

A dropped request gets an HTTP 204, the same status a stored beacon gets, and no row is written. The check runs before the rate limiter and any database write.

What it costs you

Because the event is never stored, the dashboard has no counter for what this layer dropped and no way to review it. A false positive is invisible and cannot be recovered. The match is also a plain substring, so a real browser whose User-Agent happens to contain one of these strings, a device model name containing bot for example, is dropped with the bots.

The check also runs once, when the event arrives. Adding a pattern later does not re-evaluate rows that are already stored. The network layer below works the other way: it is applied at read time, to all history.

The rate limit is a separate control

After the User-Agent check, ingest applies a rate limit of roughly 100 events a minute per source address. Requests over it get a 429 and are not stored. It bounds abuse; a slow bot stays under it.

What neither layer does

• No test for bots that look like browsers. The ingest filter reads the User-Agent string and nothing else. A headless browser that has had its User-Agent changed to an ordinary Chrome string passes it. The network layer looks only at the network, so the same browser on a connection that is not on the data-centre list passes that too.
• No referrer blocklist or allowlist. A referrer is reduced to its host, with prefixes such as www dropped, and given a type, but it is never removed. A spam domain appears as a row like any other.
• No IP-reputation list, and no complete hosting list. Nothing looks an address up against known bad actors. The only network test is a curated list of data-centre network numbers; a hosting provider that is not on it counts like any other visitor.
• No hostname filter and no IP exclusion for your own visits. The tracker sends the page path, not the full address, and IP addresses are not stored, so there is no "ignore my office" setting.
• No cleanup you control. The dashboard has no control to delete or exclude stored rows or days of your choosing. The data-centre list covers all history, but we maintain it, not you. If a day is polluted by something the filters missed, exclude it when you work from the export.

A related point about referrer reporting: source types come from a host table, not from searching the referrer string for a name. A table maps known hosts to a type. The types are Search (Bing, DuckDuckGo, Google's country domains and others), Social (Facebook, Reddit, LinkedIn and others), AI (ChatGPT, Perplexity, Claude and others), Direct (an empty referrer) and Other (any host not in the table). The match is on the host itself, so a spam domain that merely contains a search engine's name is typed Other. A fabricated referrer that names a real search or social host is still typed Search or Social, because the type follows the host as sent.

The second layer: the data-centre exclusion, and the Organizations panel

Each stored hit carries the number and name of the network it came from, taken from the Cloudflare edge at request time. The address itself is not stored. Two mechanisms use the network, and they are easy to confuse: one is a filter and one is a view.

The data-centre exclusion is a filter

A curated list of data-centre network numbers is checked against each stored hit. It includes Amazon AWS, Google's service and cloud networks, DigitalOcean, OVH, Hetzner, Linode, Vultr, Oracle Cloud, Alibaba Cloud, Tencent Cloud and several other hosting providers. The reasoning is that a visit from one of these is a script, a crawler or a monitor, not a person at a browser.

A hit from a listed network is flagged at ingest, but the row is stored. The exclusion happens when the dashboard reads your numbers: every statistic leaves out rows whose stored network number is on the current list. A change to the list therefore applies to old rows too, and stored events are never rewritten.

A network-only test has a cost. Cloudflare WARP, iCloud Private Relay and Microsoft's networks are left off the list on purpose because they carry real people. And a person browsing through a VPN or proxy that runs on a listed network is left out with the bots.

To see what was left out, look under the stat cards on the Overview tab. A line reads "N pageviews from scripts and crawlers (data-centre networks) were left out" for the period. The owner's dashboard has a Show them button on it: press it and every panel, the Organizations panel included, recomputes with those pageviews included, and a Hide them button replaces it. A published dashboard shows the count but not the button. The raw export is not filtered: it holds every stored row, with the network number, network name and the ingest-time flag as columns.

The Organizations panel is a view

The Organizations panel, on the Who's Looking tab, groups hits by network name and hides residential ISPs by default, because a home broadband provider is usually a person, not a company visiting you. The button reads Excludes residential ISPs, with a count of hidden networks. Residential is decided by a list of named consumer networks plus generic words such as broadband and residential matched in the network name.

• It changes one panel. Pageviews, referrers, countries and every other view still include residential hits. Nothing is deleted.
• It is an audit, not a verdict. The owner's dashboard loads every row flagged Residential or Business, and the badge is a toggle: click it and the hidden networks come back.

For spotting automation, the useful part is the Business side. Listed data-centre networks are already left out of the panel by default, so a hosting-looking name you do see there is a provider that is not on the list, and its hits count in every number. Bots routed through home connections are classified as residential and sit behind the badge, so when hunting a spike, turn the badge on and press Show them.

On a customer's published dashboard, neither the badge toggle nor Show them is available to viewers. Our own demo keeps the badge toggle, because it exists to show the feature.

How to sanity-check a weird spike

1. Pin down the day. Click the bar for that date in the daily chart; the day panel shows top pages and referrers for that date only.
2. Ask whether it is one page or many. One page points to a shared link, a monitor or a fixated scraper; many pages in sequence point to a crawler.
3. Read the referrer. Find the value in the Referrers table with the search box. For an unfamiliar domain, search for it as plain text rather than loading it, and check whether any page on it links to yours. No link means it is more likely spam.
4. Read the left-out line. Under the stat cards, note how many pageviews from data-centre networks were left out for the period. Press Show them to see whether the spike appears; if it does, it was already outside your default numbers.
5. Open the Organizations audit. Click the badge so the residential networks appear too. One hosting or cloud name at the top of the list, or one provider accounting for most of a spike, is a strong hint. A provider on the data-centre list will not appear unless Show them is on.
6. Read the page averages carefully. In All Pages, Avg Scroll and Avg Time are averaged only over pageviews whose leave beacon arrived. A bot that loads the page and is killed often never sends one, so a big Views jump with unchanged averages is a hint, not proof. See time on page vs session duration.
7. Check the cluster. Countries on the Geo tab and Browser, OS and Device on the Visitors tab should look like your normal audience; one country, browser and device type is rarely a crowd.
8. Export and sort. Download the events CSV with the Export data button, on every plan. The columns include the referrer, network number and name, country, page and timestamp. Sort by timestamp for even spacing, then by network and referrer.
9. Compare with a source that sees requests. Host access logs or CDN analytics record requests whether or not a script runs. A flood in the log with a flat dashboard means non-JavaScript bots.
10. Decide, and write it down. If you conclude the day is polluted, note the date and leave it out when you report from the export. The dashboard cannot exclude individual stored rows or days, so the note is how you stay consistent next month.

For a routine that catches spikes early, the weekly dashboard guide puts the referrer and trend checks in a 15-minute weekly pass.

How GA4 handles bots and spam

GA4 approaches this with more machinery, and it is the stronger tool in places. Google's documentation describes automatic exclusion of traffic from known bots and spiders, maintained by Google. The admin interface offers internal-traffic rules and a way to mark domains as unwanted referrals. Check the current Google Analytics documentation for details, because they change. Our GA4 alternative guide covers the wider comparison.

If you run paid campaigns and need Google to discard junk traffic before it distorts audiences, GA4's managed filtering is a real advantage. If you want a small, cookieless tool where you can read every stored row, a short filter (User-Agent at ingest, data-centre networks at read time) with a full export is easier to reason about.

FAQ

Keep reading

Disclosure: GhostMetrics is built by Null Agency and we use it on this site, so we are not a neutral party about our own product. GA4's bot handling and settings are set by Google and change over time; check the current Google Analytics documentation before relying on any detail.