VidiumBot
The Vidium web crawler. Operated by Vidium (vidium.co).
What it does
A focused crawler that scans a list of publisher domains, visits their pages, detects embedded video players, records page dates, and logs the results for analysis. Vidium uses the data to verify video ad placement and quality for advertisers: which video player a page runs, its size and position, whether it autoplays or is muted, and how much of it is advertising versus editorial content.
It detects HTML5 video, YouTube, Vimeo, Brightcove, JW Player, Kaltura, Wistia and other common players. It reads pages the way a browser does. It does not log in, submit forms, download media files, or interact with advertising.
How it behaves
- Obeys
robots.txt, includingcrawl-delay, and reads sitemaps from it to find pages. - Loads a small number of pages per site, one at a time by default, with a hard per-page time limit.
- Retries a page that fails to load once, then leaves the site alone.
- Crawls from Google Cloud addresses in the country the publisher serves.
- Where a publisher charges crawlers through Cloudflare Pay Per Crawl, it pays the quoted price or leaves.
How to identify it
| User agent | A Chrome user agent string followed by the token VidiumBot/1.0 (+https://bot-directory.vidium.co) |
| Request signing | Every request carries a Web Bot Auth signature (HTTP Message Signatures, Ed25519) with Signature-Agent: "https://bot-directory.vidium.co". |
| Key directory | https://bot-directory.vidium.co/.well-known/http-message-signatures-directory |
| Key id | AGOaJ7iI-I9-OH0FpGIF-0_-WcN21d3FPrEVojBkPWI |
To block it or ask a question
Disallow VidiumBot in robots.txt and it will stop. For anything else, email michael@vidium.co.