oddly The world, on the record.
Open a receipt

OddlyBot · our crawler

OddlyBot reads public pages, so the record can say what a source said and when.

It is the crawler behind oddly's public record: a sealed, timestamped account of what each source said, and when. It reads a page the way any visitor could, and only a page any visitor could.

Open a receipt How a receipt works →

Before anything else

It reads your robots.txt first, and does what it says.

OddlyBot asks a host for its robots.txt before any other page and follows the group written for OddlyBot, or the group for every crawler when there is none. It reads the file afresh on every visit, so a change applies from the next one. If your server cannot answer for its robots.txt, OddlyBot takes that as a no and asks again later.

At most one page a second

Slow on purpose.

Once it has read your robots.txt, OddlyBot asks your site for at most one page a second, and waits longer whenever your crawl-delay asks it to.

How to turn it away

One group in your robots.txt.

Write a group for OddlyBot and it is obeyed from the next visit. This one closes the whole site to it.

robots.txt

User-agent: OddlyBot
Disallow: /

narrower

A Disallow for one path closes only that path, and a Crawl-delay slows OddlyBot down without closing anything.

earlier name

A group already written for oddly-benchmark-bot, the name OddlyBot used before, is obeyed the same way.

a person

If you would rather ask someone, write to [email protected].

How to check a request is ours

Signed, with a key we publish.

When the key directory below lists a key, OddlyBot signs its requests with it, as the Web Bot Auth drafts describe. Each signature covers the host the request was sent to and names the directory, so a site can check a request that uses the name against the key we publish.

user agent

OddlyBot (+https://myoddly.com/bot; oddly-benchmark-bot)

key directory

https://myoddly.com/.well-known/http-message-signatures-directory

signature

Three header fields: Signature-Agent names the directory, Signature-Input says what was signed and when it stops being valid, and Signature carries the signature itself, tagged web-bot-auth.

What it does not do

Public pages, and nothing else.

Nothing OddlyBot reads is behind a login, and nothing it sends pretends to be anyone else.

no accounts

It never signs in, fills in a form or asks for a page that needs an account.

no disguise

It sends the same name on every request and never changes it to get past a block.

The key, read live

The public key OddlyBot's signatures are checked against.

Read at load from the key directory. While it lists no key, OddlyBot signs nothing, and every slot below says so rather than showing a key that is not in use.

directory
/.well-known/http-message-signatures-directory
key id
[no key published]
curve
[no key published]
public key
[no key published]

The key id is the key's thumbprint, which is what every signature names in its Signature-Input field. The private half never leaves the crawler.