The world, on the record.
Talk to us

OddlyBot · our crawler

OddlyBot reads public pages, so the record can say what a source said and when.

It is the crawler behind oddly's public record: a sealed, timestamped account of what each source said, and when. It reads a page the way any visitor could, and only a page any visitor could.

Open a receipt How a receipt works →

Before anything else

It reads your robots.txt first, and does what it says.

OddlyBot asks a host for its robots.txt before any other page and follows the group written for OddlyBot, or the group for every crawler when there is none. It reads the file afresh on every visit, so a change applies from the next one. If your server cannot answer for its robots.txt, OddlyBot takes that as a no and asks again later.

At most one page a second

Slow on purpose.

Once it has read your robots.txt, OddlyBot asks your site for at most one page a second, and waits longer whenever your crawl-delay asks it to.

Who runs it

OddlyBot is oddly's own crawler.

oddly operates it, and oddly is run by Oddly Even Group Pte. Ltd., Singapore and New Zealand.

operator

oddly, Oddly Even Group Pte. Ltd. (myoddly.com)

purpose

To read public sources as they are published, for oddly's public record of what each source said and when.

what a read becomes

One row on the record: what was read from the page, when the source said it and when OddlyBot looked. The row is sealed against the one before it, the day's rows are folded into one root, and that root is submitted to a chain we do not run. The method in full.

contact

[email protected], for anything about OddlyBot on your site.

How to turn it away

One group in your robots.txt.

Write a group for OddlyBot and it is obeyed from the next visit. This one closes the whole site to it.

robots.txt

User-agent: OddlyBot
Disallow: /

narrower

A Disallow for one path closes only that path, and a Crawl-delay slows OddlyBot down without closing anything.

earlier name

A group already written for oddly-benchmark-bot, the name OddlyBot used before, is obeyed the same way.

a person

If you would rather ask someone, write to [email protected].

How to check a request is ours

Signed, with a key we publish.

When the key directory below lists a key, OddlyBot signs its requests with it, as the Web Bot Auth drafts describe. Each signature covers the host the request was sent to and names the directory, so a site can check a request that uses the name against the key we publish.

user agent

OddlyBot (+https://myoddly.com/bot; oddly-benchmark-bot)

key directory

https://myoddly.com/.well-known/http-message-signatures-directory

signature

Three header fields: Signature-Agent names the directory, Signature-Input says what was signed and when it stops being valid, and Signature carries the signature itself, tagged web-bot-auth.

What it does not do

Public pages, and nothing else.

Nothing OddlyBot reads is behind a login, and nothing it sends pretends to be anyone else.

no accounts

It never signs in, fills in a form or asks for a page that needs an account.

no disguise

It sends the same name on every request and never changes it to get past a block.

The key, read live

The public key OddlyBot's signatures are checked against.

Read at load from the key directory. While it lists no key, OddlyBot signs nothing, and every slot below says so rather than showing a key that is not in use.

directory
/.well-known/http-message-signatures-directory
key id
tbuwwnImleQotdaILsbA80gQVa4X_miVp9Z6eKxZzqg
curve
Ed25519
public key
0Nle48KFWBBvjnoyP4yaxhpeoPlXrXjdfGs1yVEqxmk

The key id is the key's thumbprint, which is what every signature names in its Signature-Input field. The private half never leaves the crawler.

Read it. Build on it. Depend on it.

  1. Read it

    Free, with attribution.

  2. Build on it

    By the meter. Developers

  3. Depend on it

    By contract.

Talk to us