Skip to content
Assize Data
  • Services
  • About
  • Team
  • Blog
  • Jobs
  • Contact

Legal

  • Privacy notice
  • Data protection
  • Cookies
  • Website terms
  • Complaints
  • Security and data handling
  • Cyber Resilience Pledge
  • Data sources and licensing
  • AssizeBot, our crawler
  • Modern slavery statement
  • Accessibility

AssizeBot, our crawler

Last updated 8 November 2026

AssizeBot is the crawler Assize Data uses to collect the public sources behind our research and publications. If you have found this page in your server logs, this explains what it was doing and how to contact us.

How to recognise it

Every request it makes carries this User-Agent:

AssizeBot/0.1 (+https://assizedata.co.uk/bot; bot@assizedata.co.uk)

Requests to The National Archives’ Find Case Law also name the licence we use it under. AssizeBot runs on our own server. Anyone can copy a User-Agent, so if you think a request claiming to be AssizeBot is not ours, email us its time and IP address and we will check.

What it reads

  • Courts and judgments. The court lists HM Courts & Tribunals Service publishes on Court and Tribunal Hearings (magistrates’ courts, the Single Justice Procedure and civil daily cause lists), the judgments The National Archives publishes on Find Case Law, and the guidance and practice documents on judiciary.uk.
  • Public registers and official data. Companies House data and filed accounts, the ICO’s register of fee payers, data.police.uk’s monthly archive, the certificated enforcement agents register, and datasets and decisions on GOV.UK, such as prison and probation statistics, the unclaimed estates list and teacher misconduct decisions, and the Bank of England’s daily exchange rates.
  • Inspection and investigation reports. Reports of the prisons and probation inspectorates, the Prisons and Probation Ombudsman and Independent Monitoring Boards, and coroners’ reports to prevent future deaths.
  • Legal websites. The public pages of the websites of law firms, barristers’ chambers, other legal professionals, regulators and legal bodies in the UK: their home pages, policies, prices, news, careers pages and lists of the people who work there. This is how we study how the legal sector works and changes. We take names and roles from people pages, and no email addresses or telephone numbers.
  • Legal education and research. Universities’ course, module and staff pages for law and related subjects, open research from OpenAlex and university repositories, and the publications of campaign groups in the justice sector.
  • News. The RSS and Atom feeds that newspapers, the legal press, GOV.UK and others publish, read about once an hour and only when they change, and the GDELT Project’s free news index. We keep headlines and links to stories about courts, justice and the legal sector for our own reading, and only count reports of crimes, inquests and accidents.

It reads public pages and files only. It does not log in, and it does not get past paywalls, CAPTCHAs or other blocks.

How it behaves

  • robots.txt. On universities’, campaign groups’, public bodies’ and legal websites it reads robots.txt first and obeys it, including any crawl delay. It never fetches a page robots.txt refuses some other way. The one exception is Court and Tribunal Hearings, whose robots.txt asks crawlers not to read its court lists: we read them once a night while HMCTS considers our application for a licence for court list data.
  • One request at a time. It makes at most one request a second to a website, and fewer where a site asks. The busiest exceptions are services built for bulk use: Find Case Law at up to 2.5 requests a second, inside The National Archives’ guidance, and Court and Tribunal Hearings with half a second between requests. The enforcement agents register gets one request every six seconds, spread over several days, and a legal website gets one request every four seconds, at most 60 pages at a visit. If there is more to read it rests before coming back: 15 minutes in UK working hours, five on evenings and weekends, and 30 if the website is answering slowly.
  • Pages that need JavaScript. A few legal websites show nothing without JavaScript. For those pages, after an ordinary request, it opens the page in a headless browser with the same User-Agent. The browser loads the page, its scripts and the data those scripts read, and nothing else: no images, styles, fonts or video. It asks a website for at most one thing a second, and for each script once. It only reads, so it sends no forms. It does not run analytics, advertising, chat or bot-check code, so it should not appear in your visitor figures. robots.txt applies to everything it asks your website for, and a challenge page counts as a refusal.
  • Only what is new. It keeps what it has already read and fetches a file or list again only when it has changed. Most sources are read once a night or once a day, and university and campaign group websites once a week. On legal websites, home and news pages are read about once a day and the rest from weekly to every couple of months.
  • It backs off. It slows down when a server is struggling or asks it to (HTTP 429 or 5xx), following any Retry-After header. After three refusals in a row it leaves a website alone until its next run. A legal website that keeps refusing is asked less and less often, up to three months apart, and if the enforcement agents register refuses it, it waits a week.

Some of these sources contain personal data, such as the names in court lists and judgments. Our privacy notice explains what we hold, why, and your rights, and our page on data sources and licensing explains the terms we use sources under.

Slowing it down or stopping it

To keep AssizeBot out of all or part of your website, add a rule for it to your robots.txt:

User-agent: AssizeBot
Disallow: /

It picks up the change on its next visit. A Crawl-delay line slows it down instead. If you email us, we can also keep your website out for good, whatever list it turns up on.

If AssizeBot is causing you a problem, or you would rather we asked first, email bot@assizedata.co.uk or use our contact page. Tell us your website and what you would like us to do. We reply within two working days.

Assize Data

Assize Data provides data analysis for the UK legal sector, built on primary sources.

This page: about 0.013 g CO₂e on a first visit (107 kB). How we estimate this

Company

  • Home
  • Services
  • About
  • Team
  • Blog
  • Jobs
  • Contact
  • Our brands
  • Roadmap
  • Accreditations
  • Sustainability

Legal

  • Privacy notice
  • Data protection
  • Cookies
  • Website terms
  • Complaints
  • Security and data handling
  • Cyber Resilience Pledge
  • Data sources and licensing
  • AssizeBot, our crawler
  • Modern slavery statement
  • Accessibility

Assize Data Ltd is a private limited company registered in England and Wales, company number 17510115. Registered office: c/o IWN Accountancy, The Echo Building, 18 Albert Road, Bournemouth BH1 1BZ.

Assize Data Ltd is not a law firm and is not regulated by the Solicitors Regulation Authority or the Bar Standards Board. It does not carry out reserved legal activities for others, and it does not give legal advice.

© 2026 Assize Data Ltd