NoviaHub-News, the crawler behind NoviaHub's AI News

NoviaHub-News collects AI news for NoviaHub's AI News section. It reads the public feeds, list pages and articles of AI companies, research labs and technology media, writes a short AI summary of each item and links back to the original.

Its User-Agent is NoviaHub-News/1.0 (+https://noviahub.com/developers/news/about). In robots.txt its name is noviahub-news.

It follows robots.txt (RFC 9309): the rules written for noviahub-news, else those for *. When a site's robots.txt cannot be read (a server error, a 429 or a network error) it fetches nothing from that site. It never signs in, never gets around captchas or protections such as Cloudflare's, and never pretends to be a browser.

It visits rarely: each source's feed or list page every 1 to 12 hours, and a new article's page usually once. NoviaHub shows titles, AI summaries and links to the originals; never full texts or images.

To keep it away from your site, add this to your robots.txt:

User-agent: noviahub-news
Disallow: /

NoviaHub's developers pages have a second crawler, NoviaHub-Developers (noviahub-developers in robots.txt), which reads public APIs such as GitHub's and open-source repositories. It follows robots.txt the same way. For questions or to have something removed, write to [email protected].