NYCLearningEventsBot
This crawler builds a public calendar of learning events in New York City (talks, lectures, readings, walks, workshops and classes). Every request our crawler sends itself carries this User-Agent:
NYCLearningEventsBot/0.1 (+https://thinkingnyc.com/bot)
What we fetch
- Public event calendars and feeds (iCalendar, RSS and Atom, calendar APIs) and the public event pages of New York City organizations.
- Only pages anyone can open without signing in. Our own requests are GET only: we never submit forms or create accounts.
- Each calendar is normally checked at most three times a day, with conditional requests (ETag, Last-Modified), so an unchanged calendar costs one short response (plus robots.txt).
- Some pages show their events only after running JavaScript. We read those through a rendering service, Firecrawl, which loads the page in a browser. Its requests come from the Firecrawl servers with a browser User-Agent, not ours. We ask for a render only after our own request, on the same check, has fetched the page with robots.txt allowing it, so disallowing NYCLearningEventsBot in robots.txt stops both.
robots.txt
We follow robots.txt for the product token NYCLearningEventsBot (or *), including Disallow and Crawl-delay. If robots.txt cannot be reached (a server error or a network failure), we do not crawl the site until it can. A Crawl-delay above 60 seconds means we stop checking that site.
Politeness
- One request at a time per site, at least 1 second apart, or your Crawl-delay if that is longer.
- Responses are capped at 8 MB and 20 seconds; at most 5 redirects are followed.
No evasion
We do not try to get around blocks. We never solve CAPTCHAs, rotate IP addresses or use stealth proxies, and our own requests always identify themselves with the User-Agent above. A bot challenge, a 401 or a 403 counts as a blocked answer. After 3 blocked answers in a row we mark the calendar blocked and stop checking it, and we never send a blocked page to the rendering service.
Contact
To ask us to slow down, stop, or correct a listing, send us a message. Adding User-agent: NYCLearningEventsBot with Disallow: / to your robots.txt also works: we honor it on our next check.