Web scraper rules: AGENTS.md, CLAUDE.md & Cursor rules
Scrapers get you blocked, or worse, when they hammer a site. These rules keep yours polite, cached, and easy to fix when the page layout changes.
Stack these rules assume: Python 3, requests + Beautiful Soup (a headless browser only if truly needed). Different stack? Customize it in the generator.
Quick start
- Copy the
AGENTS.mdbelow into your project's root folder. - Fill in the three
[bracketed]lines under "Project". Delete any rule that doesn't fit. - Start your first chat with: "Read AGENTS.md and follow it for this whole session."
AGENTS.md (works across tools)
Read by OpenAI Codex, Cursor, GitHub Copilot, Claude Code (when there's no CLAUDE.md) and many other agents.
# AGENTS.md: Web scraper
Instructions for AI coding agents working on this project. Replace anything in [brackets].
## Project
- What it is: [one sentence: what you are building and who it is for]
- Stack: Python 3, requests + Beautiful Soup (a headless browser only if truly needed)
- Version 1 only needs to: [the ONE thing it must do]
## Commands
- Create a virtual environment: `python3 -m venv .venv && source .venv/bin/activate`
- Install: `pip install requests beautifulsoup4`
- Run: `python scrape.py`
## Project rules
- Check for an official API or data export first and tell me if one exists.
- Respect robots.txt and the site's terms. Wait at least 1 second between requests and send a descriptive User-Agent.
- Cache downloaded pages to a local folder and parse from the cache while developing.
- Keep all CSS selectors in one place at the top of the file.
- Use plain requests first. Ask before adding a headless browser.
- Don't collect personal data (emails, phone numbers, profiles) unless I explicitly ask and confirm it's allowed.
## How to work
- Do one task per request. If my request is really several tasks, list them and do only the first.
- Before writing code, give a 3-line plan: what you'll change, which files, and how I'll test it.
- Make the smallest change that works. Don't refactor, rename, or reformat code I didn't ask about.
- Ask before adding a dependency, deleting or moving files, changing the data/database structure, or touching more than 3 files.
- If you're not sure a library, API, or setting exists in the version we use, say so and check the installed package or the docs instead of guessing.
- Never say something works unless you ran it or I confirmed it. Report errors exactly.
- Secrets (API keys, tokens, passwords) live only in environment variables or a .env file listed in .gitignore. Never print, hardcode, or commit them.
- Never run destructive commands (rm -rf, git reset --hard, force-push, dropping tables) without asking first.
## Finish every task with
1. What changed (list the files).
2. How to run or test it in under a minute, and what I should see.
3. A suggested git commit message.
Want a step-by-step kit?
Our paid templates ($6 to $9) add a full prompt sequence and a worked example on top of rules like these, for games, Discord bots, email triage, MVPs and fixing messy code.
Why each rule exists
- Web scraper ruleCheck for an official API or data export first and tell me if one exists.Why: An API is faster, more reliable, and won't break when the site redesigns.
- Web scraper ruleRespect robots.txt and the site's terms. Wait at least 1 second between requests and send a descriptive User-Agent.Why: Polite scrapers don't get blocked, and you stay on the right side of the site owner.
- Web scraper ruleCache downloaded pages to a local folder and parse from the cache while developing.Why: You'll re-run the parser dozens of times. Don't re-download the site dozens of times.
- Web scraper ruleKeep all CSS selectors in one place at the top of the file.Why: When the site changes its layout, you fix one block instead of hunting through the code.
- Web scraper ruleUse plain requests first. Ask before adding a headless browser.Why: Headless browsers are slower and heavier. Most pages don't need one.
- Web scraper ruleDon't collect personal data (emails, phone numbers, profiles) unless I explicitly ask and confirm it's allowed.Why: Scraping personal data brings privacy-law and terms-of-service risk.
- How to workDo one task per request. If my request is really several tasks, list them and do only the first.Why: Big multi-part requests are where AI tools change 12 files and you can't tell which change broke things.
- How to workBefore writing code, give a 3-line plan: what you'll change, which files, and how I'll test it.Why: A plan you can read in 10 seconds catches the wrong approach before it costs you an hour or a pile of credits.
- How to workMake the smallest change that works. Don't refactor, rename, or reformat code I didn't ask about.Why: Drive-by "cleanups" are the #1 cause of "it was working yesterday".
- How to workAsk before adding a dependency, deleting or moving files, changing the data/database structure, or touching more than 3 files.Why: These are the changes that are hard to undo. One quick question is cheaper than a rollback.
- How to workIf you're not sure a library, API, or setting exists in the version we use, say so and check the installed package or the docs instead of guessing.Why: AI tools confidently invent functions and config options. Making it check is the cure.
- How to workNever say something works unless you ran it or I confirmed it. Report errors exactly.Why: "Should work now" isn't a test result. You want facts, not optimism.
- How to workSecrets (API keys, tokens, passwords) live only in environment variables or a .env file listed in .gitignore. Never print, hardcode, or commit them.Why: Leaked keys get found by bots within minutes of a public push. This rule is the cheapest insurance you'll ever buy.
- How to workNever run destructive commands (rm -rf, git reset --hard, force-push, dropping tables) without asking first.Why: One bad command can delete the only copy of your work.
- Finish every task withWhat changed, how to test it, and a commit message.Why: This turns every task into a checkpoint: you test it, you commit it, and you always have a last good version to go back to.
Use it as CLAUDE.md (Claude Code)
Pick one:
- Do nothing. Current Claude Code versions read
AGENTS.mdwhen the project has noCLAUDE.md. Run/memoryin a session to check it loaded. - Import it (best if you want Claude-only extras, or you're on an older Claude Code). Create
CLAUDE.mdnext toAGENTS.md:
@AGENTS.md
## Claude Code
- Use plan mode before any change that touches more than 3 files.
- Claude only? Save the same AGENTS.md text as
CLAUDE.mdinstead. Don't keep two separate copies; they drift apart.
Heads-up: if a project has both files and CLAUDE.md doesn't import AGENTS.md, Claude Code reads only CLAUDE.md by default. More in AGENTS.md vs CLAUDE.md.
Use it as Cursor rules (.cursor/rules/*.mdc)
Cursor reads AGENTS.md too, so the file above already works. If you prefer Cursor's own Project Rules, split it into two .mdc files. The .mdc extension and frontmatter are required; Cursor ignores plain .md files in .cursor/rules.
---
description: Project overview and working rules for Web scraper
alwaysApply: true
---
# Web scraper
Instructions for AI coding agents working on this project. Replace anything in [brackets].
## Project
- What it is: [one sentence: what you are building and who it is for]
- Stack: Python 3, requests + Beautiful Soup (a headless browser only if truly needed)
- Version 1 only needs to: [the ONE thing it must do]
## Commands
- Create a virtual environment: `python3 -m venv .venv && source .venv/bin/activate`
- Install: `pip install requests beautifulsoup4`
- Run: `python scrape.py`
## How to work
- Do one task per request. If my request is really several tasks, list them and do only the first.
- Before writing code, give a 3-line plan: what you'll change, which files, and how I'll test it.
- Make the smallest change that works. Don't refactor, rename, or reformat code I didn't ask about.
- Ask before adding a dependency, deleting or moving files, changing the data/database structure, or touching more than 3 files.
- If you're not sure a library, API, or setting exists in the version we use, say so and check the installed package or the docs instead of guessing.
- Never say something works unless you ran it or I confirmed it. Report errors exactly.
- Secrets (API keys, tokens, passwords) live only in environment variables or a .env file listed in .gitignore. Never print, hardcode, or commit them.
- Never run destructive commands (rm -rf, git reset --hard, force-push, dropping tables) without asking first.
## Finish every task with
1. What changed (list the files).
2. How to run or test it in under a minute, and what I should see.
3. A suggested git commit message.
The project rules only attach when a matching file is in the chat (globs with alwaysApply: false). Want them on all the time? Change it to alwaysApply: true.
---
globs: **/*.py
alwaysApply: false
---
# Project rules: Web scraper
- Check for an official API or data export first and tell me if one exists.
- Respect robots.txt and the site's terms. Wait at least 1 second between requests and send a descriptive User-Agent.
- Cache downloaded pages to a local folder and parse from the cache while developing.
- Keep all CSS selectors in one place at the top of the file.
- Use plain requests first. Ask before adding a headless browser.
- Don't collect personal data (emails, phone numbers, profiles) unless I explicitly ask and confirm it's allowed.
Use either AGENTS.md or these .mdc files, not both, so you don't send the same rules twice. See Cursor rules vs AGENTS.md.