Automation and advanced
Log analysis
What crawlers really do
What are crawlers actually fetching, and where is crawl budget going?
Server-log and crawl-budget analysis for large sites, starting with separating real crawlers from impostors.
Paste into Claude Code, Cursor, Codex or any agent that can open a link. If yours cannot, use Copy full text.
Reach for it when you say
- “Log file analysis”
- “Crawl budget”
- “What is Googlebot crawling?”
Installed as a skill, it triggers on requests like these by itself.
How it works
- Step 01Get the logs and identify the real crawlers
- Step 02Analyse
- Step 03Recommend, coordinated with the ladder
- Step 04Report
Use it
Three modes, three ways in
Run it in audit mode first: it records findings and changes nothing. Then fix applies only the findings you approve, and re-check re-tests them later. Switch modes on the prompt above.
- 01Paste the link promptOne line that points your agent at the skill on GitHub. Nothing to install, always the latest version.
- 02Paste the full textFor agents that cannot open links. The skill is self-contained, so this works in any chat.
- 03Install the packInstalled as skills, it triggers by itself when you describe the problem, and can hand work to the other skills.
Questions about this skill?
Ask me on LinkedIn, or open an issue on GitHub if something is wrong or missing.