AI Crawler Log Analyzer
See which AI crawlers appear in your server logs. Separate search, training and user-triggered fetches, verify published IP ranges, and inspect the URLs and responses behind each request.
Drop your server access log here
Apache, Nginx and structured log exports. .log · .txt · .csv · .json · .jsonl · .gz
Your log file is processed locally in your browser and is not uploaded to our servers.
Only the public IP registry is fetched. No account, log storage or analytics on this page.
Synthetic demo — example paths and reserved documentation IPs. Verification uses synthetic ranges, not the live provider registry. These are not neooenone.com logs.
Format & field mapping
For structured exports, select the original client IP and User-Agent fields. Ambiguous matches need your choice. Changes apply when you analyze again.
Choose a log or try the synthetic demo.
01 /Filter the evidence
- AI crawler requests
- —
- Verified requests
- —
- AI crawlers detected
- —
- Unique URLs crawled
- —
02 /Crawler purpose
03 /Requests over time
Charts and date filters use UTC. Rows without a usable timezone are excluded from time charts and date filters.
Time-series data table
04 /Crawler summary
05 /HTTP responses
403 Forbidden · 404 Not found · 429 may indicate rate limiting · 5xx Server error. A status code alone cannot identify the cause of the response.
06 /Top crawled URLs
07 /Crawler issues
08 /Verification report
IP mismatches do not automatically mean spoofing. Logs behind a CDN or reverse proxy may contain the proxy IP instead of the original client IP.
Verified means a known User-Agent and an IP in a published provider range. It does not prove historical identity, indexing, citations or AI visibility. Current ranges may differ from ranges used when an old log was recorded.
09 /Export filtered evidence
IP registry & sources
Ranges refresh at most once every 24 hours per source. A last successful snapshot is used if a source is unavailable; snapshots older than 30 days are not used for verification.
What is an AI crawler log analyzer?
This tool reads access-log evidence to show which supported AI crawler User-Agents requested your URLs, how often they visited, and which HTTP responses they received. It measures the uploaded requests, not search rankings or citations.
What AI crawlers can this tool detect?
OpenAI: OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot. Anthropic: Claude-SearchBot, ClaudeBot and Claude-User. Perplexity: PerplexityBot and Perplexity-User. Matching uses explicit tokens and is case-insensitive. Googlebot is not classified as Gemini or Google-Extended traffic.
Search crawlers vs training crawlers
Search bots collect material for search-related discovery. Training bots gather content that may support model development. User fetchers retrieve pages following a user action. Ads validation is reported separately. One category does not establish activity in another.
How crawler verification works
A User-Agent is a claim, not proof. The analyzer compares the logged client IP with the range list for that crawler or provider. A matching range is Verified; missing IPs or unusable ranges remain UA match. IP mismatch can also mean your log recorded a CDN or proxy address. Check the source fields and registry date.
How to find your server access logs
Nginx logs
Check your access_log configuration or hosting control panel. Common locations include /var/log/nginx/access.log. Use a log format that includes the request User-Agent; a plain Common log does not contain it.
Apache logs
Check CustomLog in your virtual host configuration. Combined logs usually include the request, referrer and User-Agent. Paths depend on your server and hosting provider.
CDN / proxy logs
Export request-level logs from your provider. Map the original visitor IP field, not a reverse proxy address. This tool does not automatically trust X-Forwarded-For chains or contact a domain from your log.
What HTTP status codes should you check?
Compare successful 2xx responses with redirects, 403, 404, 429 and 5xx responses. A 404 is not evidence of blocking; a 429 may indicate rate limiting. Investigate the URL and server configuration before drawing a conclusion.
Privacy
Log contents stay in this page and its local worker. They are not sent to a server, an LLM or analytics, and are not written to cookies or browser storage. Clear the file or leave the page to release it. CSV files are created locally when you request an export.
Formats & limits
UTF-8 Combined/Common logs, CSV with a header, JSON objects, JSONL and arrays of objects are supported. Nested JSON fields can be mapped. Gzip requires browser streaming decompression. Records over 1 MiB are skipped and counted; an aggregation memory guard asks you to narrow filters instead of presenting incomplete results. Duplicate requests are counted as separate log entries. Query strings and encoded paths are preserved.