> ## Documentation Index
> Fetch the complete documentation index at: https://docs.atomicagi.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawler logs

> Check which AI crawlers reach your website and how it responds.

<Frame>
  <img src="https://mintcdn.com/atomicai/Ry8tdAopCNKvP1zj/images/app-crawler-overview.png?fit=max&auto=format&n=Ry8tdAopCNKvP1zj&q=85&s=740f03ebf349da5ce4b570647bad2923" alt="Crawler logs in the Atomic AGI project workspace" width="1536" height="1024" data-path="images/app-crawler-overview.png" />
</Frame>

The screenshots use illustrative local data.

Open **Crawler logs** to inspect observed AI bot traffic. These are website requests, not AI answer citations or human referral visits. The page requires the website connection and the appropriate plan and Technical permission.

## Connect the website

If the page says `Connect your website to see crawler logs`, use `Connect website` and follow the connection panel. You need permission to change project data sources. If you can only read the page, ask a project administrator to complete setup. Choose the matching installation route: Cloudflare, Vercel, or WordPress. Follow its displayed instructions, then use the connection check. A created key alone is not proof that logs are arriving. Allow new requests to arrive before interpreting an empty period.

## Overview tab

Use the summary and crawl activity chart to identify changes in visits. Compare the platform, bot, folder, URL, and HTTP status breakdowns. Switch between Stacked bars, Line chart, and Pie chart to compare timing or share. Rising activity accompanied by 4xx or 5xx responses needs investigation before you assume improved discoverability.

### Stacked bars

Compare daily request volume by bot. A spike from one crawler should lead to a review of its URLs and response statuses.

<Frame>
  <img src="https://mintcdn.com/atomicai/Ry8tdAopCNKvP1zj/images/app-crawler-stacked.png?fit=max&auto=format&n=Ry8tdAopCNKvP1zj&q=85&s=1c2d22b29f5e3411b98efc1391c80236" alt="Crawler activity with Stacked bars selected" width="1536" height="1024" data-path="images/app-crawler-stacked.png" />
</Frame>

### Line chart

Follow changes over time for the selected bots. Check the underlying counts before treating a short spike as a sustained trend.

<Frame>
  <img src="https://mintcdn.com/atomicai/Ry8tdAopCNKvP1zj/images/app-crawler-line.png?fit=max&auto=format&n=Ry8tdAopCNKvP1zj&q=85&s=75211def6cf4ba446108c711518ed8e6" alt="Crawler activity with Line chart selected" width="1536" height="1024" data-path="images/app-crawler-line.png" />
</Frame>

### Pie chart

Compare each selected bot’s share of observed requests across the period. A large share measures crawling activity, not answer visibility.

<Frame>
  <img src="https://mintcdn.com/atomicai/Ry8tdAopCNKvP1zj/images/app-crawler-pie.png?fit=max&auto=format&n=Ry8tdAopCNKvP1zj&q=85&s=3117a28e7447958f063fbc568bf4e972" alt="Crawler activity with Pie chart selected" width="1536" height="1024" data-path="images/app-crawler-pie.png" />
</Frame>

## Pages tab

Open Pages to compare visited URLs, platforms, and response statuses. Filter to important pages or a folder to see whether crawlers reach the content you want discovered. A page absent from these logs has no observed visits in the selected scope; this does not prove that every AI provider is unable to access it.

<Frame>
  <img src="https://mintcdn.com/atomicai/Ry8tdAopCNKvP1zj/images/app-crawler-pages.png?fit=max&auto=format&n=Ry8tdAopCNKvP1zj&q=85&s=b7ff43783fc42bb814b52b3e9b5301d6" alt="Crawler logs Pages tab" width="1536" height="1024" data-path="images/app-crawler-pages.png" />
</Frame>

## Logs tab

Use Logs to inspect individual requests and their times, bot/platform, URL, and response. Narrow the date range and filters before paging through events. Correlate failed responses with website changes or access rules; a successful response does not guarantee inclusion in an AI answer.

<Frame>
  <img src="https://mintcdn.com/atomicai/Ry8tdAopCNKvP1zj/images/app-crawler-logs.png?fit=max&auto=format&n=Ry8tdAopCNKvP1zj&q=85&s=e6ce7f15445df767d65285d5a94a9088" alt="Crawler logs individual Logs tab" width="1536" height="1024" data-path="images/app-crawler-logs.png" />
</Frame>

## Where to go next

* [Data sources](/settings/project/data-sources)
* [SEO audit](/data/technical/seo-audit)
* [AI Visibility](/data/ai-search/visibility)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.