In this article
“AI is stealing your website traffic”.
“People are not using search engines (Google) for getting information anymore; they’re now turning to AI.”
You might have come across these kinds of posts or discussions on the internet.
Encouraging businesses to quit SEO because of AI.
Are you also thinking of quitting SEO?
We have good news for you.
According to Ahrefs, about 63 out of every 100 websites today are getting visits from AI tools. Meaning?
Most of the information AI tools give their users is generated from information published on websites. And in most cases, they return their users to the websites.
You see? There’s no need to be scared of AI… you can use it to your advantage. How?
This is where the llms.txt file comes in.
By adding an llms.txt file to your website’s main folder (root directory), you give AI systems clearer instructions on how to read and process your content or information.
It helps AI overcome obstacles like unnecessary code or blocked sections, allowing it to understand your website better and give faster, more accurate answers to users.
What is llms.txt file?
This shouldn’t cause a headache.
The llms.txt file is a special text file that you place in the main folder (root directory) of your website.
Its purpose is to help artificial intelligence (AI) systems, like ChatGPT, Google Gemini, Claude, and Perplexity, better understand the structure and content of your website.
This file provides specific instructions or information that AI models can use to crawl and interpret your website more effectively.
For example, it can tell the AI which pages or sections are most important, guide how content should be processed, or even help the AI understand the context of your website better.
As a result, it ensures that AI can analyze and present your website’s information more accurately, improving how it responds to users queries or integrates your website’s content.
Currently, websites use files like robots.txt and sitemap_index.xml to help search engines know what’s on the website and where to look.
llms.txt is trying to do the same thing but for AI models.
Instead of AI systems having to guess what information is important, this file gives them a clear map, helping them focus on the high-value content that matters.
How to use llms.txt?
You want to use llms.txt file? That’s a smart move.
You should be smarter than your competitors (those thinking AI will steal all their visitors).
Here’s how to do it:
1. Prepare the file in markdown format
First, you need to create your llms.txt file in a simple text format called Markdown.
This format helps organize the content clearly and easy for AI to read.
Here’s an example:
# Your Website/Project Name
> A brief description of your website or project
## Documentation
- [Getting Started](/docs/getting-started) - Guide for new users
- [API Reference](/docs/api) - Complete API documentation
- [Tutorials](/docs/tutorials) - Step-by-step guides
## Examples
- [Basic Implementation](/examples/basic) - Simple integration example
- [Advanced Features](/examples/advanced) - Using advanced capabilities
## Optional Resources
- [Community Forum](/community) - Get help from other users
- [Change Log](/changelog) - Track updates and changes2. Place the File in the Correct Location2. Upload the file to the root directory
Once the file is ready, you need to upload it to your website’s root directory.
This is the main folder where your website files are stored. It’s like placing the file in the “home” folder of your website.
3. Reference it in the robots.txt file
To ensure that search engines and AI bots know the file is there, you’ll need to add a reference to it in your robots.txt file.
This tells AI systems where to find the llms.txt file on your website.
4. Test the file accessibility
After uploading the file, you should test it by typing the URL of the file directory into your browser (e.g., yourwebsite.com/llms.txt).
This checks if the file is accessible to AI systems and humans alike.
5. Set the right permissions for AI bots
You need to make sure that AI bots (like Gemini or ChatGPT) are allowed to access the llms.txt file.
This can be done by adjusting the settings in your robots.txt file and giving permission for these bots to read the llms.txt.
6. Regularly update and test the file
As your website content changes, the llms.txt file will need to be updated to reflect the new information.
It’s important to regularly update the file and test it with validation tools to make sure it remains accurate and works properly for AI systems.
llms.txt examples
A few sample llms.txt files in action:
- Block Agency: https://blockagency.co/llms.txt
- Perplexity: https://docs.perplexity.ai/llms-full.txt
- Anthropic: https://docs.anthropic.com/llms.txt
- Stripe: https://docs.stripe.com/llms.txt
- Mangopay: https://docs.mangopay.com/llms.txt
- Cloudflare: https://developers.cloudflare.com/llms.txt
- Docker: https://docs.docker.com/llms.txt
- Hugging face: https://huggingface-projects-docs-llms-txt.hf.space/accelerate/llms.txt
What is llms txt (without the dot)
If you’ve been searching for “llms txt” without the dot, don’t worry… you’re still in the right place.
Whether you search for “llms txt”, or “llms.txt”, they both refer to the same thing.
The dot before the “txt” just indicates that it’s a file type, like robots.txt or sitemap_index.xml.
But people often leave out the dot when they search online. So, if you’re wondering if “llms txt” is something different… it’s not. It’s all the same.
llms.txt generator or tools
Once you understand how the llms.txt file works, there are several tools available to help you create one (especially if you’re not comfortable building it from scratch).
These tools can scan your website and generate a properly formatted llms.txt file that highlights your most important content.
For small websites, many of these tools are free to use. But for larger or more complex websites, you might need to pay for a custom solution or build your own tool to collect and format the data.
However, be careful… not all these tools are secure.
Before using any generator, do your research to make sure it’s safe… especially if the tool asks for sensitive information or backend access.
Always check the file it creates before uploading it to your website.
Here are a few popular tools that can help:
- Markdowner: A free, open-source tool that turns your website content into clean Markdown format, which is ideal for llms.txt files.
- Appify by Jacob Kopecky: A dedicated generator built specifically for creating llms.txt files.
- Website LLMs plugins (for WordPress): This plugin automatically builds your llms.txt file. You just choose whether it should crawl your posts, pages, or both.
- FireCrawl: One of the earliest tools made for llms.txt file creation. It helps extract and organize key website content.
These tools can save you time, but always check their security and review the file before adding it to your website. Be careful.
Wrapping up
That’s almost everything you need to know about llms.txt.
If you don’t want to deal with the technical side, our team at Block Agency can take care of it for you.
Need help adding llms.txt file or improving your website?
Reach out to us here: hey@blockagency.co