HomeRoadmapCommunity forum
Knowledge base
Go to Uplisting
Start typing to search…

Site Configuration Files & IndexNow

Site Configuration Files

All published Website Builder sites include a default sitemap.xml, robots.txt, and llms.txt files. Custom files, including other types, can also be uploaded to the root of your website for further customization.

📘

WARNING

As of December 2023, Google deprecated the ability for sitemaps to be automatically submitted. To ensure prompt indexing, it's advised to manually submit your sitemap using Google Search Console (for more information, please visit Improve your performance on Google Search). Alternatively, consider using our app designed to assist with this process, Google Search Console.

Default Sitemap.xml and Robots.txt Files

To access the sitemap.xml and robots.txt files:

  1. Open your published URL and add the desired file (either sitemap.xml or robots.txt) at the end (e.g., https://www.example.com/sitemap.xml or https://www.example.com/robots.txt).
  2. Enter the modified URL into your browser.
  3. Save the file or link as needed.
📘

NOTE

• Only published sites have these files.

• These files are note able to be edited, though a custom one can be uploaded, see the section, Replace Default Files, in this article for more information.

Sitemap.xml Specific Information

A sitemap lists all of the URLs for your site, including eCommerce category pages and product URLs (for both native and third-party stores), as well as blog posts and dynamic page records.

The sitemap cannot be edited, though a custom one can be uploaded, see the section, Replace Default Files, in this article for more information.

📘

WARNING

As of December 2023, Google deprecated the ability for sitemaps to be automatically submitted. To ensure prompt indexing, it's advised to manually submit your sitemap using Google Search Console (for more information, please visit Improve your performance on Google Search). Alternatively, consider using our app designed to assist with this process, Google Search Console.


Default Sitemap.xml and Robots.txt Files

To access the sitemap.xml and robots.txt files:

  1. Open your published URL and add the desired file (either sitemap.xml or robots.txt) at the end (e.g., https://www.example.com/sitemap.xml or https://www.example.com/robots.txt).
  2. Enter the modified URL into your browser.
  3. Save the file or link as needed.

📘

NOTE

• Only published sites have sitemaps.

• Pages that are set to not index (located in the page's SEO settings) will not appear in the sitemap. If the entire site is set to not index in the site's settings, the sitemap will be empty.

• If your sitemap contains non-secure URLs (HTTP instead of HTTPS), consider requiring visitors to use a secure connection (HTTPS).To enable the Force visitors to use secure connection (HTTPS) setting, in the side panel, click SEO & Settings, and then click the Site SSL tab.

LLMS.txt File

A LLMS.txt file is a simple text file that provides LLMs, such as ChatGPT, with a structured overview of a website's most important content. The file includes a list of the site's URLs, along with brief descriptions, making it easy for LLMs to scan and understand.

Website Builder creates and updates a LLMS.txt file for every site that is published. The file is in markdown format, and contains a list of pages that are available in the site, with descriptions and URLs.

All sites will have this via www.sitedomain.com/llms.txt.

The following pages are included in the file:

• Regular pages

• Dynamic pages

• Blog posts

• Translated pages

• Native store category pages

• Native store product pages


Replace Default Files or Add Custom Files

To override the default files (sitemaps and robots.txt files), ensure the Source URL matches the file names (sitemap.xml and robots.txt) exactly. Supported file types include .xml, .txt, and .json. Note that the .txt files can be any .txt file and not just ads.txt or robots.txt.

📘

WARNING

• Keep in mind the Website Builder platform does not validate custom files, so even if a file is corrupt, it will still be served.

📘

NOTE

Only site configuration file types are supported (for example .txt and .xml).

From SEO Settings

To upload from the SEO Settings:

  1. In the side panel, click SEO.
  2. Click the Additional Settings tab.
  3. Locate the Indexing & Crawling section, then click Go to URL Redirect (you will be directed to the URL Redirect screen with the File option pre-selected).
  4. In the Source URL field, enter the file path you would like the file to have. For example, if you enter in ads.txt, the path for it on the live site will be www.mydomain.com/ads.txt. Note that the .txt files can be any .txt file and not just ads.txt or robots.txt.
  5. Click + Upload file.
  6. Ensure the Redirect type is set to 2xx OK (Redirect) and click Add.
  7. Publish or republish your site to make the changes live.

From URL Redirects

To upload in URL redirects:

  1. In the side panel, click More, then click Settings.
  2. Click URL Redirect.
  3. Click Add New Redirect.
  4. In the Source URL field, enter the file path you would like the file to have. For example, if you enter in ads.txt, the path for it on the live site will be www.mydomain.com/ads.txt. Note that the .txt files can be any .txt file and not just ads.txt or robots.txt.
  5. Select the File option. If you don't see the File option on the URL Redirect screen, access it initially from the SEO settings and upload a file. Once done, you can continue uploading additional files directly from the URL Redirect screen.
  6. Click + Upload file.
  7. Ensure the Redirect type is set to 2xx OK (Redirect) and click Add.
  8. Publish or republish your site to make the changes live.

Revert Back to Website Builder Generated Files

If you wish to revert to using Website Builder's default sitemap or robots.txt files:

  1. In the side panel, click More, then click Settings.
  2. Click URL Redirect.
  3. Locate the file you want to remove and click Delete.
  4. Publish or republish your site for the changes to take effect.

IndexNow

Website Builder sites automatically support and use IndexNow, a protocol that ensures your website’s most recent content is immediately crawled and indexed by supporting search engines instead of passively waiting for web crawlers to find it. Currently, Bing and Yandex support this new protocol.

Without IndexNow, it can take anywhere from several hours to several weeks for search engines to index new or updated content. While waiting for search engines to index your content, it will not be discoverable via search. This delay can cause you to miss out on opportunities to engage with your online audience.

📘

NOTE

Just because your content is indexed faster doesn’t mean you’ll rank better in search results. You still need to create original, high-quality, SEO-optimized content and follow best practices for web content if you want to achieve the top positions in search engine results pages.

How IndexNow Works with Website Builder Sites

You don’t need to do anything to begin leveraging the benefits of IndexNow, your Website Builder site will do it for you automatically. This saves you time and makes it accessible for those who do not have the technical skills to interact with IndexNow’s API.

Website Builder supports the IndexNow API for you by:

• Discovering which pages have updated content and submitting only those to the IndexNow API.

• Fulfilling IndexNow security measures by generating a unique token per website and submitting that to search engines. This ensures that no one can spoof updated content for your site.

• Automatically submitting updated content on your site.

If you don’t want your context indexed by search engines, you can turn off indexing in the page settings of each page. For instructions on how to do this, see Single Page Management.


Updated 18 days ago

What’s Next