It might happen that you receive an error message when trying to create a new search demo. That means that our crawler couldn’t index your website.
There are a number of possible issues that might have prevented AddSearch from properly crawling/indexing your site.
The most common errors are mentioned below. If none of these apply, contact us for assistance.
Your website might have a .txt file, located at your-domain.com/robots.txt, that automatically gives instructions to web crawlers. It could be that your robots.txt tells our crawler that it is forbidden to crawl the site.
Change your robots.txt file to allow our crawlers to enter:
# Allow AddSearch crawler
User-agent: AddSearchBot
Disallow:
# Block all other crawlers (if you prefer this)
User-agent: *
Disallow: /
Read more here: http://www.robotstxt.org/robotstxt.html
Your website might also contain an HTML tag, most likely in the <head> section of your page, which forbids our crawlers to enter the site. In that section, there will then be a meta tag, looking like this:
<meta name=”robots” content=”noindex”>
If you want our crawlers to enter your site, then you can either remove this tag or replace the “noindex” with “index” so that it looks like this:
<meta name=”robots” content=”index”>
This will allow our crawlers to enter your site and subsequently index it properly.
If you want to allow AddSearch bot only and restrict all other crawlers, use the following combination of rules:
<meta name="robots" content="noindex" />
<meta name="AddSearchBot" content="index" />
It is possible that part of the HTML code on your website is incomplete or incorrect. This might not even cause any visual problems, so it could appear that your site is working fine, while for our crawlers it is impossible to crawl and index the site properly.
You can do an automatic check of your code on http://validator.w3.org/.
If your website has a main menu that is dependent on JavaScript, this could also cause problems for our crawlers. This might not only be the case with AddSearch’s crawlers, but also with those of web search engines, such as Google. Therefore it is not a practice that is widely recommended for website owners.
Some websites generate content dynamically using libraries such as AJAX and React. In these cases, crawling may fail. If the website you’d like to crawl generates content dynamically, our support team can enable AJAX crawling.
It could be that you have such a small website, that our crawlers have overlooked your pages and reported the link back as empty. Contact us to find out if this was the case.
We’re always happy to help with code or other questions you might have. Search our documentation, contact support, or connect with our sales team.