While AddSearch bots can easily crawl most sites, the content you’d like to crawl may be behind a login. AddSearch supports indexing content behind authentication and securing your index in such cases. For more information on securing your index, take a look at our documentation on using JWT tokens.

For authenticated crawling, we support the following login mechanisms:

• Basic authentication
• Form-based authentication

If you would like to set up the authenticated crawling for your account, contact AddSearch Support to proceed. If you are using a login method not noted above, feel free to reach out to us to discuss possible ways to set up the crawling for your protected areas.

Another way to allow AddSearch crawlers to access your private content is whitelisting our bots, for examples, by the IP addresses.

An alternative way of indexing private content is pushing it to the index with our Indexing API This allows you to have full control over what information is indexed and how it is stored in the index.