By default, AddSearch includes all pages of your website in your search. You might want to exclude certain parts of your site from the search in certain situations. Also, sometimes, you might want to make sure that some specific content on your page gets included in or excluded from the search.
If you don’t want certain site areas or pages to appear in the search, you can do that in multiple ways:
You can exclude content from the search without touching your site code, by logging in to the AddSearch Dashboard and using the Site Areas tool under Manage results. This is the easiest method.
To exclude all pages located under /temp/, just enter /temp/ into the text field and select Exclude from the dropdown box. Changes, that are done in the Site Areas, take effect with the next recrawl of your site, which for large sites might take some time.
You can also use robots.txt, which is a text file that resides in the root folder of your website. To make AddSearch and other search engines exclude pages under /temp/ from search, insert the following lines into your robots.txt file:
User-agent: *
Disallow: /temp/
To make only AddSearch exclude pages under /temp/, insert the following lines into your robots.txt file:
User-agent: AddSearchBot
Disallow: /temp/
You can also use robots meta tags and the index/noindex and follow/nofollow rules. These need to be placed in the HEAD section of your page.
To make AddSearch and other search engines exclude a page from the search, insert the following into the HEAD section of that page:
<meta name="robots" content="noindex">
To make AddSearch and other search engines exclude all pages from the search that a page links to (i.e. so that the crawler won’t follow links from this page) insert the following into the HEAD section of that page:
<meta name="robots" content="nofollow">
You can also combine instructions. To make AddSearch and other search engines follow links from this page, but exclude this page from the search, insert the following into the HEAD section of the page:
<meta name="robots" content="follow, noindex">
If you want to have the robots meta tags apply to AddSearch only, change name="robots"in the above examples to name="AddSearchBot".
Finally, you can use the rel="nofollow" attribute in links, to exclude the page you’re linking to from the search. Include the rel="nofollow" attribute in the link like this:
<a href="/content/excluded_page" rel="nofollow">Link to page that is excluded from search</a>
AddSearch detects automatically the main content in your pages and indexes only that main content. This makes the search results better, as superfluous content (such as sidebars, headers, footers, etc.) and duplicate content is not included in the search.
Sometimes you might want to manually make sure that certain content is included in or excluded from the search. You can insert HTML5 tags to your pages to mark such content.
To make AddSearch include a certain section in your page into the search, use the data-addsearch="include" attribute in any element, typically a DIV or P. Example:
<div data-addsearch="include">Content that is included in the search</div>
To make AddSearch exclude a certain section in your page from the search, use the data-addsearch="exclude" attribute in any element, typically a DIV or P element. Example:
<div data-addsearch="exclude">Content that is excluded from the search</div>
If you have multiple versions of a web page and want to tell the crawlers to index only one you can use canonical links. For detailed information on canonical links visit the here.
AddSearch allows you to define parameters that should be ignored when our crawlers collect new links. Read more about setting up the ignored parameters here
We’re always happy to help with code or other questions you might have. Search our documentation, contact support, or connect with our sales team.