Skip to main content

How do search engines find my pages?

There are several factors affecting how a search engine's indexer (or crawler) finds your pages.

As a first step, ensure that your pages are set to allow search engines:

  1. Go to Edit > Edit page properties.
  2. Select the checkbox Allow search engines.

External search engines

An external search engine (e.g. Google) has no special knowledge of a site's structure or content. Therefore, the search engine needs directing to the site's content.

The crawler starts at the home page, follows any links to other pages on the site, follows any further links it finds, and so on. One consequence is that any content not linked to by another page on the site, or linked to from another page elsewhere on the internet, is not indexed.

This is relevant to SiteBuilder pages too. Even if Show in local navigation and Allow search engines are selected in page properties, it will not be indexed when no other previously-indexed pages link to your page.

This problem can also occur if a page or file's parent page has the Allow search engines property unselected. (SiteBuilder alerts you when you select Allow search engines on a page with this problem.)

SiteBuilder's search engine

The SiteBuilder search engine indexes all pages and files regardless of whether they are linked to from any other page. (See the help article on what type of content is included in searches.)

The search engine also indexes all pages and files regardless of their view permissions. However, when a person performs a search, the results are filtered to only display content the person has permission to see.

For example, when a person who is not signed in performs a search, they only see publicly viewable pages in search results. If a member of staff is signed in and performs the same search, results include staff-only restricted pages and pages they explicitly have view permissions for.

There's a side effect of this system. Unselecting Show in local navigation in Page properties, and restricting the view permissions, does not hide the page from the indexer. The only way to hide the page from the search engine is by using the method described below.

Note: even though the indexer ignores the page's view permissions, a person can only see the content of that page if they have sufficient permissions.

How to prevent search engines indexing a page

  1. Go to the page you want to prevent being indexed.
  2. Go to Edit > Edit page properties.
  3. Ensure the Allow search engines checkbox is unselected. (The checkbox is selected by default when you create a new page because, generally, you do want search engines to index your pages.)
  4. Select the Save button.

Note: Even though this is a request to search engines – not an instruction they must obey – some badly-behaved search engines may ignore the request. SiteBuilder's search engine honours the request though, as do Google and most major search engines.

Related help articles


Email webteam at warwick dot ac dot uk
How to report a problem