Allow search engines to index this page

Hello,

Editing a Page Title and Properties, under the Advanced Options, is a checkbox, "Allow search engines to index this page".

I couldn't find anything in the documentation on this for 4.0.  In 3.7, it was a radio button that defaulted to yes.

Should this default to checked?  Is there a global setting somewhere that overrides it?

Thanks
-Noah

Hello Noah,

You can control this from the "Advanced" options of each page.

Regards,
Ivan Dimitrov
the Telerik team

Dear Ivan

When talking about search engines are we talking about google and friends?

If yes what does it to to the page. Would not see a <meta tag?

If this is ment to be that the page can be searched by SF internal then it should defenitely be renamed.

Markus

Hi Ivan,

I know where it is located, but it is defaulting to unchecked.  Do you have to go through each page and enable it?

Thanks
-Noah

Hello Noah,

In SP1 the default value for Allow search engines to index this page is checked . You can set the value programmatically for each pages using the fluent API.

    App.WorkWith().Pages().ForEach(p =>
               
                    p.Page.Crawlable = true;
              
                ).SaveChanges();

Regards,
Ivan Dimitrov
the Telerik team

Dear Ivan

Can you answer my questions as well?

Markus

Hello Markus,

I believe that we are talking about the Sitefinity's search capabilities. All meta tags for the public engines can be added through page properties sections, but if the page is published and it is navigable it will be shown on the public side - this page will be accessible by the public users.

Regards,
Ivan Dimitrov
the Telerik team

Thanks Ivan

So please let me make two feature requests

1) Rename existing to "Allow Sitefinity search engines to index this page".

2) Make a second one "Allow external search engines to index this page".  which will set an meta robot noindex

I think 1) is important because when I hear someone talk about Search Engines I would assume he is talking about Google and freinds (SEO)

Markus

Hi Ivan,

I agree with Markus.  As the check box says search engines, plural, I would think it controlled external search engines such as Google.  I was under the impression that this was related to external search engines being able to index the page.  It would be helpful if the interface (and documentation which is missing on this topic), was updated to reflect which search engine(s) were affected.

Thanks
-Noah

@Noah

Thanks for backing me up

@Ivan

When you write 'I believe ....' does that mean you are not sure?

Markus

Hi Markus Berchtold,

This option is only for external search engines like Google which means that if it is enabled the page will be indexed by the external engines. 


Greetings,
Antoaneta
the Telerik team

Dear Antoaneta

Thanks for clearing that up. So it acutally makes sense.

This leaves the question how is indexing by search engines surpressed when using this option?

I dont see a mate robots noindex there.

How should it work?

Markus

PS: How about make an extrea feature make available for internal search engine then :-)

Hi Markus Berchtold,

If you uncheck this option the page will not be indexed by the search engines, but will still be indexed internally.

Kind regards,
Antoaneta
the Telerik team

Dear Antoaneta

<meta name="robots" content="noindex" /> is really present when unchecked. Sorry missed that.

Would it still make sense to have a secend option for SF internal search?

markus

Hello Markus ,

I believe that such an option is not needed at this moment but I will discuss it internally with the team to get more opinions.

Best wishes,
Antoaneta
the Telerik team

@Antoaneta

Is this functionality still working in 4.1?  I unchecked the "index" option and I'm not seeing a "noindex" meta tag.

Brett

Hi Brett ,

Yes, it is working. Could you double check please? If it is still not working, please tell us if the template of the page is based on a master page.

Best wishes,
Pavel
the Telerik team

Hi-

This functionality isn't working on the projects we are using templates based off of Master Page files.  I assume this is a bug? 

BW

Hello Brett,

Yes it is a bug, we'll fix it for the service pack. I am sorry for the inconvenience caused.

Best wishes,
Pavel
the Telerik team

We have started a discussion on this feature and would like to invite you to share your feedback here: 

www.sitefinity.com/.../exclude-pages-from-sitefinity-search

We will be happy to hear your specific cases and scenarios in regards to excluding pages from SiteFinity search. 

Thank you in advance.
Charl