Bots still indexing files not in sitemap

Posted by Community Admin on 03-Aug-2018 18:43

Bots still indexing files not in sitemap

All Replies

Posted by Community Admin on 15-Feb-2011 00:00

The SEO people upstairs are using an SEO tool to give suggestions about pages.  The software is finding issues with pages that aren't found in the sitemap.xml file.
My thought is that there is a directory in Sitefinity that is referencing these pages, and isn't being dis-allowed in the robots.txt file.  Is this true, and if so, what is the directory?
Otherwise, what could be the reason for this?

Note:  The robots.txt isn't dis-allowing anything at the moment, it just references the sitemap.xml.

Thanks, Joe

Posted by Community Admin on 17-Feb-2011 00:00

Hi Joe,

The fastest way for solving the issue would be to disable the access for bots to those directories. Otherwise we will have to go through every template and check if there are links pointing to the dialogs.

Greetings,
Georgi
the Telerik team

This thread is closed