Include external websites in Sitefinity search indexes
Is there a way to accomplish this? The aim is to be able to include in a search index for a certain Sitefinity site pages from a website that is not in Sitefinity.
We're hoping to centralize our search results in the main Corporate site (which would be in Sitefinity 4.0) and that means we must find a way to "crawl" Sitefinity's search function into our other sites under the same brand which are obviously not managed by the CMS. Thanks for the help as always. :)
Hello jkregala,
Out of the box you cannot search external pages from Sitefinity. There is no base implementation for this. Possible solution would be creating a custom class that inherits from PagePipe and there you can override ConvertImportedItemForMapping where you can add your custom data as "Content"
protected override Dictionary<string, object> ConvertImportedItemForMapping(object item) var result = base.ConvertImportedItemForMapping(item); InMemoryPageRender render = new InMemoryPageRender(); result.Add("Content", "your custom data here"); Bootstrapper.Initialized += new EventHandler<Telerik.Sitefinity.Data.ExecutedEventArgs>(this.AfterInit);Hello,
I have a Sitefinity 4.0 installation and I'm trying to create a custom search provider.
I'm not looking to index another site, I just want to add some custom content to be indexed, while keeping the clasic index for all pages.
Adding custom search provider was pretty straight forward with SF 3.7 but with SF 4.0 and Pipes, even with the indications Ivan provided, I was not able to do it.
So far, I was able to change the indexed content for an already existing page using a PagePipe starting from the above post.
What I would like is to index some custom content that is not inside sitefinity pages/news/blogs (it is taken from an external database).
How can I add custom content to be indexed and link to an external page? I also want to keep the index SF has for its own pages.
Hi,
Generally the way that you will index the content from another database is the same. You have to return your content in the result as shown in the code above. I found issues related to creating custom pipes and I logged a bug with ID 385822 for this. The current status of the bug is - not done.
Best wishes,
Ivan Dimitrov
the Telerik team
I appreciate the update.
Is there a location where the bug is posted, is there a way for me to follow the progress?
Thank you.
Hi jkregala,
We do not have a section in PITS for Sitefinity from where you can track the status of the bug. I am adding the ID here for reference in case someone else has the same issue and request the status of the bug.
Kind regards,
Ivan Dimitrov
the Telerik team
Has this issue been resolved? I need to do something similar ....
Also, what do you mean by this, "your custom data here", is this a class, string, etc? Do you have an example?
Hi Joseph ,
The issue with custom pipes has been resolved. We will have a sample how to create a custom pipe with Q1 release. We will include the sample in the SDK, Products sample module.
All the best,
Ivan Dimitrov
the Telerik team
Hi Joseph,
Can you confirm if the samples for creating the custom pipes have been released with Sitefinity 4.1. If so can you please point me to where it is in the documentation as I tried to look for it and could not find it.
Thanks
Hello Imran,
We did not include the sample in the SDK, because we are going to change the pipes API this Q which might cause some issues with the custom pipes and basically you will have to rewrite them on the next upgrade. We decided to provide more elegant and easiest way to crawler custom data.
Regards,
Ivan Dimitrov
the Telerik team
has anyone had any success creating a custom search index based off of a custom pipe thus far?
I've made a custom pipe. Will post it on the marketplace shortly for sale.
Hi,
Is there a sample available in documentation or on the market place?
Thanks
Hello Ancuta Me,
I attached a working sample that shows how to implement a custom pipe for Products sample module that comes with the SDK. You can extend it to fit your need.
Best wishes,
Ivan Dimitrov
the Telerik team
Hi There,
I added the CustomPipe folder to my ProductCatalogSample project.
I expected it should add a new scope to the Search Index page! but there weren't any changes there, and I still cannot see any search result if I select other Scopes!
Did I miss something?
Thanks,
Saeed!
Hello Saeed,
We are working on refactoring of the publishing system. We are adding options for custom pipes to work properly. With the current implementation custom PopeSettings could not be used by the default providers, so you cannot have custom pipe settings. This is something that will be fixed for Q2.
Regards,
Ivan Dimitrov
the Telerik team
Has this become available yet? I am trying to add a scope for items stored in a SQL database and can't figure out how to set this up.
Hello Jeremy,
I will try to explain you how to do what you want.
1. You need to iterate all of the objects that you want to include in the index and for each object - create WrapperObject, that holds it's properties (new WrapperObject(your object here)).
2. Set the language for this wrapper object, if your site is multilingual (wrapperObject.Language = lang)
3. Create the document that will be included in the search index
(new Document(wrapperObject, "Id", new List<Mapping>()) - "Id" is the name of the identity field)
4. Now, when you have the document, you just need to include it in the search index - SearchManager.UpdateCatalogue(catalogueName,documents) , where "documents" represents all documents that you want to be included in the search index.
We don't have example for such scenario at the moment, but as you see, this is not hard to be implemented.
Keep in mind that our default behaviour is to index only "Title" and "Content" fields. So if you need to index another properties, you may need to look at this post:
http://www.sitefinity.com/blogs/teodorgeorgiev/posts/11-11-14/search_index_with_custom_fields_in_sitefinity_4.aspx
We refactored the publishing system and you can use custom pipes, but I think that this is much more easier way to achieve what you need.
Best wishes,
Bonny
the Telerik team
What I don't understand is where do I put this code? On what backend or what page? I've already created a search pipe using the above sample but it is not showing in the search scopes. what do I need to do to make it show? I did comment out the.inbound and outbound pipes. Are they needed?
Hello Jeremy,
You can write this code in new aspx page in your project. For example in Page_Load method. When you open this page, the code will run. After it completes, the normal search index for pages (for example with name "TestPageIndex") will be updated and in the catalogue with name "TestPageIndex" will contains the pages from your external database. Keep in mind that those pages should have Title and Content properties, because by default we index only those properties.
You dont need any custom pipes to achieve this. Just run this code, to push all data from external database in your normal pages search index.
Also keep in mind that when you make changes to your external database, you should run this page again to keep your search index updated.
Regards,
Bonny
the Telerik team
Does the Identity Field need to be a GUID or something? When I add it with the ID field as a Long, it gives an error about a GUID not being in the correct format.
I added a zRestart.aspx page and added the following Procedure under if (!IsPostback), but when go to the Admin side and look at the Search Indexes, the Courses Scope does not show.
private void RefreshCourseSearch() try List<Document> locDocs = new List<Document>(); ClsSearch_Data locSearch = new ClsSearch_Data(); ClsSearch_Data.ClsInputParamsSearchCourses locSearchInput = new ClsSearch_Data.ClsInputParamsSearchCourses(); ClsDataSearchReturnSearch locSearchResult = new ClsDataSearchReturnSearch(); locSearch.zValInput = locSearchInput; if (locSearch.zProSearch(ref locSearchResult)) foreach (DataRow locRow in locSearchResult.zValSearchReturn.Rows) string locTitle = ""; string locContent = ""; //string locIDStr = "-1"; string locSearchTerms = ""; //long locID = -1; if (locSearchResult.zValSearchReturn.Columns.Contains("Title") && (locRow["Title"].ToString() != null) && (locRow["Title"].ToString() != "")) locTitle = locRow["Title"].ToString(); if (locSearchResult.zValSearchReturn.Columns.Contains("Description") && (locRow["Description"].ToString() != null) && (locRow["Description"].ToString() != "")) locContent = locRow["Description"].ToString(); if (locSearchResult.zValSearchReturn.Columns.Contains("SearchString") && (locRow["SearchString"].ToString() != null) && (locRow["SearchString"].ToString() != "")) locSearchTerms = locRow["SearchString"].ToString(); //if (locSearchResult.zValSearchReturn.Columns.Contains("WebPage") && (locRow["WebPage"].ToString() != null) && (locRow["WebPage"].ToString() != "")) // // locIDStr = locRow["WebPage"].ToString(); // if (locIDStr.IndexOf("&page=") > -1) // // locIDStr = locIDStr.Substring(locIDStr.IndexOf("&page=") + 6); // // //long.TryParse(locIDStr, out locID); ClsSearch locData = new ClsSearch(Guid.NewGuid(), locTitle, locContent, locSearchTerms); WrapperObject locWrapper = new WrapperObject(locData); Document locDoc = new Document(locWrapper, "ID", new List<Mapping>()); locDocs.Add(locDoc); SearchManager locManager = new SearchManager(); locManager.UpdateCatalogue("Courses", locDocs); lblRefreshedSearchCourse.Visible = true; catch (Exception ex) SCAppVars.AppLogEntry locErrorHandler = new SCAppVars.AppLogEntry(); locErrorHandler.zVal_LastErrorCodeException = SCAppVars.AppLogEntry.ZENUM_ErrorCodeException.ZPRO_EXCEPTION; //same for every procedure StackFrame locSF = new StackFrame(); MethodBase locMB = locSF.GetMethod(); locErrorHandler.ZPRO_ERROR_HANDLE_ExceptionAlerts(locMB.Name + ":Produced EXCEPTION", locErrorHandler.ZPRO_ERROR_GetLastExceptionMessage(), ex, locMB.Name); Hi Jeremy,
About first question - yes, the id should be guid. However, keep in mind that this id should be unique for every item and every item should have exactly one id. What I mean is that if you push one item 2 times with different id-s it will exist 2 times in the search index. I see that you use Guid.NewGuid(), which is really unique, but executing this code 2 times for one search index will add the items 2 times each.
About the "scopes" that you can't see - let me try to explain what search index is :
1. you should create new search index (I see that you want to idex both internal Sitefinity pages and external pages from your DB), so you can create normal page index (static html content) and reindex it. This way all pages that you have will be added to it. (name the search index "Courses" like in your code)
2. If you check the directory "App_Data\Sitefinity\Search\Courses" you should see some files in it - there the search objects are stored.
3. Now you can run your code - all external items will be added in the catalog, so when you search in the search index named "Cources", that you initially created, you should see both your external pages and sitefinity internal pages.
Kind regards,
Bonny
the Telerik team