Building Search Driven Solution with SharePoint 2013 Part II

Part I is about Search Driven in on-premise environments

Part II will show the options and differences with O365 SharePoint Online

In Part I I wrote about the options and scenarios with Search and Search Driven Apps in SharePoint 2013 on-premise. Now let’s see which option we have using the online version of new SharePoint 2013.

There are two big differences in SharePoint Online:

1.       We did not have the new Webpart Family “Search Driven Content”.

So in fact we only can use the common Search Result Webpart and its options within the Query Builder to create dynamic search solutions

2.       In SharePoint Online we are not able to configure own content sources ore manipulate the settings like Crawler scheduling etc.

Missing "content source" option in SharePoint Online and its impact

Because of the option “content sources” is not available in SharePoint Online we have to look for alternatives to bring in own content in our SharePoint Online environment. Another point is the index freshness which cannot be manipulated setting up Crawler scheduling ore manually decide to configure continuous crawling option.

Index freshness

Also in the official technet documentation is not a clear statement continuous crawling is set up by default for SharePoint Online or not. My own experience say: “Yes, it is”. I get search result in the index within 2min – 4min. This is really fast and for me in the most scenarios ok.

Own content

To bring in own content in you SharePoint Online Search you have to use “Result Sources” To set up those 3th Party Result Sources you can use “Site Setting” to configure them in the context of a Site Collection ore SharePoint Admin Center to do it global.

The dialog for setting up a Result Source shows the option for bring in own content:

·         Local SharePoint

·         Remote SharePoint

·         OpenSearch 1.0/1.1

·         Exchange

 

For all options you have to configure a security context to access the result source. Using Remote SharePoint you can use SSO or path thru authentication. Using Open Search we have the option to use Anonymous, Basic Authentication, Digest Authentication, NTLM, Form Authentication or Cookie Authentication. Using Remote SharePoint to call an on-premise SharePoint Search you have to set up a Search Federation based on an Identity Federation.

In all cases we had to configure a Remote Address and we can configure a Query Transformation (as described in Part I) to filter ore manipulate the query which is send to the remote system.

In my demo tenant I simply use some open search based systems.

 

Source	Source Url
Twitter	http://search.twitter.com/search.atom?q={searchTerms}
Facebook	http://search.live.com/results.aspx?q={searchTerms}+site%3afacebook.com&format=rss
YouTube	https://gdata.youtube.com/feeds/api/videos?q={searchterms}

(These can of cause also be remote endpoints for your LOB systems or other on-premise sources)

Setting up those systems as a “Result Source” get me the option to use them in my SharePoint Online system to build Search Driven experiences.

Search Driven experiences and solutions in SharePoint Online

Easiest way to use the new result source and within them remote content is using them in the result Webpart. In this example you can see an overview site in my SharePoint Online that aggregate News from configured social media sources based on Result Webparts:

For example filling the atwork area for Twitter news I configured the Result Webpart with those setting:

·         Setting up the “Select a query” with the “Twitter” Result Source

·         Fill in a query that focus to the desired results

Using the configured result sources within Query Rules (as described in Part I) we can build focused search experience like this social media search page:

Search Driven Solution can of cause also build based on SharePoint Online content. Here you see an example based on content placed in SharePoint Online lists ad libraries:

And of course you can mix results coming from remote sources and results coming from your SharePoint Online source.

This Webcast shows the described samples in action:

SharePoint 2013 Analytics Feature - IT-Pro, DB Admin & PowerPivot issues

This post covers some facts and details about the new SharePoint 2013 Analytics feature. For common stuff about that feature have a look at this TechNet article: LINK

In fact we have two different types of analytics in SharePoint 2013:

·         Search analytics analyzes content that is being crawled and added to the search index

·         Usage analytics analyzes user actions, or usage events, such as clicks or viewed items, on the SharePoint site

Search Analytics & Usage Analytics processing is based on several components. Continuous or incremental crawls must be scheduled and run. Also there are several timerjobs involved:

·         Analytics Timer Job for Search Service Application Search Service Application

·         Microsoft SharePoint Foundation Usage Data Import

·         Microsoft SharePoint Foundation Usage Data Processing

·         Usage Analytics Timer Job for Search Application Search Service Application

For more details about the jobs and job descriptions have a look here: LINK

This picture shows an overview for the new components:

Search analytics

PowerShell

Some Facts:

·         There are some cmdlets for create / delete/ get / move etc.  SharePoint Analytics infrastructure issues:

o   SPEnterpriseSearchAnalyticsProcessingComponent

o   SPEnterpriseSearchLinksDatabase

·         There are no direct PowerShell cmdlets to get informations from the SharePoint Analytics

·         We can use the .dot net SearchServiceApplication class, from Microsoft.Office.Server.Search.dll. in combination with PowerShell to work with SharePoint Analytics

Here is an example script from Radu Tut showing how we can connect via PowerShell to SharePoint Analytics:

$searchApp = Get-SPEnterpriseSearchServiceApplication

$site = Get-SPSite "{SiteUrl}"

$result = $searchApp.GetRollupAnalyticsItemData(1,[System.Guid]::Empty,$site.ID,[System.Guid]::Empty)$result

#for a specific date or for a specific month

$date = Get-Date "2013-01-18"

$result.GetHitCountForDay($date)

$result.GetHitCountForMonth($date)

So for example using the first part of the script:

$searchApp = Get-SPEnterpriseSearchServiceApplication

$site = Get-SPSite "{SiteUrl}"

$result = $searchApp.GetRollupAnalyticsItemData(1,[System.Guid]::Empty,$site.ID,[System.Guid]::Empty)

$result

Gets the following result in my demo environment:

There are two methods available. You can use:

·         GetSearchReport

·         GetRollupAnalyticsItemData

Find complete post and info’s about using this in Radu Tuts blog here:  How to get Search Analytics Reports programmatically in SharePoint 2013

SQL Server DB´s

For me as an old SQL guy it was also interesting to see how this all will be established in SQL Server Databases. In an enterprise environment a direct connect to SharePoint Databases isn´t used in general, two point why this is interesting anyway:

·         It’s interesting to see how the analytics database is used to plan you databases files, partitions etc.

·         Only reading from the database using given functions and stored procedures can of cause be an option.

Additionally configuration overview for your environment can be found here: MSSSearchAnalysisEngineConfiguration in your Search_Service_Application_DB

The interesting table in Search_Service_Application_DB is [dbo].[SearchReportsData].

In this table the data for the Usage & Search reports is stored. The data from yesterday is imported once per day from log folder under:  %InstallDrive%\Microsoft Office Servers\15.0\Data\Office Server\Analytics\EventStore

 

The table shows the result in my demo environment:

Using PowerPivot for Excel we can connect to that table and analyze the data:

Usage analytics

The usage analytics data is stored in the logging database. This article describes how to access the data using SQL Views: LINK

Using these views together with PowerPivot we can create our own reports. Here is an example for RequestUsager showing all sites request by “Administrator” for CA:

Also there is an official way consuming Usage Data directly from the logfiles and use it to feed PowerPivot hosted in SharePoint itself: LINK

SharePoint 2013 empty Usage and Health data report

Why the “Usage and Health Data Report” can be empty in SharePoint 2013.

In SharePoint 2013 the feature WebAnalytics known from SharePoint 2010 is no longer part of the product. There is new feature called Analytics .

3 facts why “Usage and Health Data Report” can be empty

·       permission for the database Wss_logging as described in this article: http://social.msdn.microsoft.com/Forums/en-US/sharepointadmin/thread/681b6c53-3969-4984-8a78-4d6e6c55f29c

·       No Crawls: Actual data is processed during crawling. So if there is no crawler scheduled for content sources or there is a problem with the crawlers no actual data is processes.

·       Named Users: Only queries and interaction done by “normal” named users are captured. If you access SharePoint using an administrative account you queries etc. would not be showmen in the reports.

My first podcast @themossshow

My first podcast @themossshow the podcast channel from Hilton Giesenow our South African Experts Inside associate. Thx Hilton!: http://www.themossshow.com/2013/03/episode-65-nicki-borell-builds-search-driven-applications/

Building Search Driven Solution with SharePoint 2013 Part I

Part I is about Search Driven in on-premise environments

Part II will show the options and differences with O365 SharePoint Online

Search Driven Solutions are not new in SharePoint 2013. But with SP2013 they reached a new dimension and there are much more out-of-the-box Webparts and options to work with content that is in your search index.

Admin Stuff

Why to use Search Driven? Good question. Let’s ask “why not”? The answer is the index latency. Search Driven Solutions are based on the search index and that means that the data freshness depends on the index freshness. With new feature Continuous Crawling and other solutions like Event Driven Crawling we can realize a really up to date index. Depending on you environment you can realize an index freshness in the scope of 2 minutes or so. Some other points in the context of Search Driven Solutions are:

·         Separate presentation from storage

·         Flexible and dynamic

·         Breaking down site collection boundaries

·         Eliminate large list thresholds

·         Allows flexible & dynamic publishing

 

Special Data means Special Search and special Search Results

Not in every case we have to choose between normal SharePoint Search and Search Result Webpart ore using Search Driven Solutions. There is a very useful and powerful option between. With SharePoint 2013 there are some new features. In the context of “Special Date means Special Search and special Search Results” we will now have a closer look on “Result Source” and “Query Rules”.

Result Source

Working with search based solutions mostly we start with a “Result Source”. Result Sources are places under Site Settings or if you would configure them for the complete farm in the search service application. A Result Source had some basic parameters:

Protocol: defines from where the results are coming

Type: focused between content and people search

Query Transformation: gives us the option to focus which data is shown in this Result Source using Search Syntax. Also we had the option using the Query Builder to define the Query Transformation.

To use a Result Source we had to configure search result Webpart to use this source. This is simply configured in the settings of the result Webpart:

Query Rules

Query Rules are used to manipulate search query. A Query Rule always based on a Result Source. That’s why we have to start with a Result Source. Query Rules are also based in the site setting ore in search service application. Working with Query Rules we had two main parameters.

1.       Query Condition: this parameter defines under which condition the Query Rules take effect.

To do this we had several options. The easiest way is “Query matches Keyword exactly”. But we also can use Termstore using the option “Query matches Dictionary exactly” This brings many powerful options. For example if you extend you Termset which is referred you not need to reconfigure you Query Condition. Also in Multilanguage environment this can be useful.

2.       Actions: The section configures what should happen if a Query matches.

We can configure a promoted result which is similar to Best Bets we know from SharePoint 2010 are we can place a Result Block.

Result Blocks are a new feature that allows us placing a separate block containing the data we configured based on our Result Source in top of the search result Webpart. Every Result Block can be configured using Query Builder. For example if you will use a special Result Block only showing pictures the configuration should be like this one:

·         Query: {subjectTerms} contenttype:image

·         Settings: Item Display Template -> Picture Item

Display Templates are also a new feature in SharePoint 2013. They allow us to use different visualizations based on content type or so. Details see here: LINK

As shown in this screenshot:

Bring all this together we can deliver special search for special data.

To get a result like this we had to configure a Result Source, based on this Query Rules and then use the Result Source in a search result Webpart. Detail Step by Step walkthrough is shown in the Webcast at the end of the post.

Search Driven Publishing Model

The above solutions are all based on a search query which had to be filled in a search box by a user or had to be configured as a “fixed keyword query” in the settings of the search result Webpart. Now let’s see how we can create dynamic pages showing content based on Search Querys using the new Webpart Family “Search Driven Content”.

As you can see there are preconfigured Webparts for different scopes. The context driven Webparts like “Popular Items” and “Recommended Items” are based on search analytics, user context and user activity. Other ones like “Pictures” or “Pages” containing a special visualization based on the contented type. The “Search Driven Content” Webparts can also be used to visualize search results based on a search query which is typed into a search box. All search Webparts can combined which each user. This is used configuring the following example:

Here you can see the “Search Driven Content” Webpart for Pictures. In the settings dialog the Display Template is configured to show “Picture on top, 3 lines on bottom”. Under Property Mappings you can choose which Managed Property are used to fill the lines. In the context of the shown Refiner Webpart “Refinement Target” is configured to the “Search Driven Content” Webpart Pictures. The binding is based on the Title of the Webpart.

 

Solutions based on the Search API

The Search API allows use building Apps ore other solutions based on the content coming from the search index. For more information about the SharePoint 2013 Search API look here: LINK

Here is a example based on my demo environment:

http://win-ful28kv4389/_api/search/query?querytext='contenttype%3Aorbeabike'

Using this query the result looks like this:

Using this XML we build a demo App showing the same data like in the above shown search result Webpart using the Result Blocks:

 

Webcast with hands on system demos: