Overview of shared with Externals and shared Anonymous in Office 365

The GDPR highlights the need for protection of personal data held by organizations. To be able to do this Microsoft inverted a lot in new features and functions like the Office 365 Security & Compliance Center or the GDPR Assessment.

One of the backend systems helping to fulfill those regulations is the SharePoint Online Search Service. In the SharePoint Online Search schema, we can find two managed properties focusing on sharing and access from outside of your organization.

ViewableByExternalUsers and ViewableByAnonymousUsers

Both had the same setting: Query, Retrieve, Refine and Sort. So we can use them to create some reports based on search queries.

Personal overview

Office 365 let every user search in his SharePoint Online sites, OneDrive for Business files and also in Emails for content. In this scenario Email is of topic. But using this search function at the landing page of Office 365 a user can create a personal overview of content he shared to externals or anonymous.

To do this a user needs to fill in the following query in the search box at the Office 365 landing page:

ViewableByAnonymousUsers=true

In this example, I search for documents located in SharePoint Online sites or in my personal OneDrive for Business which are shared based on an anonymous guest link.

Using the query ViewableByExternalUsers=true shows me the files shared with external users through a sharing link that requires them to log in before they can view the file.

This gives a user an overview of documents he has shared from his OneDrive for Business with externals or anonymous. Because the URL is generic you can use this link for all your users and every user get his person overview: https://www.office.com/search?auth=2&home=1&q=ViewableByAnonymousUsers%3Dtrue

Also you can use this link to create a tile in the Office 365 App Launcher as described in the article: Add custom tiles to the app launcher

The result may look like this:

Team Site overview

Microsoft integrated a new out the box reporting capability in every Team Site. The article: View usage data for your SharePoint Online site is showing all details you need to know. There is also a new tab called “Shared externally”.

The article says: List of files you have access to that have been shared with users outside your organization through a sharing link that requires them to log in before they can view the file. Files shared with anonymous users or files available to users with guest permissions are not included.

To get a list of files shared anonymous in this Team Site we can again use the query: ViewableByAnonymousUsers=true followed by a path filter like for example: path:https:\\yourTeamSiteName.sharepoint.com.

Using Search Center to get an overview

As an administrator, you can also use the search center to get an overview of anonymous shared content or about data and also SharePoint Online Sites them self, shared to externals. The queries are basically the same and you can extend them with additional keyword queries properties.

For example, search all Office 366 Groups external users can access:

ViewableByExternalUsers=true contentclass:sts_site WebTemplate:GROUP

(Because of security trimming in SharePoint Search the user who runs the query needs access to all Team Sites to gets an complete report.)

Of cause there are also options archiving this using PowerShell for Office 365 Groups or using Reports in the Office 365 Security & Compliance Center. Using the SharePoint Online search gives you the power and flexibility to integrate all managed properties as metadata in you report like for example ViewsLifeTime, LastModifiedTime, CreatedBy or ModifiedBy. In addition you can easily scope your report to only show documents using the IsDocument=true query parameter or to focus to special Site Templates like WebTemplate:GROUP to only show Office 365 Groups Team Sites etc.

Using PowerShell to get the report

Using PowerShell to get results from SharePoint Online Search also offers the option to save the report as an *.csv file. To call SharePoint Online Search API using PowerShell and save the result to an *.csv file you can follow the steps explained by Prasham Sabadra in his article Office 365/Sharepoint Online - PowerShell Script To Call Search API And Get The Result.

This example is based on his description. The report is showing all external shared content and sites in an Office 365 Tenant and is saving the result to C:\Temp\ViewableByExternalUsers.csv

# add references to SharePoint client assemblies and authenticate to Office 365 site - required for CSOM   

Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"   

Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"   

Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Search.dll"

#Specify tenant admin and URL 

$User = "Admin@yourTenant.onmicrosoft.com"   

#Configure Site URL and User 

$SiteURL = "https://yourTenant.sharepoint.com"  

#Password 

$Password ="yourPassword"   

$securePassword = ConvertTo-SecureString -String $Password -AsPlainText –Force  

$Creds = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($User,$securePassword)

#client context object and setting the credentials  

$Context = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL) 

$Context.Credentials = $Creds

#Calling Search API - Create the instance of KeywordQuery and set the properties 

$keywordQuery = New-Object Microsoft.SharePoint.Client.Search.Query.KeywordQuery($Context)  

#Sample Query - To get the last year result 

$queryText="ViewableByExternalUsers=true" 

$keywordQuery.QueryText = $queryText 

$keywordQuery.TrimDuplicates=$false 

$keywordQuery.SelectProperties.Add("LastModifiedTime") 

$keywordQuery.SelectProperties.Add("ViewsLifeTime") 

$keywordQuery.SelectProperties.Add("ModifiedBy") 

$keywordQuery.SelectProperties.Add("ViewsLifeTimeUniqueUsers") 

$keywordQuery.SelectProperties.Add("Created") 

$keywordQuery.SelectProperties.Add("CreatedBy") 

$keywordQuery.SortList.Add("ViewsLifeTime","Asc")

#Search API - Create the instance of SearchExecutor and get the result 

$searchExecutor = New-Object Microsoft.SharePoint.Client.Search.Query.SearchExecutor($Context) 

$results = $searchExecutor.ExecuteQuery($keywordQuery) 

$Context.ExecuteQuery() 

#Result Count 

Write-Host $results.Value[0].ResultRows.Count

#CSV file location, to store the result 

$exportlocation = "C:\Temp\ViewableByExternalUsers.csv" 

foreach($result in $results.Value[0].ResultRows) 

{ 

$outputline='"'+$result["Title"]+'"'+","+'"'+$result["Path"]+'"'+","+$result["ViewsLifeTime"]+","+$result["ViewsLifeTimeUniqueUsers"]+","+$result["CreatedBy"]+","+$result["Created"]+","+$result["ModifiedBy"]+","+$result["LastModifiedTime"] 

Add-Content $exportlocation $outputline  

}

Create and use own usage event type in SharePoint 2013

The original technet article says the following about usage event types in SharePoint 2013

“Usage events enable you to track how users interact with items on your site. Items can be documents, sites, or catalog items. When a user interacts with an item on your site, SharePoint Server 2013 generates a usage event for this action. For example, if you want to monitor how often a catalog item is viewed from a mobile phone, you can track this activity. This article describes how to create custom usage event types, and how to add code to record custom usage events so that they can be processed by the analytics processing component. You can use the data that is generated by usage events to show recommendations or popular items on your site. This article also explains how to influence how recommendations are shown by changing the level of importance for a specific usage event type.”

So far, so good. Now let’s see how this can be done because of some code snippets in the original article won’t work very well.
In the following example we will create a custom event called “UseFulPage” and add a button to SharePoint to fire that event.

Step 1: create a custom usage event type

#Connect to SSA

$SSP = Get-SPEnterpriseSearchServiceApplicationProxy

#create Event:

#modify the $EventName varibale to your value

$EventGuid = [Guid]::NewGuid()

$EventName = "UseFulPage"

$tenantConfig = $SSP.GetAnalyticsTenantConfiguration([Guid]::Empty)

$newEventType = $tenantConfig.RegisterEventType($EventGuid, $EventName, "")

$tenantConfig.Update($SSP)

#manipulate the weight of you event by setting RecommendationWeight & RelevanceWeight

$customEvent = $tenantConfig.EventTypeDefinitions | where-object { $_.EventName -eq $EventName }

$customEvent.RecommendationWeight = 10

$customEvent.RelevanceWeight = 10

$tenantConfig.Update($SSP)

#check if the event is created
$tenantConfig.EventTypeDefinitions | select eventTypeId,EventName | ft

The result should be something like that:

The new custom event had several properties and also 2 managed properties (UsageEvent1LifeTime & UsageEvent1Recent) which can directly be used in Search or Search Driven WebParts after the next analytics processing run.

Step 2:  add code to record the custom usage event
In the following script we need the GUID of the event. To get the GUID use this script:

$SSP = Get-SPEnterpriseSearchServiceApplicationProxy
$tenantConfig = $SSP.GetAnalyticsTenantConfiguration([Guid]::Empty)
$tenantConfig.EventTypeDefinitions | select AppEventTypeId,eventTypeId,EventName | ft

To add the event to a SharePoint site I use a JavaScript snippet (thx Markus Alt).

1. Replace %EventGUID% with the GUID of your custom event.
2. Replace %URL% with the URL of the site where you place the event. For example: http://intranet.contoso.com
3. Use a ScriptEditor WebPart to host the script in your SharePoint site.

<script language="javascript">

function ToEventStore(url)
{
    alert("Useful Page Event recorded");
    ExecuteOrDelayUntilScriptLoaded(function()
    {
        var spClientContext = SP.ClientContext.get_current();
        var eventGuid = new SP.Guid("%EventGUID%");
        SP.Analytics.AnalyticsUsageEntry.logAnalyticsAppEvent(spClientContext, eventGuid, url);
        spClientContext.executeQueryAsync(null, Function.createDelegate(this, function(sender, e){ alert("Failed to log event for item: " + document.URL + " due to: " + e.get_message()) }));
    }, "SP.js");
}
</script>

<button onclick="ToEventStore('http://%URL%')">Useful Page</button>

Step 3: work with the event
Clicking the button brings up a message box which tells you that your click is recorded. That means that the event is send to analytics engine and is processed during the next run. So you have to wait until the next day to see the result. By the way: Analytics data is process only once a day. You can configure the schedule using the Set-AnalysisConfiguration cmdlet. Other important point is that only data from last day is processes, not from the actual day. Bella Engen descried in that post how to manually kick the Analytics so there is no waiting until next day to see the data: http://blogs.technet.com/b/tothesharepoint/archive/2014/01/21/modify-the-content-search-web-part-display-template-and-use-windows-powershell-to-start-usage-analytics-in-sharepoint-server-2013.aspx

Kicking the event of cause has an impact to the ranking of that site. Another option to bring the result out is to use the managed property “UsageEvent1Recent” in a Search WebPart as a sort option. Doin this you can build an overview based on search showing the most “Useful Pages” in you SharePoint:

Session preview video for SPC14

Uploaded the preview video about my session at this year SPC in Las Vegas:

Using SharePoint PropertyBag in the Context of Search

My first post @ MVP Award Blog :-)

http://blogs.msdn.com/b/mvpawardprogram/archive/2013/11/25/using-sharepoint-propertybag-in-the-context-of-search.aspx

again with a hands on lab video @ my YouTube channel:

 

 

 

Building Search Driven Solution with SharePoint 2013 Part II

Part I is about Search Driven in on-premise environments

Part II will show the options and differences with O365 SharePoint Online

In Part I I wrote about the options and scenarios with Search and Search Driven Apps in SharePoint 2013 on-premise. Now let’s see which option we have using the online version of new SharePoint 2013.

There are two big differences in SharePoint Online:

1.       We did not have the new Webpart Family “Search Driven Content”.

So in fact we only can use the common Search Result Webpart and its options within the Query Builder to create dynamic search solutions

2.       In SharePoint Online we are not able to configure own content sources ore manipulate the settings like Crawler scheduling etc.

Missing "content source" option in SharePoint Online and its impact

Because of the option “content sources” is not available in SharePoint Online we have to look for alternatives to bring in own content in our SharePoint Online environment. Another point is the index freshness which cannot be manipulated setting up Crawler scheduling ore manually decide to configure continuous crawling option.

Index freshness

Also in the official technet documentation is not a clear statement continuous crawling is set up by default for SharePoint Online or not. My own experience say: “Yes, it is”. I get search result in the index within 2min – 4min. This is really fast and for me in the most scenarios ok.

Own content

To bring in own content in you SharePoint Online Search you have to use “Result Sources” To set up those 3th Party Result Sources you can use “Site Setting” to configure them in the context of a Site Collection ore SharePoint Admin Center to do it global.

The dialog for setting up a Result Source shows the option for bring in own content:

·         Local SharePoint

·         Remote SharePoint

·         OpenSearch 1.0/1.1

·         Exchange

 

For all options you have to configure a security context to access the result source. Using Remote SharePoint you can use SSO or path thru authentication. Using Open Search we have the option to use Anonymous, Basic Authentication, Digest Authentication, NTLM, Form Authentication or Cookie Authentication. Using Remote SharePoint to call an on-premise SharePoint Search you have to set up a Search Federation based on an Identity Federation.

In all cases we had to configure a Remote Address and we can configure a Query Transformation (as described in Part I) to filter ore manipulate the query which is send to the remote system.

In my demo tenant I simply use some open search based systems.

 

Source	Source Url
Twitter	http://search.twitter.com/search.atom?q={searchTerms}
Facebook	http://search.live.com/results.aspx?q={searchTerms}+site%3afacebook.com&format=rss
YouTube	https://gdata.youtube.com/feeds/api/videos?q={searchterms}

(These can of cause also be remote endpoints for your LOB systems or other on-premise sources)

Setting up those systems as a “Result Source” get me the option to use them in my SharePoint Online system to build Search Driven experiences.

Search Driven experiences and solutions in SharePoint Online

Easiest way to use the new result source and within them remote content is using them in the result Webpart. In this example you can see an overview site in my SharePoint Online that aggregate News from configured social media sources based on Result Webparts:

For example filling the atwork area for Twitter news I configured the Result Webpart with those setting:

·         Setting up the “Select a query” with the “Twitter” Result Source

·         Fill in a query that focus to the desired results

Using the configured result sources within Query Rules (as described in Part I) we can build focused search experience like this social media search page:

Search Driven Solution can of cause also build based on SharePoint Online content. Here you see an example based on content placed in SharePoint Online lists ad libraries:

And of course you can mix results coming from remote sources and results coming from your SharePoint Online source.

This Webcast shows the described samples in action:

Office 365 - upgrade to SharePoint 2013 Search Problems

Actually the first O365 tenants are updated to SharePoint 2013 backend components in the background. This results in some technical difficulties. One is the Search. The SharePoint 2013 Search Engine based on a new / upgraded Search Backend system. The FAST Engine had a big impact in all the new functions and features. More details can be found here: http://nbsharepointtalk.blogspot.de/2012/07/whats-new-within-sharepoint-2013-search.html .

If your tenant is updated to the new backend system but your frontend is still SharePoint 2010 you have some loosing. The point is that a lot of keyword calls not working anymore. For example: Path:"Posts" AND -Path:"personal" (meaning: show all which contains “Posts” but not contains “personal” in result it shows blogposts but not posts from blogs hosted in MySites). The reason is that the Managed Property “Path” no longer exists in the new Search Engine. It is replaced by ”SPSiteUrl” and so on…. Another point is, that the new Keyword Query Language and also FAST Query Language which is now supported in SharePoint 2013 cannot full functionally be used with the SharePoint 2010 UI, even the search backend is upgraded. This is because the SharePoint 2010 Webpart are not able to handle search term like author:starts-with("Nicki"). So this is a poor interims situation at the moment. But the complete update is availible soon ;-) and then we can use the full functionality of the new Search Engine.

What’s new within SharePoint 2013 Search? – PART II

PART II: What happens with FAST in SharePoint 2013

(Concern that this is BETA stuff. Features and functions can be changed or shift until the final release!)

Fact 1: In SharPoint 2013 the two Search Engines “SharePoint Search” and “FAST Search Server for SharePoint” was combined in one Search Engine.

Fact 2: FAST as a standalone product is still available and supported. Details see here: http://support.microsoft.com/lifecycle/search/default.aspx?sort=PN&alpha=fast

Future of FAST ESP: See this statement from Rob Va from July 16: http://social.technet.microsoft.com/Forums/en-US/fastinternetesp/thread/86e5e64f-1fd0-4ee4-a025-1dea0f1693df

Mostly all the features / functions that we know from FAST Search Central Admin sites  had been integrated in Enterprise Search in SharePoint 2013:

Let’s have a look at the feature and functions level:

Managed Properties:

The technique with crawled properties and managed properties was similar in SharePoint Enterprise Search and FAST Search Server for SharePoint. But with FAST we had additional configuration options:

The functions / fields: Name, Type and Mappings to Crawled Properties were the same in both SharePoint 2010 Search Engines.

The features Sort Property, Query Property, Refiner Property and Full-text Index Mapping moved into the new Managed Property configuration:

There are some more options in the new SharePoint 2013 Managed Property configuration, but this will be part of “Part IV: Admin Stuff”.

Crawled Properties:

Crawled Properties are the same in FAST, Enterprise Search, and SharePoint 2013 Search.

Managed Property Extraction:

This section was completely reworked. In FAST 2010 we have to work with some PowerShell calls like: Get-FASTSearchResource dictionaries\spellcheck\sk_spell_iseck_en.txt and dictionary files based on an XML structure to include new terms. In SharePoint 2013 this feature used the Managed Metadata / Termstore to handle the term / dictionaries for company individual property extraction:

You can see that the function for ignore list and spell checking also moved to the Termstore. In detail we will have a look at this in Part III.

FAST Service Application:

So in fact only one integrated Search Engine left we didn’t have to configure the different Shared Service Applications for FAST Contend and Fast Query as we have to in SharePoint 2010:

Query Language / FAST Query Language:

FAST brings his own Query Language which was just different / extended from the SharePoint Search Query Language. Some of its characteristics are now found into Search Query Language. For example the XRANK Operator:

In SharePoint Server 2010, the XRANK operator was available only with FAST Query language (FQL). The XRANK operator provides dynamic control of ranking.

SharePoint 2013 Search does not longer support SQL syntax. Search in SharePoint 2013 supports FQL syntax and KQL syntax for custom search solutions.

For more details about building Search Querys ins SharePoint 2013 Search have a look here:

http://msdn.microsoft.com/en-us/library/sharepoint/ee558911(v=office.15)

FAST Stuff:

FAST stuff and FAST specific components like the “extended WebCrawler” or commanding tools like indexerinfo.exe are no longer part of SharePoint, at least not part of the SharePoint 2013 Preview version.

Developing custom connectors / crawler is now standardized.

Content Processing:

In FAST for SharePoint 2010 we had the Advanced Content Processing Pipeline architecture:

In SharePoint 2013 this is part of the common Search Architecture.

Next parts in this series:

Part III: A look in the deep what’s behind the new Search functions like “Search Dictionaries”, “Query Builder”, “Query Client Type”

Part IV: Admin Stuff

Part V: Frontend Stuff

Show up all blog entries from different sites at one place

Show up all blog entries from different sites at one place

Last week at the ShareConf I meet the SPUG community member Dennis de Vries. He told me from a solution based on Search Driven Ideas that he builds. The scope was aggregating all blogs from different SharePoint sites bases on a Search Driven Solution. Here is his post:

http://www.ilikesharepoint.de/2012/06/sharepoint-2010-search-show-up-all-blog-entries-from-different-sites-at-one-place/

Search Driven Applications with Office 365 / SharePoint Online - Part I

Part I: Common stuff about Search Driven Applications and first solution proposal with Office 365 / SharePoint Online.

I wrote about building Search Driven Application with SharePoint 2010 some stuff and also some of my sessions during conferences were around this. For example here are some links:

Article in SharePoint Magazine:

http://it-republik.de/dotnet/sharepoint-magazin-ausgaben/Business-Analytics-000498.html

Video from last year Collaboration Days: (http://www.collaborationdays.ch )

http://mediasite.lm-ag.de/mediasite/SilverlightPlayer/Default_2.aspx?peid=3bfcfe8c6d154ed5ab2b23f8a4de795b

You can build Search Driven Applications in several different ways and based on different techniques. One of the easiest is using the Query Object Model from the SharePoint Search to create “Fixed Keyword Queries”. For example this query shows all SharePoint sites which you have access to:

contentclass:STS_Site contentclass:STS_Web contentClass:sts_listitem_850

For more Details have a look at the BlogPost from Laura Rogers: LINK

But the “magic” within Search Driven Solution is aggregating some similar crawled properties together in one managed property. See schema diagram:

Example: you have two lists in your SharePoint farm which both contains information about customers. Because of some historical reasons, maybe migration from SharePoint 2007 or merging from other systems, these are both independent and different fields in different SharePoint List. Merging this two Crawled Property together in one Managed Property called for example “cKunde” allowed you querying both fields with one call (for example: cKunde:AOL - you will get results from both list). Pooling Crawled Properties in Managed Properties as shown in the next picture is one of the technical basics for Search Driven Application:

Pooling properties is no problem in a SharePoint on premise installation. But with Office 365 / SharePoint Online we ran in some problems. The main problem is that we cannot configure the Search Service Application in Office 365 / SharePoint Online. So we are not able to do the merge from Crawled to Managed Properties, and without Managed Properties we are not able to query the content for specific issues. We can of cause use the given default Managed Properties like “Author, Filetype, Name etc.” and build Search Driven Solutions based on them. The constrain here is that we cannot edit them free as we need. The Managed Property “Filetype” for example is filed by the system with the given value. So we have to use some “tricks” to build Search Driven Application with Office 365 / SharePoint Online.

My solution for this problem is using the tagging feature in SharePoint 2010. Within this feature we are able to tag any content with special and individual information that can later be used to aggregate it under this scope or in the mix with other tags / scopes. Using the feature: “Metadata Publishing -> Save metadata on this list as social tags” allowed us to force users to fill in metadata in keyword fields or set default values which will then be automatically published as a social tag.

Based on this there are several ways to build Search Driven Solutions. During this Blogpost series I will come closer to two of them.

Let’s have a look at the first easier and simple one:

There is a Crawled Property called SocialTagId that can now be used to query for a special tag and so it can be used to build a search driven webpart.

Example search query: SocialTagId:"6a…b-754b-480e-86e8-5a…ae"

This Managed Property can also be used with standard search syntax for building AND / OR queries and be mixed with other search terms.

Example: SocialTagId:"fb2d872a-8750-4913-bc23-b0b32252b489" OR SocialTagId:"25dee478-f426-4dc1-9d25-340e9ecce093" – shows any content which is tagged with one of the given tags behind this SocialTagID´s

Some points to attend:

-          The result is just the same you get when you navigate to the Tag Profile page for a special tag.

-          Benefit is that you can use the resultset coming from the search in a separate webpart und customize the style etc. in an easy way

-          We are not able to do a  fuzzy search within this

-          You have to find out the SocialTagId which depends to your tag. This is not really a problem. To do these navigate to the Tag Profile, select “To find content related to '%tag%' in search, please click here.” The result is a search with the Managed Property “SocialTagId” and the given ID for the selected tag:

To Customize those Search Result using SharePoint Designer and XSLT follow the steps described by Laura Rogers: LINK

Webcast with hands on system demos:

Summary: we can build search driven solutions based on the tagging feature in Office 365 / SharePoint Online. Within advanced Document Library and List features like “Metadata Publishing” we can increase the usability and the user experience for adding this kind of information.

To aggregate any content tagged with special tags we can use the Managed Property SocialTagId.
Next post I will show how we can aggregate content based on any different similar tags.