My Ignite 2017 notes

Overview of shared with Externals and shared Anonymous in Office 365

The GDPR highlights the need for protection of personal data held by organizations. To be able to do this Microsoft inverted a lot in new features and functions like the Office 365 Security & Compliance Center or the GDPR Assessment.

One of the backend systems helping to fulfill those regulations is the SharePoint Online Search Service. In the SharePoint Online Search schema, we can find two managed properties focusing on sharing and access from outside of your organization.

ViewableByExternalUsers and ViewableByAnonymousUsers

Both had the same setting: Query, Retrieve, Refine and Sort. So we can use them to create some reports based on search queries.

Personal overview

Office 365 let every user search in his SharePoint Online sites, OneDrive for Business files and also in Emails for content. In this scenario Email is of topic. But using this search function at the landing page of Office 365 a user can create a personal overview of content he shared to externals or anonymous.

To do this a user needs to fill in the following query in the search box at the Office 365 landing page:

ViewableByAnonymousUsers=true

In this example, I search for documents located in SharePoint Online sites or in my personal OneDrive for Business which are shared based on an anonymous guest link.

Using the query ViewableByExternalUsers=true shows me the files shared with external users through a sharing link that requires them to log in before they can view the file.

This gives a user an overview of documents he has shared from his OneDrive for Business with externals or anonymous. Because the URL is generic you can use this link for all your users and every user get his person overview: https://www.office.com/search?auth=2&home=1&q=ViewableByAnonymousUsers%3Dtrue

Also you can use this link to create a tile in the Office 365 App Launcher as described in the article: Add custom tiles to the app launcher

The result may look like this:

Team Site overview

Microsoft integrated a new out the box reporting capability in every Team Site. The article: View usage data for your SharePoint Online site is showing all details you need to know. There is also a new tab called “Shared externally”.

The article says: List of files you have access to that have been shared with users outside your organization through a sharing link that requires them to log in before they can view the file. Files shared with anonymous users or files available to users with guest permissions are not included.

To get a list of files shared anonymous in this Team Site we can again use the query: ViewableByAnonymousUsers=true followed by a path filter like for example: path:https:\\yourTeamSiteName.sharepoint.com.

Using Search Center to get an overview

As an administrator, you can also use the search center to get an overview of anonymous shared content or about data and also SharePoint Online Sites them self, shared to externals. The queries are basically the same and you can extend them with additional keyword queries properties.

For example, search all Office 366 Groups external users can access:

ViewableByExternalUsers=true contentclass:sts_site WebTemplate:GROUP

(Because of security trimming in SharePoint Search the user who runs the query needs access to all Team Sites to gets an complete report.)

Of cause there are also options archiving this using PowerShell for Office 365 Groups or using Reports in the Office 365 Security & Compliance Center. Using the SharePoint Online search gives you the power and flexibility to integrate all managed properties as metadata in you report like for example ViewsLifeTime, LastModifiedTime, CreatedBy or ModifiedBy. In addition you can easily scope your report to only show documents using the IsDocument=true query parameter or to focus to special Site Templates like WebTemplate:GROUP to only show Office 365 Groups Team Sites etc.

Using PowerShell to get the report

Using PowerShell to get results from SharePoint Online Search also offers the option to save the report as an *.csv file. To call SharePoint Online Search API using PowerShell and save the result to an *.csv file you can follow the steps explained by Prasham Sabadra in his article Office 365/Sharepoint Online - PowerShell Script To Call Search API And Get The Result.

This example is based on his description. The report is showing all external shared content and sites in an Office 365 Tenant and is saving the result to C:\Temp\ViewableByExternalUsers.csv

# add references to SharePoint client assemblies and authenticate to Office 365 site - required for CSOM   

Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"   

Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"   

Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Search.dll"

#Specify tenant admin and URL 

$User = "Admin@yourTenant.onmicrosoft.com"   

#Configure Site URL and User 

$SiteURL = "https://yourTenant.sharepoint.com"  

#Password 

$Password ="yourPassword"   

$securePassword = ConvertTo-SecureString -String $Password -AsPlainText –Force  

$Creds = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($User,$securePassword)

#client context object and setting the credentials  

$Context = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL) 

$Context.Credentials = $Creds

#Calling Search API - Create the instance of KeywordQuery and set the properties 

$keywordQuery = New-Object Microsoft.SharePoint.Client.Search.Query.KeywordQuery($Context)  

#Sample Query - To get the last year result 

$queryText="ViewableByExternalUsers=true" 

$keywordQuery.QueryText = $queryText 

$keywordQuery.TrimDuplicates=$false 

$keywordQuery.SelectProperties.Add("LastModifiedTime") 

$keywordQuery.SelectProperties.Add("ViewsLifeTime") 

$keywordQuery.SelectProperties.Add("ModifiedBy") 

$keywordQuery.SelectProperties.Add("ViewsLifeTimeUniqueUsers") 

$keywordQuery.SelectProperties.Add("Created") 

$keywordQuery.SelectProperties.Add("CreatedBy") 

$keywordQuery.SortList.Add("ViewsLifeTime","Asc")

#Search API - Create the instance of SearchExecutor and get the result 

$searchExecutor = New-Object Microsoft.SharePoint.Client.Search.Query.SearchExecutor($Context) 

$results = $searchExecutor.ExecuteQuery($keywordQuery) 

$Context.ExecuteQuery() 

#Result Count 

Write-Host $results.Value[0].ResultRows.Count

#CSV file location, to store the result 

$exportlocation = "C:\Temp\ViewableByExternalUsers.csv" 

foreach($result in $results.Value[0].ResultRows) 

{ 

$outputline='"'+$result["Title"]+'"'+","+'"'+$result["Path"]+'"'+","+$result["ViewsLifeTime"]+","+$result["ViewsLifeTimeUniqueUsers"]+","+$result["CreatedBy"]+","+$result["Created"]+","+$result["ModifiedBy"]+","+$result["LastModifiedTime"] 

Add-Content $exportlocation $outputline  

}

Experts Inside South Africa Event

Experts Inside South Africa invites you to an an exclusive event to showcase and discuss some of the latest offerings from Microsoft in the Office 365 and Hybrid ecosystems. We’ll be looking at some of the newest and most exciting Office 365 product, Teams, as well as practical guidance to better managing and levering experiences in Delve, Search, the Office Graph and more.

These sessions will be presented by two of Expert Inside’s principle consultants and MVPs, Hilton Giesenow (South Africa) and Nicki Borell (Germany). We look forward to seeing you there! 

Enterprise Search

Many excellent blog posts, interviews and videos about SharePoint and Office 365 search, Delve and Office Graph certainly

exist. This session differs by showing real-world customer scenarios and solutions including the user stories, end-user and

decision maker perspectives and how Delve formed the primary component of an Office 35-based global Intranet. Topics

include data security and privacy aspects and concerns. 

Microsoft Teams

Nearly 2 decades of enterprise social and collaboration technologies from Microsoft have resulted in what looks to be the

company’s most powerful, intuitive, integrated and mobile-capable toolset. Only just in it’s first release (Nov 216) it already

rivals market leaders like Slack, even surpassing it in some areas. Is this the Slack killer some claim? Will it replace Yammer

altogether? In this session we’ll see how Teams works and address and discuss these and other topics.

PnP Partner Pack – Microsoft’s Free Provisioning, Branding and Governance Toolset

The Office 365 Patterns and Practices team comprises experts from around the globe, both within and external to Microsoft,

who provide guidance, tools, libraries and scripts to better provision and maintain Office 365 and on premises SharePoint

sites, lists, libraries and more. One of their recent projects, the PnP Partner Pack, packages many of these existing solutions

into a free turnkey solution to manage provisioning, managing, maintaining and governing solutions, branding, responsive 

design and other elements across SharePoint, Groups, Yammer and other Office 365 workloads. This session examines 

aspects these and other aspects like like self service site creation, site templates and governance automation.

SharePoint 2016 BETA Search – a first look under the hood

PowerShell for SharePoint Search

Installing and configuration using PowerShell works similar to SharePoint 2013. You can use the same cmdlets and scripts as in SharePoint 2013.

Even if you not created a Cloud SSA as part of “Cloud Hybrid Search Preview for SharePoint Server 2013 and 2016” cloud and hybrid topics can be found everywhere.

For example I checked some new parameter within Get-SPEnterpriseSearchServiceApplication:

$sa = Get-SPEnterpriseSearchServiceApplication

$sa.CloudIndex

$sa = Get-SPEnterpriseSearchServiceApplication

$sa.QueryGraph

There is also another interesting information when we have a detailed look at the search topology. As you can see for every component we have a status information “ExperimentalComponent”. This can give us an information whether “Cloud Hybrid Search Preview for SharePoint Server 2013 and 2016” is configured or not.

SharePoint 2016 BETA Ceres – a first look

The Ceres shell is:

• -          a set of PowerShell cmdlets.
• -          It allow to control the internal workings of SharePoint Server search.
• -          It is taken over from the FAST search engine
• -          We can use it to tweak the search engine inner working
• -          modifying the settings can completely crash SharePoint Search
• -          it is part of SharePoint since SharePoint 2013

Manipulations are not for production use, nor supported by Microsoft – but, it can get us a better understanding of what is going on under the hood. If you are interested in further information about the Ceres have a look here: SharePoint 2013 Search internals: The Ceres shell.

Bevor we can use the cmdlets we need to set up PowerShell and connect to the Ceres:

• cd "C:\Program Files\Microsoft Office Servers\16.0\Search\Scripts\"
• .\ceresshell.ps1
• Connect-System –Uri (Get-SPEnterpriseSearchServiceApplication).SystemManagerLocations[0] -ServiceIdentity (Get-SPEnterpriseSearchService).ProcessIdentity
• Connect-Engine
• Connect-AnalysisEngine -NodeName AdminComponent1

To get an overview of all available cmdlets use this:

Get-Command -Module analysisenginepssnapin

Comparing this overview to SharePoint 2013 we see new commands:

• -           Get-AnalysisEngineIsPaused
• -           Get-AnalysisEngineWorker
• -           Register-Analysis
• -           Unregister-Analysis

Actually no further information or documentation is available for this new cmdlets.

Using the get-analysis cmdlet we get an overview for all Analytics component SearchAnalytics, UsageAnalytics and SearchReports. If you are interested in further information about the Analytics components in SharePoint 2013 have a look here: PowerShell for SharePoint 2013 Analytics Feature

In SharePoint 2016 BETA we have a new Analytics component called “LinksStoreInput”

This new component also results in a new subfolder within the Office Server Analytics directory:

Actually no further information or documentation is available about this new component. Based on the name we only can suppose what it is doing.

Get-Flow

Get-flow is about the definition of “what” is happening in the Search internals. Manipulations are not for production use, nor supported by Microsoft. The Content Enrichment Web Service is the primary documented and supported mechanism for implementing "custom" processing logic in SharePoint Search – but, it can get us a better understanding of what is going on under the hood and what is new in SharePoint 2016 Search.

Using the get-flow cmdlet results in an overview showing all flows.

In SharePoint 2016 BETA the following FLOWS are new:

-           Microsoft.ContentPushFlow

-           Microsoft.ContentPushHybridFlow

-           Microsoft.ConversationalLatencyFlow

-           Microsoft.PropertyRetrievalSearchFlow

-           Microsoft.SearchAnalyticsBuzzFeederFlow

-           Microsoft.SearchAnalyticsBuzzInputFlow

-           Microsoft.SearchAnalyticsBuzzImportFlow

-           Microsoft.SearchAnalyticsBuzzOutputFlow

-           Microsoft.SearchAnalyticsExternalContentMapFlow

-           Microsoft.SearchAnalyticsExternalContentPushFlow

-           Microsoft.SearchAnalyticsFollowsImportFlow

-           Microsoft.SearchAnalyticsFollowsInputFlow

-           Microsoft.SearchAnalyticsGraphIndexFeederFlow

-           Microsoft.SearchAnalyticsInFeedInputFlow

-           Microsoft.SearchAnalyticsTagsImportFlow

-           Microsoft.SearchAnalyticsTagsInputFlow

-           Microsoft.SignalStoreInputFlow

-           Microsoft.UsageAnalyticsImportFlow

-           Microsoft.DocumentPreviewMetadataFeederFlow

-           Microsoft.PeopleSuggestionDictionaryDeploymentFlow

-           Microsoft.QueryCompletionPeopleOutputFlow

-           Microsoft.QueryCompletionPersonalizedQueryLogsOutputFlow

-           Microsoft.QueryCompletionPreFetchOutputFlow

-           Microsoft.QueryCompletionQueryLogsInputFlow

-           Microsoft.QueryCompletionQueryLogsOutputFlow

-           Microsoft.CrawlerPropertyReportingSubFlow

-           Microsoft.CrawlerSecurityInsertHybridSubFlow

-           Microsoft.ExplorerFlow

-           Microsoft.GraphSearchProviderFlow

-           Microsoft.QuerySuggestionsQueryLogsFlow

-           Microsoft.DeleteTenantFromIndexSystemFlow

-           Microsoft.IndexingSubFlow

-           Microsoft.DataLossPreventionSubFlow

This can give us an overview of what has changed and what is new in SharePoint 2016 Search. So the main topics based on this get-flow list are:

• -          Content Push
• -          External Content Push
• -          Graph Index Feeder
• -          Graph Search Provider
• -          Signal Store Input Flow
• -          Crawler Security Insert Hybrid Flow
• -          Data Loss Prevention

If you interested in a deeper look in what’s going on in a flow you can use this cmdlet:

get-flow %flow name%

Roundup:

Cloud-awareness can be found everywhere in SharePoint 2016 Search, even if no Cloud SSA is configured.

New features in SharePoint 2016 Search will be:

• -          Content Push
• -          External Content Push
• -          Office Graph interactions
• -          Hybrid Crawler
• -          Data Loss Prevention support

Chapter 2 - SharePoint 2013 Analytics Feature

Using the SharePoint 2013 analytics feature

Found chapter two of my tutorial about using the SharePoint 2013 analytics feature on itunity:

Using the SharePoint 2013 analytics feature

Visit us under www.itunity.com                          Follow us on Twitter @itunity01 

Chapter 1 - SharePoint 2013 Analytics Feature

First chapter of my tutorial about SharePoint 2013 Analytics Feature is only on itUnity:

The SharePoint 2013 Analytics feature

Visit us under www.itunity.com                          Follow us on Twitter @itunity01 

Future of DocPush feature in SharePoint 2013

One of the most missing features in SharePoint Search is the option to push content direct to the index. In FAST ESP we have that functionality. Within FAST for SharePoint 2010 there was a tool called docpush.exe. Using this tool you can push content to the index. But this tool was not designed for productive use. There were several problems around this. For example if you push a document direct to the index using this tool and the crawler catches it also you have duplicates. During SPC14 the information came up that a push functionality for SharePoint 2013 search was already on the roadmap for SP1. Because of some unsolved problems it did not come with SP1. But this means it’s no longer a question if this function is coming – it’s only the question when it comes.

The function is no longer realized in a separate tool like the docpush.exe in FAST for SharePoint 2010. The function would be integrated as part of the SharePoint 2013 Search REST API. So there is a good changes that it also comes for SharePoint online. Locking forward what that all could mean for missing option to create own content sources for SharePoint Online. Stay patient. News are coming up soon.

Problems fixed by CU´s in SharePoint 2013 Search since release

March 2013 update
Server: KB 2767999
Foundation: KB 2768000

• When you perform a search on a SharePoint Server 2013-based server, you experience performance problems on the server.
• The output cache feature does not work on a page on a SharePoint Server 2013 site. This issue occurs when the page contains a Content By Search (CBS) web part.
• Assume that you use the Cross-Site Publishing feature to create a site on a SharePoint Server 2013-based server. When you perform a search on the site, the search results is greater than the expected number.
• Assume that you access a SharePoint site that is not in your trusted sites list. When you try to search an item on the site, you receive the following error message:
  • “The display template had an error. You can correct it by fixing the template or by changing the display template used in either the Web Part properties or Result Types.”
• When you try to perform a search crawl on a list that contains many columns in SharePoint Server 2013, you receive the following error message:
  • “The filename or extension is too long”
• Assume that you filter data in an Access form on a non-English version of a SharePoint Server 2013 server. In this situation, Full-Text Search in SQL Server does not work correctly. More specifically, the filtering operations are slow or return incomplete results.
• When you perform a search in a SharePoint Server 2013 search center in backward compatibility mode, the out of box (OOB) site refiner returns incorrect results.

April 2013 update
Server: KB 2726992
Foundation: KB 2751999

• Assume that you install multiple Search Service applications in SharePoint Server 2013. When you use a Search Service application to crawl a site, managed properties are not displayed in the Search Service application. Instead, the managed properties are unexpectedly displayed in another Search Service application. 

June 2013 update
Server: KB 2817414
Foundation: KB 2817346

• When you pass query parameters to query rules from Search Web Part on a SharePoint Server 2013 site, the search results are not updated. 

August 2013 update

Server: KB 2817616
Foundation: KB 2817517

• When you remove a SharePoint Server 2013 server from a SharePoint Server 2013 farm, the search admin component on another SharePoint Server 2013 server crashes continuously.

October 2013 update

Server: KB 2825647
Foundation: KB 2825674

• This update improves the stability of search components in large server farms.
• The language selection drop-down list on the search page does not extended fully.

December 2013 update
Server: KB 2850024
Foundation: KB 2849961

• When you add a friendly URL or a URL to a non-published item under Search Center Navigation on a SharePoint Server 2013 site, you receive the following error message:
  • Sorry, something went wrong
  • An unexpected error has occurred. 

Service Pack 1
Server: KB 2817429
Foundation: KB 2817439

• Search schema compression is now enabled by default to allow larger search schemas.
• Highlighting for FQL queries is now enabled for FQL as well as KQL.

Query SharePoint Search Analytics using PowerShell

Analytics have been completely revised in SharePoint 2013. Some of the core tech in the previous FAST release has been fused together with SharePoint enterprise search to create a new and powerful analytics engine.

The Analytics Engine is based on two main parts. We have Search Analytics and we have Usage Analytics features. Both parts have own reports which can be found under Site setting and as part of the Search Service Application in Central Admin.

Usage Analytics out-of-the-box report:

Search Analytics out-of-the-box report:

By using PowerShell we can query both Analytics parts and get custom reports.

Usage Analytics:

First of all we have to set the Search Service Application:

 $searchApp = Get-SPEnterpriseSearchServiceApplication

Next step is setting up a site scope for the report:

$Site = Get-SPSite http://intranet.contoso.com/sites/contoso/

Now we can execute the request:

$result = $searchApp.GetRollupAnalyticsItemData(1,[System.Guid]::Empty,$Site.ID,[System.Guid]::Empty)

If the report should not have a site focus we had to replace the “$Site.ID” part with “[System.Guid]::Empty”.

To focus the report data to a specific day or month we can use the GetHitCountForDay and the GetHitCountForMonth method.

First we had set up the data to which we want to focus:

$FilterDate = Get-Date "2014-01-20"

Then we can use the $FilterDate variable together with both methods:

$result.GetHitCountForDay($FilterDate)

$result.GetHitCountForMonth($FilterDate)

Search Reports:

Search Analytics cmdlets follows a different syntax. To get data from that engine we also had to setup the 

Search Service Application:

$searchApp = Get-SPEnterpriseSearchServiceApplication

Also in this case we can focus to a specific site using a variable:

$site = Get-SPSite http://intranet.contoso.com/sites/contoso/

The difference now is that we had to set up a date time value for the report. Leaving it blank is no option here:

$searchApp.GetSearchReport(1,[System.Guid]::Empty,[System.Guid]::Empty,[System.DateTime]::Now,$false,100)

That command gets the report data for the actual month: [System.DateTime]::Now . 

The next parameter which is set to “false” is the switch for getting data for the month = false or the day = true.

If we want the data for another month than the we have to set up a date variable and use it instead of [System.DateTime]::Now :

$FilterDate = Get-Date "2014-01-20"

$searchApp.GetSearchReport(1,[System.Guid]::Empty,$site.ID,$FilterDate,$false,100)

Getting farm wide data we also had to replace the $site.ID variable with “[System.Guid]::Empty”:

$searchApp.GetSearchReport(1,[System.Guid]::Empty,[System.Guid]::Empty,$FilterDate,$false,100)

The first parameter with the value “1” is setting the Event Type. Event Type 1 = views. The next parameter is the TenantID, followed by SiteID, date value, month / day switch and the max result value (100 means show me the top 100 search terms).