Sitecore XP certificate update

Changing the xDB Certificate on Sitecore 10 XP (Azure PaaS)

Rotating certificates in a distributed enterprise CMS environment like Sitecore XP on Azure PaaS takes quite some steps. Maybe this can be automized, maybe not...  I just know we never did as this only happens very often. But when it does, whether you are dealing with an upcoming certificate expiration  date or a mandatory security rotation, missing even a single configuration file or setting will cause xDB communication to fail.

In a Sitecore xDB setup, services talk to each other securely over https using certificate thumbprints. If you miss a role, update the wrong configuration file, or forget to update  the website_load_certificates environment variable, the App Service won't be able to access the certificate. Communication will fail and errors will occur.

Certificate errors

Roles like Content Management (CM) or Content Delivery (CD) will fail to communicate securely with xDB Collection, Reference Data, or Marketing Automation services. Your visitors might not notice as the website will probably be still running, but visitors' interaction data will fail to be recorded or processed, leading to faulty analytics dashboards and broken visitor profiles.

However, if you check your logs (Application Insights and/or Sitecore log files) you will quickly notice a flood of messages like: 

• "X509Certificate" exceptions
• "The certificate with thumbprint [...] could not be found"
• "A call was made on a closed Stream"...
• ...

I can assure you you don't want this. But almost every "Sitecore developer" has bumped into a certificate error at least once. The problem is that the error not always tells you exactly what and where is wrong - you just know it's a certificate issue.

Checklist

To make life easier for my team and anyone else tackling this task, I’ve put together this checklist for updating the xDB communication certificate on a Sitecore 10 XP Azure PaaS setup. Note that this is a checklist for one of our setups. Other setups might (probably will) need  some adjusting - not in the locations as such but mainly on the environments as you can have more servers, slots and so on. The main idea however will stay the same: make sure not to forget any location on any environment. 

Prerequisites / Global Setup

First of all, you'll need to install the new certificate into your Azure App Service Certificate store and/or Key Vault.

At the end (once everything is switched, restarted and verified) remove the old certificate. This might seem unnecessary but it isn't.  Just make sure everything is tested before you do so.

Role-by-Role Update Checklist

Connectionstrings

When we talk about updating connectionstring note that this can be on several locations. Well, each app will have them one one location but it can be different for each solution and even within a solution it can be different for the various apps. The options are:

• in Azure in the KeyVault - referenced in the App environment variables : in this case you need to update the value in the vault and make sure the app is restarted
• in Azure in the App service environment variables connection strings section : use the bulk edit - in this case the edits will trigger a restart
• a connectionstrings.config file in the app_config : this is the default in the Sitecore setup but normally on apps where you desploy your own code this should not be used

1. Content Management (CM)

• Update the certificate thumbprint in your connection strings. This might trigger an App Service restart - depending on the location.
• Update the `WEBSITE_LOAD_CERTIFICATES` app environment variable with the new thumbprint.
  Without this, the App Service won't mount the certificate into the local machine store, resulting in "certificate not found" errors so it's rather crucial.
• If the certificate is also used for SSL, bind the certificate to the CM App Service
• Restart.

2. Content Delivery (CD)

• Update the certificate thumbprint in the connection strings.
• Update the thumbprint on `WEBSITE_LOAD_CERTIFICATES`. 
• Assign the SSL certificate for each custom domain that uses this certificate bound to the CD instances (restarts).
• In case you have one or more deployment slots: don't forget to repeat all the steps above for your each slot. Not all might be needed, but check to be sure.
• Restart (all needed slots)

3. Experience Analytics / Collection (XC-collect)

• Update validateCertificateThumbprint in AppSettings.config in the App_Config folder
• Restart

4. Experience Analytics / Reference Data (XC-RefData)

• Update validateCertificateThumbprint in AppSettings.config.
• Restart

5. Marketing Automation Operations (MA-Ops)

• Update validateCertificateThumbprint in AppSettings.config.
• Update the certificates in ConnectionStrings.config.
• Update the thumbprint on WEBSITE_LOAD_CERTIFICATES in the environment variables.
• Update the certificates in the connectionstrings.config of the automation job in App_Data\jobs\continuous\AutomationEngine\App_Config
• Restart

6. Marketing Automation Reporting (MA-Rep)

• Update validateCertificateThumbprint in AppSettings.config.
• Update the thumbprint on WEBSITE_LOAD_CERTIFICATES.
• Restart

7. Processing (PRC)

• Update the thumbprint on WEBSITE_LOAD_CERTIFICATES.
• Update the thumbprint in the connection strings.
• Restart

8. Experience Analytics Search (XC-Search)

• Update validateCertificateThumbprint in AppSettings.config.
• Restart

9. Sitecore Identity Server (IS)

• Update the thumbprint in wwwroot\Config\production\Sitecore.IdentityServer.Host.xml.
• Update the thumbprint on WEBSITE_LOAD_CERTIFICATES. 
• Restart

10. Cortex Reporting (Cortex-Rep)

• Update the thumbprint on  WEBSITE_LOAD_CERTIFICATES.
• Update validateCertificateThumbprint in AppSettings.config
• Update the certificates in the connectionstrings.config for the processing job in App_Data\jobs\continuous\ProcessingEngine\App_Config.
• Restart

Infrastructure / Code Base Updates (ARM / Bicep / Terraform)

Whatever you are using for deployment, your templates or variable files might include the thumbprint. To make sure your next automated deployment does not overwrite your fixes, update your infrastructure-as-code files.

• Update the thumbprints in your setup templates (`environmentsettings.json`).
• Replace the physical certificate file/reference in your repository if stored or referenced directly.

Post-Update Verification Checklist

Once all roles have been updated and restarted, check the logs (Application Insights probably) and make sure you do not have any certificate exceptions.

You can also verify the Experience Analytics dashboard in Sitecore to ensure graphs are populating correctly.

Keep an eye on the logs for the next 24h just to be sure...

And then convince your customer to move to SitecoreAI so you do not have to do this again. I know tis is easier said than done - the fact that I am writing blogs on Sitecore XP is the best proof :)

Sitecore SXA resolvedlinks

Sitecore SXA ComputedIndexField: resolvedlinks

A while ago - actually 8 years apparently, time flies... - I wrote an article about Sitecore SXA and the search related features. It included the use of "resolvedlinks".  This is a piece of code that is used by SXA for some ComputedFields in the index, but it can be used for our own use as well. 

A Sitecore "computed index field" lets you inject custom logic to generate and/or transform index data dynamically during the indexing process instead of mapping raw field text directly. This allows you to add a field that is not a direct copy of the content of a field. For the rest it is a normal index field, with all the options for such a field on tokenization and such. 

The idea behind the resolvedLinks implementation is simple. If you have an item that contains a reference field like a Multilist of Treelist , the index will store the guids used in that reference field as that is the raw content. In some cases you do want not the guid but an actual value representing that reference. 

In that old blog I had an example as we used the resolvedLinks to add computed index fields to be used as facets. You do not want to show guids as facets - you want an actual string that defines the referenced item.

I already used it on other places as well when I wanted the data to be available in the index for retrieval to avoid having to fetch the referenced items on each request. Several use cases are possible, but recently we bumped into an issue where it didn't fully match our requirements anymore. 

The issue was quotes. Or other weird characters probably. But definetely quotes. 

Taxonomy information architecture

Actually it comes to down to how we setup things in the first place. We never assumed the editors would be using those characters in page titles and that was our mistake. So the "resolvedLinks" is linked to the title of those pages (as requested by the customer and the analysts as this was the most consistent solution). And that is a bad idea...  so we learned now that we should provide an additional field to enter the name or title to use in the taxonomy. 

But of course, editors are human (for now) and as such lazy.  I know there are options to make sure this field is filled, using a copy of the title if needed and such.  But we also have other options. 

The solution

First of all, we did create the extra field on the items to add an optional taxonomy title. Optional, because if the page title was sufficient that is also ok for us. And this also means all the existing data is also still good - backwards compatibility is important on existing sites with quite a bit of content.

We decided to create our own version of the resolvedLinks, one that supports a fallback field. This way we can attach it to the new taxonomy title field but keep the title field as backup in case the new field is empty.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Xml;
using Sitecore.ContentSearch;
using Sitecore.ContentSearch.ComputedFields;
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Globalization;

namespace Foundation.Taxonomy.Index
{
    public class ResolvedLinksWithFallback : AbstractComputedIndexField
    {
        public ResolvedLinksWithFallback()
        {
        }

        public ResolvedLinksWithFallback(XmlNode node) : base(node)
        {
            ContentField = node.Attributes?["contentField"]?.Value;
            FallbackContentField = node.Attributes?["fallbackContentField"]?.Value;
            ReferenceField = node.Attributes?["referenceField"]?.Value;
        }

        public string ContentField { get; set; }

        public string FallbackContentField { get; set; }

        public string ReferenceField { get; set; }

        public override object ComputeFieldValue(IIndexable indexable)
        {
            if (string.IsNullOrEmpty(ReferenceField))
            {
                return null;
            }

            var referencedValues = GetReferencedValuesForField(indexable, ReferenceField);
            return referencedValues != null && referencedValues.Any() ? (object)referencedValues : null;
        }

        protected virtual IEnumerable<string> GetReferencedValuesForField(IIndexable indexable, string fieldName)
        {
            var sourceField = GetSourceField(indexable, fieldName);
            if (sourceField == null)
            {
                return null;
            }

            var ids = GetIds((string)sourceField.Value);
            return !ids.Any() ? null : GetReferencedValues(ids, indexable);
        }

        protected virtual IEnumerable<string> GetReferencedValues(IEnumerable<ID> referencedIds, IIndexable indexable)
        {
            var referencedValues = new List<string>();
            if (!(indexable is SitecoreIndexableItem indexable1))
            {
                return referencedValues;
            }

            var language = Getlanguage(indexable1);
            foreach (ID referencedId in referencedIds)
            {
                var referecedItem = GetReferencedItem(referencedId, indexable1.Item.Database, language);
                if (referecedItem == null)
                {
                    continue;
                }

                var contentValue = !string.IsNullOrEmpty(ContentField) ? GetFieldValue(referecedItem, ContentField) : null;

                if (string.IsNullOrEmpty(contentValue) && !string.IsNullOrEmpty(FallbackContentField))
                {
                    contentValue = GetFieldValue(referecedItem, FallbackContentField);
                }

                if (!string.IsNullOrEmpty(contentValue))
                {
                    referencedValues.Add(contentValue);
                }
            }

            return referencedValues;
        }

        protected virtual string GetFieldValue(Item item, string fieldName) => item.Fields[fieldName]?.Value;

        protected virtual IEnumerable<ID> GetIds(string fieldValue) => fieldValue
            .Split(new char[1] { '|' }, StringSplitOptions.RemoveEmptyEntries)
            .Where(ID.IsID)
            .Select(ID.Parse)
            .ToList();

        protected virtual Language Getlanguage(SitecoreIndexableItem indexable)
        {
            return indexable.Item?.Language;
        }

        protected virtual ID GetId(string id) => ID.IsID(id) ? ID.Parse(id) : null;

        protected virtual IIndexableDataField GetSourceField(IIndexable indexable, string fieldName)
        {
            return indexable.Fields.FirstOrDefault(field => field != null && field.Name.Equals(fieldName, StringComparison.OrdinalIgnoreCase));
        }

        protected virtual Item GetReferencedItem(ID id, Database database, Language language)
        {
            return database.GetItem(id, language);
        }
    }
  }

So with this bit of code we can now also use the resolved links feature with a fallback field, making it backwards compatible for us. Editors can keep on being lazy and fill in just the title - but they can overrule it if wanted and/or needed for the taxonomy to work.

Using the new code is the same as for the out-of-the-box resolvedlinks:

<sitecore>
  <contentSearch>
    <indexConfigurations search:require="solr">
      <defaultSolrIndexConfiguration>
        <documentOptions>
          <fields hint="raw:AddComputedIndexField">
            <field fieldName="F1" type="Sitecore.XA.Foundation.Search.ComputedFields.ResolvedLinks, Sitecore.XA.Foundation.Search" returnType="stringCollection" referenceField="Field1" contentField="title"/>
            <field fieldName="F2" type="Foundation.Taxonomy.Index.ResolvedLinksWithFallback, Foundation.Taxonomy" returnType="stringCollection" referenceField="Field2" contentField="taxonomytitle" fallbackContentField="title"/>
          </fields>
        </documentOptions>
      </defaultSolrIndexConfiguration>
    </indexConfigurations>
  </contentSearch>
</sitecore>

Once you move to SitecoreAI this becomes all obsolete, but for all those still working on Sitecore XM (with SXA - with or without JSS) this is hopefully still useful.