Showing posts with label ComputedIndexField. Show all posts
Showing posts with label ComputedIndexField. Show all posts

Monday, October 12, 2026

Sitecore SXA resolvedlinks

Sitecore SXA ComputedIndexField: resolvedlinks


A while ago - actually 8 years apparently, time flies... - I wrote an article about Sitecore SXA and the search related features. It included the use of "resolvedlinks".  This is a piece of code that is used by SXA for some ComputedFields in the index, but it can be used for our own use as well. 

A Sitecore "computed index field" lets you inject custom logic to generate and/or transform index data dynamically during the indexing process instead of mapping raw field text directly. This allows you to add a field that is not a direct copy of the content of a field. For the rest it is a normal index field, with all the options for such a field on tokenization and such. 

The idea behind the resolvedLinks implementation is simple. If you have an item that contains a reference field like a Multilist of Treelist , the index will store the guids used in that reference field as that is the raw content. In some cases you do want not the guid but an actual value representing that reference. 

In that old blog I had an example as we used the resolvedLinks to add computed index fields to be used as facets. You do not want to show guids as facets - you want an actual string that defines the referenced item.

I already used it on other places as well when I wanted the data to be available in the index for retrieval to avoid having to fetch the referenced items on each request. Several use cases are possible, but recently we bumped into an issue where it didn't fully match our requirements anymore. 

The issue was quotes. Or other weird characters probably. But definetely quotes. 

Taxonomy information architecture

Actually it comes to down to how we setup things in the first place. We never assumed the editors would be using those characters in page titles and that was our mistake. So the "resolvedLinks" is linked to the title of those pages (as requested by the customer and the analysts as this was the most consistent solution). And that is a bad idea...  so we learned now that we should provide an additional field to enter the name or title to use in the taxonomy. 

But of course, editors are human (for now) and as such lazy.  I know there are options to make sure this field is filled, using a copy of the title if needed and such.  But we also have other options. 

The solution

First of all, we did create the extra field on the items to add an optional taxonomy title. Optional, because if the page title was sufficient that is also ok for us. And this also means all the existing data is also still good - backwards compatibility is important on existing sites with quite a bit of content.

We decided to create our own version of the resolvedLinks, one that supports a fallback field. This way we can attach it to the new taxonomy title field but keep the title field as backup in case the new field is empty.


using System;
using System.Collections.Generic;
using System.Linq;
using System.Xml;
using Sitecore.ContentSearch;
using Sitecore.ContentSearch.ComputedFields;
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Globalization;

namespace Foundation.Taxonomy.Index
{
    public class ResolvedLinksWithFallback : AbstractComputedIndexField
    {
        public ResolvedLinksWithFallback()
        {
        }

        public ResolvedLinksWithFallback(XmlNode node) : base(node)
        {
            ContentField = node.Attributes?["contentField"]?.Value;
            FallbackContentField = node.Attributes?["fallbackContentField"]?.Value;
            ReferenceField = node.Attributes?["referenceField"]?.Value;
        }

        public string ContentField { get; set; }

        public string FallbackContentField { get; set; }

        public string ReferenceField { get; set; }

        public override object ComputeFieldValue(IIndexable indexable)
        {
            if (string.IsNullOrEmpty(ReferenceField))
            {
                return null;
            }

            var referencedValues = GetReferencedValuesForField(indexable, ReferenceField);
            return referencedValues != null && referencedValues.Any() ? (object)referencedValues : null;
        }

        protected virtual IEnumerable<string> GetReferencedValuesForField(IIndexable indexable, string fieldName)
        {
            var sourceField = GetSourceField(indexable, fieldName);
            if (sourceField == null)
            {
                return null;
            }

            var ids = GetIds((string)sourceField.Value);
            return !ids.Any() ? null : GetReferencedValues(ids, indexable);
        }

        protected virtual IEnumerable<string> GetReferencedValues(IEnumerable<ID> referencedIds, IIndexable indexable)
        {
            var referencedValues = new List<string>();
            if (!(indexable is SitecoreIndexableItem indexable1))
            {
                return referencedValues;
            }

            var language = Getlanguage(indexable1);
            foreach (ID referencedId in referencedIds)
            {
                var referecedItem = GetReferencedItem(referencedId, indexable1.Item.Database, language);
                if (referecedItem == null)
                {
                    continue;
                }

                var contentValue = !string.IsNullOrEmpty(ContentField) ? GetFieldValue(referecedItem, ContentField) : null;

                if (string.IsNullOrEmpty(contentValue) && !string.IsNullOrEmpty(FallbackContentField))
                {
                    contentValue = GetFieldValue(referecedItem, FallbackContentField);
                }

                if (!string.IsNullOrEmpty(contentValue))
                {
                    referencedValues.Add(contentValue);
                }
            }

            return referencedValues;
        }

        protected virtual string GetFieldValue(Item item, string fieldName) => item.Fields[fieldName]?.Value;

        protected virtual IEnumerable<ID> GetIds(string fieldValue) => fieldValue
            .Split(new char[1] { '|' }, StringSplitOptions.RemoveEmptyEntries)
            .Where(ID.IsID)
            .Select(ID.Parse)
            .ToList();

        protected virtual Language Getlanguage(SitecoreIndexableItem indexable)
        {
            return indexable.Item?.Language;
        }

        protected virtual ID GetId(string id) => ID.IsID(id) ? ID.Parse(id) : null;

        protected virtual IIndexableDataField GetSourceField(IIndexable indexable, string fieldName)
        {
            return indexable.Fields.FirstOrDefault(field => field != null && field.Name.Equals(fieldName, StringComparison.OrdinalIgnoreCase));
        }

        protected virtual Item GetReferencedItem(ID id, Database database, Language language)
        {
            return database.GetItem(id, language);
        }
    }
  }

So with this bit of code we can now also use the resolved links feature with a fallback field, making it backwards compatible for us. Editors can keep on being lazy and fill in just the title - but they can overrule it if wanted and/or needed for the taxonomy to work.

Using the new code is the same as for the out-of-the-box resolvedlinks:

<sitecore>
  <contentSearch>
    <indexConfigurations search:require="solr">
      <defaultSolrIndexConfiguration>
        <documentOptions>
          <fields hint="raw:AddComputedIndexField">
            <field fieldName="F1" type="Sitecore.XA.Foundation.Search.ComputedFields.ResolvedLinks, Sitecore.XA.Foundation.Search" returnType="stringCollection" referenceField="Field1" contentField="title"/>
            <field fieldName="F2" type="Foundation.Taxonomy.Index.ResolvedLinksWithFallback, Foundation.Taxonomy" returnType="stringCollection" referenceField="Field2" contentField="taxonomytitle" fallbackContentField="title"/>
          </fields>
        </documentOptions>
      </defaultSolrIndexConfiguration>
    </indexConfigurations>
  </contentSearch>
</sitecore>

Once you move to SitecoreAI this becomes all obsolete, but for all those still working on Sitecore XM (with SXA - with or without JSS) this is hopefully still useful.


Tuesday, March 22, 2016

Custom index update strategy

ComputedIndexField with dependencies to other items

Ever had a ComputedIndexField that gathered data from other items than the item currently being indexed? For example, from its children. Or from items referred to..  I just had a situation where we needed (a property of) the children to be included in our ComputedIndexField.
But what happens if you update a child item? The child is re-indexed but the parent is not as this was not changed, not published, ...  We were using the onPublishEndAsync update strategy and didn't want to have a solution that needed a rebuild periodically just to keep the index up to date.

There is a GetDependencies pipeline that can be used to add extra items to the indexable list, but that was no option as this pipeline is for all indexes and we wanted it just for our custom index and preferably configurable per index (thinking about performance as well..)

Extend the onPublishEnd update strategy

So, we started thinking of extending the update strategy. We found some examples on the internet but not in our Sitecore version (we are using Sitecore 8.1) and the examples didn't go far enough.

What we wanted was:
  • the base onPublishEnd strategy (that would still check for rebuilds and so on)
  • an extension that would add the first ascendant of the item of a defined template to the list of indexable items

I had a look at the code of the OnPublishEndAsynchronousStrategy with DotPeek and noticed that this was extendable indeed. 

Let's start by creating our class by doing what a developer is good at: copy/paste :)

[DataContract]
public class OnPublishEndWithAncestorAsynchronousStrategy : OnPublishEndAsynchronousStrategy
{
    public string ParentTemplateId { get; set; }
    public string ChildTemplateId { get; set; }

    public OnPublishEndWithAncestorAsynchronousStrategy(string database) : base(database)
    {
    }
}

We created a class that extends OnPublishEndAsynchronousStrategy and gave it a constructor that needs the database name (which will be passed in the config). We also defined two variables to identify the templates that are affected - both parent (ancestor item to look for) as child (item to start from).

Performance

The child item template(s) are requested because our strategy code is executed before the crawler's root path is checked and before the 'DocumentOptions' are checked (like 'IncludeTemplates'). As this extended strategy is already heavier than the original one we wanted to prevent getting even more performance hits for items we don't need to check. This will become clear later on...

Configuration


<strategies hint="list:AddStrategy">
  <onPublishEndWithAncestorAsync type="Xx.OnPublishEndWithAncestorAsynchronousStrategy, Xx">
    <param desc="database">web</param>
    <ParentTemplateId>{0F5141D6-F264-4D03-B5D2-3505E6F308E7}</ParentTemplateId>
    <ChildTemplateId>{2A993FF2-5F17-4EEA-AD53-5343794F86BB}{066DEA00-31D7-4838-94A6-8D05A7FC690E}</ChildTemplateId>
  </onPublishEndWithAncestorAsync>
</strategies>

In the strategies section where you normally add your strategies by pointing towards the one(s) defined in the default Sitecore index configurations we define our custom strategy by providing the type. We send the database (web) as parameter and define the guids for the templates. In this example code we can send multiple child templates.

The index run

After some investigation, it turned out we only had to override one method: "Run". 
We started by copy/pasting the original code and checked the extension points:
  • if the item queue is empty: we leave the original code 
  • if the item queue is so big a rebuild is suggested: we keep the original code as a rebuild will also update the ancestor we might add
  • else..
We kept the original code to fetch the list of items to refresh. We don't actually get the items as "Item" but as "IndexableInfo" objects. For each entry in this list we call our GetAncestor function. The result is checked for null and added to the original list only if is wasn't already in there.


protected override void Run(List<QueuedEvent> queue, ISearchIndex index)
{
    CrawlingLog.Log.Debug($"[Index={index.Name}] {GetType().Name} executing.");
    if (Database == null)
    {
        CrawlingLog.Log.Fatal($"[Index={index.Name}] OperationMonitor has invalid parameters. Index Update cancelled.");
    }
    else
    {
        queue = queue.Where(q => q.Timestamp > (index.Summary.LastUpdatedTimestamp ?? 0L)).ToList();
        if (queue.Count <= 0)
        {
            CrawlingLog.Log.Debug($"[Index={index.Name}] Event Queue is empty. Incremental update returns");
        }
        else if (CheckForThreshold && queue.Count > ContentSearchSettings.FullRebuildItemCountThreshold())
        {
            CrawlingLog.Log.Warn($"[Index={index.Name}] The number of changes exceeded maximum threshold of '{ContentSearchSettings.FullRebuildItemCountThreshold()}'.");
            if (RaiseRemoteEvents)
            {
                IndexCustodian.FullRebuild(index).Wait();
            }
            else
            {
                IndexCustodian.FullRebuildRemote(index).Wait();
            }
        }
        else
        {
            var list = ExtractIndexableInfoFromQueue(queue).ToList();
            // custom code start here...
            CrawlingLog.Log.Info($"[Index={index.Name}] Found '{list.Count}' items from Event Queue.");
            var result = new List<IndexableInfo>();
            CrawlingLog.Log.Info($"[Index={index.Name}] OnPublishEndWithAncestorAsynchronousStrategy executing.");
            foreach (var itemInfo in list)
            {
                var ancestor = GetAncestor(itemInfo);
                if (ancestor != null)
                {
                    if (list.Any(i => i.IndexableId.Equals(ancestor.IndexableId, StringComparison.OrdinalIgnoreCase)))
                    {
                        CrawlingLog.Log.Info($"[Index={index.Name}] Ancestor already in list '{ancestor.IndexableId}'.");
                    }
                    else
                    {
                        CrawlingLog.Log.Info($"[Index={index.Name}] Adding ancestor '{ancestor.IndexableId}'.");
                        result.Add(ancestor);
                    }
                }
            }

            list.AddRange(result);
            CrawlingLog.Log.Info($"[Index={index.Name}] Updating '{list.Count}' items.");
            IndexCustodian.IncrementalUpdate(index, list).Wait();
        }
    }
}

Job(s)

One of the noticeable things here is that we add the extra indexable items to the existing list called with the incremental update. We could also call the Refresh method on the IndexCustodian but that would create extra (background) jobs so this way seems more efficient.

The ancestor check

Last thing to do is the ancestor check itself. For our requirements we needed to find an ancestor of a defined template but this functions could actually do anything. Just keep in mind the performance as this function will be called a lot.. (any ideas how to further improve this are welcome)

private IndexableInfo GetAncestor(IndexableInfo info)
{
    try
    {
 var childTemplateId = ChildTemplateId.ToLowerInvariant();
 var item = Database.GetItem(((ItemUri)info.IndexableUniqueId.Value).ItemID);
 if (item != null && childTemplateId.Contains(item.TemplateID.Guid.ToString("B")))
 {
     var ancestor = item.Axes.GetAncestors().ToList().FindLast(i => i.TemplateID.Guid.ToString("B").Equals(ParentTemplateId, StringComparison.OrdinalIgnoreCase));
     if (ancestor != null)
     {
  return new IndexableInfo(
                        new SitecoreItemUniqueId(
                            new ItemUri(ancestor.ID, ancestor.Language, ancestor.Version, Database)), 
                            info.Timestamp)
    {
        IsSharedFieldChanged = info.IsSharedFieldChanged
    };
     }
 }
    }
    catch (Exception e)
    {
 CrawlingLog.Log.Error($"[Index] Error getting ancestor for '{info.IndexableId}'.", e);
    }

    return null;
}


Using the child template in the config as well, might seems like a limitation but here it gives us a good performance gain because we limit the number of (slow) ancestor look-ups a lot. We still need to do that first lookup of the actual item to detect the template though.
We catch all exceptions - ok, might be bad practice - just to make sure in our test that one failure doesn't break it all.

Conclusion

As usual, we managed to tweak Sitecore in a fairly easy manor. This example can hopefully lead you towards more optimizations and other implementations of custom index features. Suggestions and/or improvements are welcome...


Thursday, October 16, 2014

Security in ComputedIndexFields

Custom ComputedIndexField

How to create a custom ComputedIndexField in Sitecore can be found on several other location on the net. But what if you want to check the security of your item when populating this field? 

A simple function could check if your item was readable by an anonymous user:
const string anonymous = @"extranet\Anonymous";
var account = Account.FromName(anonymous, AccountType.User);
return AuthorizationManager.IsAllowed(item, AccessRight.ItemRead, account);

This will work in a normal environment, but if you try this during population of your ComputedIndexField you will always get true. Why?

SitecoreItemCrawler

The responsible crawler - Sitecore.ContentSearch.SitecoreItemCrawler - is designed in such a way to wrap the calls for your ComputedIndexField in a SecurityDisabler. So, the true value in our security check is by design.

And maybe for the best.. it seems like a much better approach to perform security checks indexed results are being accessed. And how do I do this?

Security and index results


Well, you don't. Sitecore does this for you. When you query your index, Sitecore has a Security check build in to the pipeline:  <processor type="Sitecore.Pipelines.Search.SecurityResolver, Sitecore.Kernel" /> . This checks the security of the item involved, not of any related items that you might have used to fill your ComputedIndexField. If you also want a security check on those, write a processor to extend the pipeline :)

Tuesday, October 14, 2014

ComputedIndexField with extra attributes

ComputedIndexField attributes

In the code that populates my ComputedIndexField I needed a reference to a Sitecore item. I needed to know a "context" item from which I had t retrieve data. Apparently you can extend Sitecore to do this (off course).

My configuration look like this (in a separate config file):

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
 <sitecore>
  <contentSearch>
   <configuration type="Sitecore.ContentSearch.LuceneProvider.LuceneSearchConfiguration, 
                                  Sitecore.ContentSearch.LuceneProvider">
    <indexes hint="list:AddIndex">
     <index id="MyIndexName" 
         type="Sitecore.ContentSearch.LuceneProvider.LuceneIndex,Sitecore.ContentSearch.LuceneProvider">
      <configuration type="Demo.Search.LuceneIndexConfiguration, Demo">
       <documentOptions type="Sitecore.ContentSearch.LuceneProvider.LuceneDocumentBuilderOptions">
        <ProcessDependencies>true</ProcessDependencies>
       </documentOptions>
       <fields hint="raw:AddComputedIndexField">
        <field fieldName="DemoContent" label="{1174978C-47CA-405D-13FE-4C808F3A85E7}">
          Demo.Search.DemoField, Demo
        </field>
       </fields>
      </configuration>
     </index>
    </indexes>
   </configuration>
  </contentSearch>
 </sitecore>
</configuration>

 So I added an attribute "label" in my field definition. Now we need a custom IndexConfiguration, and we will start with the one from Sitecore (reverse engineer that one and you will see we only added a few lines).

Custom IndexConfigration

public class LuceneIndexConfiguration : Sitecore.ContentSearch.LuceneProvider.LuceneIndexConfiguration
{
  public override void AddComputedIndexField(XmlNode configNode)
  {
    Assert.ArgumentNotNull(configNode, "configNode");
    string fieldName = XmlUtil.GetAttribute("fieldName", configNode, true);
    string type = XmlUtil.GetValue(configNode);            
    if (string.IsNullOrEmpty(fieldName) || string.IsNullOrEmpty(type))
    {
      throw new InvalidOperationException("Could not parse computed index field entry: " + configNode.OuterXml);
    }
    var field = ReflectionUtil.CreateObject(type) as IComputedIndexField;

    // here starts custom code
    var constructableField = field as IConstructable;
    if (constructableField != null)
    {
     constructableField.Constructed(configNode);
    }
    // end custom code

    if (field != null)
    {
      field.FieldName = fieldName.ToLowerInvariant();
      DocumentOptions.ComputedIndexFields.Add(field);
    }
  }
}

The extra stuff is based on the IConstuctable interface. This allows you to created a Constructed method that can read your xml node. You can use your own interfaces as well surely, but this one is already in the box..

And then we adapt the code for our ComputedIndexField:

IConstructable

public class DemoField : Sitecore.ContentSearch.ComputedFields.IComputedIndexField, IConstructable
{
  private Item MyItem { get; set; }

  public void Constructed(XmlNode configuration)
  {
    var label = XmlUtil.GetAttribute("label", configuration, true);
    if (string.IsNullOrEmpty(label))
    {
      ....
      return;
    }

    Guid id;
    if (Guid.TryParse(label, out id))
    {
      MyItem = Factory.GetDatabase("master").GetItem(new ID(id));
    }
  }

  public object ComputeFieldValue(IIndexable indexable)
  {
    ...
  }
}

We implement the Constructed method and read the xml node. The GetAttribute function is very useful here. We set the MyItem that will be used later on in the ComputedFieldValue function called by the crawler.