Custom Sitecore DocumentOptions with Solr

Almost 2 years ago I wrote a post about using custom indexes in a Helix environment. That post is still accurate, but the code was based on Lucene. As we are now all moving towards using Solr with our (non-PAAS) Sitecore setups, I though it might be a good idea to bring this topic back on the table with a Solr example this time.

(custom) indexes

I am assuming that you know about Helix, and about custom indexes. If you ever created a custom index you probably have used the documentOptions configuration section - maybe without noticing. It is used to include and/or exclude fields and templates and define computed fields. So you probably used it :)

And it wouldn't be Sitecore if we couldn't customize this...

Our own documentOptions

Why? Because we can. No..  we might have a good reason, like making our custom index definitions (more) Helix compliant. Normally your feature will not have a clue about "page" templates. But what if you want to define the include templates in your index? Those could be page templates.. or at least templates that inherit from your feature template. That is why I build my own documentOptions - to include a way to include templates derived from template-X.

Configuration

So the idea now is to create a custom document options class by inheriting from the SolrDocumentBuilderOptions. We add a new method to allow adding templates in a new section with included base templates. This will not break any other existing configuration sections.

An example config looks like:

<documentOptions type="YourNamespace.TestOptions, YourAssembly">
    <indexAllFields>true</indexAllFields>
    <include hint="list:AddIncludedBaseTemplate">
        <BaseTemplate1>{B6FADEA4-61EE-435F-A9EF-B6C9C3B9CB2E}</BaseTemplate1>
    </include>
</documentOptions>

This looks very familiar - as intended. We create a new include section with the hint "list:AddIncludedBaseTemplate". The name 'AddIncludedBaseTemplate' will come back later in our code.

Code

AddIncludedBaseTemplate

public virtual void AddIncludedBaseTemplate(string templateId)
{
  Assert.ArgumentNotNull(templateId, "templateId");
  ID id;
  Assert.IsTrue(ID.TryParse(templateId, out id), "Configuration: AddIncludedBaseTemplate entry is not a valid GUID. Template ID Value: " + templateId);
  foreach (var linkedId in GetLinkedTemplates(id))
  {
    AddTemplateFilter(linkedId, true);
  }
}

To see the rest of the code, I refer to the original post as nothing has to be changed to that in order to make it work on Solr (instead of Lucene).

Conclusion

To change the code from the Lucene example to a Solr one, we just had to change the base class to SolrDocumentBuilderOptions. 
We are now again able to configure our index to only use templates that inherit from our base templates. Still cool. And remember you can easily re-use this logic to create other document options to tweak your index behavior. 

Custom indexes in a Sitecore Helix architecture

Sitecore Helix/Habitat

Most Sitecore developers probably know what Sitecore Habitat and Sitecore Helix is about right now, especially since the 2016 Hackathon. Several interesting blog posts have already been written about the architecture (e.g. this one by Anders Laub) and video's have been posted on Youtube by Thomas Eldblom.

Custom indexes

I use custom indexes very frequently. So I started thinking about how I could use custom indexes in a "Helix" architecture. When creating a feature that uses such a custom index, the configuration for that index has to be in the feature. That is perfectly possible as we can create a separate index config file. But what are the things we define in that config file?

• index
• a name
• the strategy
• crawler(s)
• ...

• index configuration
• field map
• document options
• computed index fields
• include templates
• ...
• ...

Some of these settings can be defined in our feature without issue: the name -obviously- and the stategy (e.g. onPublishEndAsync) can be defined. A crawler might be the first difficulty but this can be set to a very high level (e.g. <Root>/sitecore/content</Root>)

In the index configuration we can also define the fieldMap and such. In the documentOptions section we can (must) define our computed index fields. But then we should define our included templates. And that was were I got stuck..  in a feature I don't know my template, just base templates..

Patching

A first thought was to use the patching mechanism from Sitecore. We could define our index and it's configuration in the feature and patch the included templates and/or crawlers in the project.
Sounds like a plan, but especially for the included templates it didn't feel quite right.

For the index itself patching will be necessary in some cases.. e.g. to enable or disable Item/Field language fallback. If needed it is also possible to patch the content root in the project level.

Included templates? Included base templates!

For the included templates in the document options I was searching for another solution so I decided to throw my question on the Sitecore Slack Helix/Habitat channel and ended up in a discussion with Thomas Eldblom and Sitecore junkie Mike Reynolds. Thomas came up with the idea to hook into the index process to enable it to include base templates and Mike kept pushing me to do it and so.. I wrote an extension to configure your index based on base templates.

The code is a proof of concept.. it can -probably- be made better still but let this be a start.

Custom document options

I started by taking a look at one of my custom indexes to see what Sitecore was doing with the documentOptions sections and took a look at their code in Sitecore.ContentSearch.LuceneProvider.LuceneDocumentBuilderOptions. As you can guess, the poc is done with Lucene..

Configuration

The idea was to create a custom document options class by inheriting from the LuceneDocumentBuilderOptions. I could add a new method to allow adding templates in a new section with included base templates. This will not break any other configuration sections.

An example config looks like:

<documentOptions type="YourNamespace.TestOptions, YourAssembly">
    <indexAllFields>true</indexAllFields>
    <include hint="list:AddIncludedBaseTemplate">
        <BaseTemplate1>{B6FADEA4-61EE-435F-A9EF-B6C9C3B9CB2E}</BaseTemplate1>
    </include>
</documentOptions>

This looks very familiar - as intended. We create a new include section with the hint "list:AddIncludedBaseTemplate". The name 'AddIncludedBaseTemplate' will come back later in our code.

Code

Related templates

The first function we created was to get all templates that relate to our base template:

private IEnumerable<Item> GetLinkedTemplates(Item item)
{
  var links = Globals.LinkDatabase.GetReferrers(item, new ID("{12C33F3F-86C5-43A5-AEB4-5598CEC45116}"));
  if (links == null)
  {
    return new List<Item>();
  }

  var items = links.Select(i => i.GetSourceItem()).Where(i => i != null).ToList();
  var result = new List<Item>();
  foreach (var linkItem in items)
  {
    result.AddRange(GetLinkedTemplates(linkItem));
  }

  items.AddRange(result);
  return items;
}

We use the link database to get the referrers and use the Guid of the "Base template" field of a template to make sure that we get references in that field only - which also makes sure that all results are actual Template items.
The function is recursive because a template using your base template can again be a base template for another template (which will by design also include your original base template). The result is a list of items.

A second function will use our first one to generate a list of Guids from the ID of the original base template:

public IEnumerable<string> GetLinkedTemplates(ID id)
{
  var item = Factory.GetDatabase("web").GetItem(id);
  Assert.IsNotNull(item, "Configuration : templateId cannot be found");

  var linkedItems = GetLinkedTemplates(item);
  return linkedItems.Select(l => l.ID.Guid.ToString("B").ToUpperInvariant()).Distinct();
}

As you can see what we do here is try to fetch the item from the id and call our GetLinkedTemplates function. From the results we take the distinct list of guid-strings - in uppercase.

Context database

One big remark here is the fact that I don't know what the database is - if somebody knows how to do that, please let me (and everybody) know. The context database in my tests was 'core' - I tried to find the database defined in the crawler because that is the one you would need but no luck so far.

And finally...

AddIncludedBaseTemplate

public virtual void AddIncludedBaseTemplate(string templateId)
{
  Assert.ArgumentNotNull(templateId, "templateId");
  ID id;
  Assert.IsTrue(ID.TryParse(templateId, out id), "Configuration: AddIncludedBaseTemplate entry is not a valid GUID. Template ID Value: " + templateId);
  foreach (var linkedId in GetLinkedTemplates(id))
  {
    AddTemplateFilter(linkedId, true);
  }
}

Our main function is called "AddIncludedBaseTemplate" - this is consistent with the name used in the configuration. In the end we want to use the "AddTemplateFilter" function from the base DocumentBuilderOptions - the 'true' parameter is telling the function that the templates are included (false is excluded). So we convert the template guid coming in to an ID to validate it and use it in the functions we created to get all related templates.

Performance

Determining your included templates is apparently only done once at startup. So if you have a lot of base templates to include which have lots of templates using them, don't worry about this code being called on every index update. Which of course doesn't mean we shouldn't think about performance here ;)

Conclusion

So we are now able to configure our index to only use templates that inherit from our base templates. Cool. Does it end here? No.. you can re-use this logic to create other document options as well to tweak your index behavior. 

And once more: thanks to Thomas & Mike for the good chat that lead to this.. The community works :)

SOLID Sitecore templates

Every programmer in an object-oriented environment should know the S.O.L.I.D. principles:

• Single Responsibility
• Open/closed
• Liskov substitution
• Interface segregation
• Dependency inversion

More information about this topic can be found all over the internet, on wikipedia.

What I want to talk about here is using these principles when working with Sitecore, and Sitecore templates in particular. It will come down to a basic conclusion.
I hope that for most of I will not tell anything new, but I've seen quite a lot of code already that does not apply to these principles and the reason is almost always: we had to do it "fast".

They are lots of blog posts already over solid principles in C# which of course also apply to a Sitecore based solution. I will not repeat that but focus on the architecture of the Sitecore templates - not quite the same as oo-coding but as you will see the solid principles even have their meaning here.

Single Responibility

"A template should have only a single responsibility"

Do not use a template called "News" for something else then a news article. Even if your other item has the same fields. You will create confusion and the maintenance of the templates will become harder. You might consider using base templates if your items have the same fields, but there also goes the single responsibility principle.

Base templates
Do not create base templates that have more than a single responsibility. If you want to re-use fields for SEO purposes, create a seo base template. Do not mix them with e.g. fields for your navigation in a single global "page" base template. If you really need such a 'base', create it from several other base templates.
A base template that has more than a single section should get you thinking...

Open/Closed

"An entity should be open for extension, but closed for modification"

For this one my example does not apply to the template itself, but to functions that switch on template(-id)'s. Quite often I get to see code that has a set of if-statements (or a switch) to identify the template of an item and do something somehow related but still different per template. After a while you might even get to see these if/switch statements more than once in the same class. If you create a new template you might break those function(s) because your new template will not be set in the selections.

How can we make this better maintainable? Create an interface (if you don't already have one, as you should) and use that interface to the outside world. Implement the interface for each (set of) template(s) that you need - use a base class with virtual functions if needed. In front of those classes you create a façade (class) that does the template detection and uses the matching implementation of the interface. The consumers of your code will always see the interface and the implementing classes do not need to check templates anymore.
If you create a new template you need to check your façades and maybe create a new implementation of an interface. But you don't need to scan your solution for template detecting statements anymore.

Liskov substitution

"Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program"

Applied on templates this comes down to a few simple rules that everyone finds very obvious and still.. the "fast" way, you know.

• Do not "overwrite" fields from a base template: if a base template already has a field "Title" do not create another "Title" field. Also make sure you do not have multiple base templates that include the same fieldname. 
• Do not re-use a base template field for a purpose it was not designed. You will not only confuse your fellow programmers, but also you content editors.

Interface segregation

"Specific interfaces are better than one general-purpose interface"

"Clients should not be forced to implement interfaces they don't use"

Applying this on Sitecore templates comes down to using base templates. Use base templates, and use them wisely. Create a base template for every need and make sure that your templates do not end up with fields they do no use. 

Unused fields will confuse your content editors, it might give them the impression that they lack functionality. 

Dependency inversion

"One should depend upon Abstractions. Do not depend upon concretions. A high-level modules or class should not depend upon low-level modules or classes"

This principle is mostly implemented with Dependency Injection. As there are already lots of blog posts already on that subject (also in combination with Sitecore) I won't go into details here. And I couldn't find an example related to templates..

One thing worth mentioning however is not to think that you are done when using inversion of control. If you want you classes/function independent and testable you must also make sure they do not use Sitecore.Context. A simple (but often forgotten) one is using the "current language". Your business functions should ask for the language if they need it (a parameter) and not depend on the Sitecore.Context!

Conclusion

Wrapping this up for using templates one could say:

• use base templates
• use base templates wisely
• use small base templates wisely

and try to keep your business logic code independent of Sitecore.Context and templateId's.