Monday, May 13, 2019

Generic caching framework for Sitecore with relevant clearance

Caching with clearance on a relevant publish

As you all know caching is important and used by Sitecore a lot. But after a publish Sitecore clears caches rather like an elephant - just remove it no matter what was published. We have been using a custom framework for quite a while that allows us to use granular caching of data - data that we have put in our own cache that is, not the actual Sitecore caches - and clearance only when relevant.
It's a long post.. but mainly just a lot of code ;)


The idea

The idea behind it is rather simple: you can put objects in the cache and you add a removal strategy for that cache entry. After a publish we run a cache clear process that calls all the removal strategies. Each strategy can decide on the information from the publish if the cache entry should be removed or not. This way we make each cache entry responsible for its own removal logic. Which is good as it makes it very flexible and we keep the logic together with the actual data. Note that the removal strategy is not limited to removing the object from the cache - it might also perform other actions if needed.

The code

Before we take a look at the code, let's check the dependency graph:
Let 's start with the CacheEntry class. This is the object we will store in the cache. It contains the value (actual object you want to cache) and the PublishRemoveStrategy. It also contains the function to invoke the removal strategy.
public class CacheEntry : ICacheEntry
{
  public CacheEntry(object valueToCache)
  {
    CachedValue = valueToCache;
  }

  public CacheEntry(object valueToCache, PublishRemoveStrategy publishRemoveStrategy)
  {
    CachedValue = valueToCache;
    PublishRemoveStrategy = publishRemoveStrategy;
  }

  public object CachedValue { get; }

  public PublishRemoveStrategy PublishRemoveStrategy { get; }

  /// <summary>
  /// Runs the on-publish remove strategy for the entry.
  /// The strategy (if provided) is expected to do the necessary work such as actually removing the entry from the cache.
  /// </summary>
  /// <param name="cache">Cache instance</param>
  /// <param name="key">Cache entry key</param>
  /// <param name="language">Publish language</param>
  /// <param name="database">Target Database</param>
  /// <param name="rootNode">Root node of the publish action</param>
  /// <param name="publishMode">Publish mode</param>
  public virtual void RunOnPublishRemoveStrategy(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode)
  {
    PublishRemoveStrategy?.Invoke(cache, key, language, database, rootNode, publishMode);
  }
}
The PublishRemoveStrategy strategy itself is a delegate that requires the language, the target database, the root node and the mode (of the publish):
public delegate void PublishRemoveStrategy(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode);

The core of the module is the actual SitecoreCache class:
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Globalization;
using Sitecore.Publishing;

public class SitecoreCache : ISitecoreCache
{
  private const string LanguageIndependentKey = "xx";

  public virtual void Add(string key, ICacheEntry cacheEntry, string language)
  {
    HttpRuntime.Cache.Insert(BuildKey(key, language), cacheEntry);
  }

  public virtual void Add(string key, ICacheEntry cacheEntry)
  {
    Add(key, cacheEntry, null);
  }

  public virtual object GetObject(string key, string language)
  {
    var cacheObject = HttpRuntime.Cache[BuildKey(key, language)] as ICacheEntry;
    return cacheObject?.CachedValue;
  }

  public virtual object GetObject(string key)
  {
    return GetObject(key, null);
  }

  public virtual T GetObject<T>(string key, string language) where T : class
  {
    return GetObject(key, language) as T;
  }

  public virtual T GetObject<T>(string key) where T : class
  {
    return GetObject<T>(key, null);
  }

  public virtual void Remove(string key, string language)
  {
    HttpRuntime.Cache.Remove(BuildKey(key, language));
  }

  public virtual void Remove(string key)
  {
    Remove(key, null);
  }

  public virtual void RemoveItemsOnPublish(Database database, Item rootNode, PublishMode publishMode, Language language)
  {
    if (language == null)
    {
    return;
    }

    var prefixLanguage = BuildKey(string.Empty, language.Name);
    var entriesInLanguage = from DictionaryEntry entry in HttpRuntime.Cache
          let key = entry.Key as string
          where IsValidKey(key, prefixLanguage)
          select entry;

    foreach (var entry in entriesInLanguage)
    {
    if (!(entry.Value is ICacheEntry cacheObject))
    {
      continue;
    }

    var key = ExtractKey(entry.Key.ToString());
    cacheObject.RunOnPublishRemoveStrategy(this, key, language, database, rootNode, publishMode);
    }
  }

  public virtual void Clear()
  {
    var keys = (from DictionaryEntry entry in HttpRuntime.Cache where entry.Value is ICacheEntry select entry.Key).ToList();

    foreach (string key in keys)
    {
      Remove(key);
    }
  }

  private static string BuildKey(string key, string language)
  {
    var prefix = language?.ToLowerInvariant() ?? LanguageIndependentKey;
    return prefix + "|" + key;
  }

  private static string ExtractKey(string compositeKey)
  {
    return compositeKey.Substring(compositeKey.IndexOf('|') + 1);
  }

  private static bool IsValidKey(string key, string prefixLanguage)
  {
    return !string.IsNullOrEmpty(key)
    && (key.StartsWith(prefixLanguage, StringComparison.OrdinalIgnoreCase) || key.StartsWith(LanguageIndependentKey, StringComparison.OrdinalIgnoreCase));
  }
}

The SitecoreCache class has functions to:
  • Add entries: with or without language (some entries might be language dependent, others are not)
  • Get entries: again with or without the language, but also a (generic) version that will try to cast to a given type
  • Remove entries or Clear all entries
  • Handle the publish event: this gets all entries from the cache and calls their remove strategy
Entries are stored with a key (any string -should be unique- given by the using application that is combined with the language).

Clearing the cache

Clearing the cache is done in the CacheClearer class that we will attach to publish events:
using Sitecore.Common;
using Sitecore.Configuration;
using Sitecore.Data;
using Sitecore.Data.Events;
using Sitecore.Events;
using Sitecore.Globalization;
using Sitecore.Publishing;

public class CacheClearer
{
  private readonly ISitecoreCache sitecoreCache;

  public CacheClearer(ISitecoreCache sitecoreCache)
  {
    this.sitecoreCache = sitecoreCache;
  }

  public event EventHandler SitecoreCacheClearedHandler;

  public virtual void ClearCache(object sender, EventArgs args)
  {
    var publishingOptions = FetchPublishingOptions(args).ToList();

    if (!publishingOptions.Any())
    {
      return;
    }

    if (sitecoreCache == null)
    {
      return;
    }

    RemoveFromCache(publishingOptions);
  }

  protected virtual void AfterSitecoreCacheCleared(EventArgs e)
  {
    if (SitecoreCacheClearedHandler == null)
    {
      return;
    }

    SitecoreCacheClearedHandler(this, e);
  }

  protected virtual IEnumerable<DistributedPublishOptions> FetchPublishingOptions(EventArgs args)
  {
    var sitecoreRemoteArgs = args as PublishCompletedRemoteEventArgs;
    if (sitecoreRemoteArgs != null)
    {
      if (sitecoreRemoteArgs.PublishOptions != null)
      {
        return sitecoreRemoteArgs.PublishOptions;
      }

      return new List<DistributedPublishOptions>();
    }

    var sitecoreArgs = args as SitecoreEventArgs;
    if (sitecoreArgs == null)
    {
      return new List<DistributedPublishOptions>();
    }

    var publishingParameter = sitecoreArgs.Parameters[0] as IEnumerable<DistributedPublishOptions>;

    if (publishingParameter != null)
    {
      return publishingParameter;
    }

    return new List<DistributedPublishOptions>();
  }

  protected virtual void RemoveFromCache(IEnumerable<DistributedPublishOptions> publishingOptions)
  {
    foreach (var publishingOption in publishingOptions)
    {
      var database = GetDatabase(publishingOption.TargetDatabaseName);
      if (database == null)
      {
        continue;
      }

      RemoveFromCache(database, publishingOption.RootItemId.ToID(), publishingOption.LanguageName, publishingOption.Mode);
    }

    AfterSitecoreCacheCleared(EventArgs.Empty);
  }

  protected virtual void RemoveFromCache(Database database, ID id, string languageName, PublishMode mode)
  {
    Language publishingLanguage;
    if (!Language.TryParse(languageName, out publishingLanguage))
    {
      return;
    }

    var rootItem = database.GetItem(id, publishingLanguage);
    sitecoreCache.RemoveItemsOnPublish(database, rootItem, mode, publishingLanguage);
  }

  protected virtual Database GetDatabase(string name)
  {
    if (!Factory.GetDatabaseNames().Contains(name, StringComparer.OrdinalIgnoreCase))
    {
      return null;
    }

    return Factory.GetDatabase(name);
  }
}
In this class we are actually not doing much more than fetching the publish options and getting the parameters to call the RemoveItemsOnPublish from our SitecoreCache: the target database, the root item, the language and the publish mode. When fetching the publish options, we are trying twice to support local and remote events. We could also split the functions but decided not to. The SitecoreCacheClearedHandler event can be used to inject extra code after the clear.

Attaching the clearance code to the events is done with configuration:
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <events>
      <event name="publish:complete">
        <handler type="XX.Caching.CacheClearer, XX" method="ClearCache" resolve="true" />
      </event>
      <event name="publish:complete:remote">
        <handler type="XX.Caching.CacheClearer, XX" method="ClearCache" resolve="true" />
      </event>
    </events>
  </sitecore>
</configuration>
Note that the clear code is attached to the local and the remote publish event to support scaled architectures.


Cache removal strategies

The last part of the puzzle is the actual cache removal strategies. You can easily write your own one if needed and use that, but we provided some that cover most of our needs. Each removal strategy class must implement the ICacheRemoval interface which has two functions: one for clearing the cache with a language and one without:
public interface ICacheRemovalStrategy where T : class
{
  void RemoveInLanguage(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode);

  void RemoveLanguageIndependent(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode);
}

Four examples of a removal strategy:
1. Always: this strategy will always clear the cache, no matter what the options are.
public class Always : ICacheRemovalStrategy<object>
{
  public virtual void RemoveInLanguage(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode)
  {
    if (cache != null && language != null)
    {
      cache.Remove(key, language.Name);
    }
  }

  public virtual void RemoveLanguageIndependent(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode)
  {
    if (cache != null)
    {
      cache.Remove(key);
    }
  }
}

2. On root node: this will remove the entry if the root node of the publish is the item or one of its descendants of the given parent - the parent is injected through a method that fetches the item based on the database.
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Globalization;
using Sitecore.Publishing;

public class OnRootNode : ICacheRemovalStrategy<Item>
{
  public OnRootNode(Func<Database, Item> dependency)
  {
    GetDependencies = dependency;
  }

  public Func<Database, Item> GetDependencies
  {
    get;
    set;
  }

  public virtual void RemoveInLanguage(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode)
  {
    if (cache == null || string.IsNullOrEmpty(key) || language == null)
    {
      return;
    }

    if (IsDescendant(database, rootNode))
    {
      cache.Remove(key, language.Name);
    }
  }

  public virtual void RemoveLanguageIndependent(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode)
  {
    if (cache == null || string.IsNullOrEmpty(key))
    {
      return;
    }

    if (IsDescendant(database, rootNode))
    {
      cache.Remove(key);
    }
  }

  private bool IsDescendant(Database database, Item publishRootNode)
  {
    if (publishRootNode == null)
    {
      return false;
    }

    var parent = GetDependencies?.Invoke(database);
    
    if (parent == null)
    {
      return false;
    }

    return publishRootNode.ID.Guid.Equals(parent.ID.Guid) || publishRootNode.Paths.IsDescendantOf(parent);
  }
}

3. On templates: this will remove the entry if the id of the template of the root node of the publish is amongst the given template id's
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Globalization;
using Sitecore.Publishing;

public class OnTemplates : ICacheRemovalStrategy<IEnumerable<Guid>>
{
  public OnTemplates(Func<IEnumerable<Guid>> dependency)
  {
    GetDependencies = dependency;
  }

  public Func<IEnumerable<Guid>> GetDependencies
  {
    get;
    set;
  }

  public virtual void RemoveInLanguage(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode)
  {
    if (cache == null || string.IsNullOrEmpty(key) || language == null)
    {
      return;
    }

    if (IsTemplateBased(rootNode))
    {
      cache.Remove(key, language.Name);
    }
  }

  public virtual void RemoveLanguageIndependent(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode)
  {
    if (cache == null || string.IsNullOrEmpty(key))
    {
      return;
    }

    if (IsTemplateBased(rootNode))
    {
      cache.Remove(key);
    }
  }

  private bool IsTemplateBased(Item publishRootNode)
  {
    if (publishRootNode == null)
    {
      return false;
    }

    var templates = GetDependencies?.Invoke();
    
    if (templates == null)
    {
      return false;
    }

    return templates.ToList().Contains(publishRootNode.TemplateID.Guid);
  }
}

4. On root node & template: combination of the 2 above
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Globalization;
using Sitecore.Publishing;

public class TemplatesRootNode
{
  public Item RootNode { get; set; }
  public IEnumerable<Guid> Templates { get; set; }
}

public class OnTemplatesRootNode : ICacheRemovalStrategy<TemplatesRootNode>
{
  public OnTemplatesRootNode(Func<TemplatesRootNode> dependency)
  {
    GetDependencies = dependency;
  }

  public Func<TemplatesRootNode> GetDependencies
  {
    get;
    set;
  }

  public virtual void RemoveInLanguage(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode)
  {
    if (cache == null || string.IsNullOrEmpty(key) || language == null)
    {
      return;
    }

    var dependency = Dependencies();
    
    if (IsTemplateBased(rootNode, dependency) && IsDescendant(rootNode, dependency))
    {
      cache.Remove(key, language.Name);
    }
  }

  public virtual void RemoveLanguageIndependent(ISitecoreCache cache, string key, Language language, Database database, Item rootNode, PublishMode publishMode)
  {
    if (cache == null || string.IsNullOrEmpty(key))
    {
      return;
    }

    var dependency = Dependencies();
    
    if (IsTemplateBased(rootNode, dependency) && IsDescendant(rootNode, dependency))
    {
      cache.Remove(key);
    }
  }

  private TemplatesRootNode Dependencies()
  {
    return GetDependencies?.Invoke();
  }

  private static bool IsTemplateBased(Item publishRootNode, TemplatesRootNode dependency)
  {
    if (publishRootNode == null)
    {
      return false;
    }

    if (dependency == null)
    {
      return false;
    }

    return dependency.Templates.ToList().Contains(publishRootNode.TemplateID.Guid);
  }

  private static bool IsDescendant(Item publishRootNode, TemplatesRootNode dependency)
  {
    if (publishRootNode == null)
    {
      return false;
    }

    var parent = dependency?.RootNode;

    if (parent == null)
    {
      return false;
    }

    return publishRootNode.ID.Guid.Equals(parent.ID.Guid) || publishRootNode.Paths.IsDescendantOf(parent);
  }
}

Usage

If you're still here reading after such a long post, you probably also want to see an example of how to use this:
var rootNode = .. // get the parent item
var removeStrategy = new OnTemplatesRootNode(() => GetRemovalStrategy(rootNode));
sitecoreCache.Add(CacheKey, new CacheEntry(myObject, removeStrategy.RemoveInLanguage), language);

private TemplatesRootNode GetRemovalStrategy(Item rootItem)
{
  return new TemplatesRootNode
  {
    RootNode = rootItem,
    Templates = ..  // get the id's of the templates
  };
}
In this example I am caching a language dependent object and am using the combination of templates and a root node. The calls to fetch the parent node and the ids of the templates are left blank as those are really dependent on your solution.

Conclusion

I hope this all made some sense (and I didn't forget anything). It has proven to be a rather powerful way of caching data and clearing the data based on relevant Sitecore publish events. It's not perfect yet -there are still some ideas to extend and improve- but it's a good start. So if you did get this far I hope you enjoyed it - any feedback is welcome.

One last remark: be aware that objects returned from the cache are references - if you change them you also change the cached value! If you don't want this, take a copy first.

Thanks to Özkan Sayın for reading and correcting the draft version.


Thursday, April 25, 2019

Publishing target in Sitecore SXA

Publishing target

Sitecore has the notion of publish targets that you can use to publish to different databases. This can be several live databases if you need that, or in many cases this is used for a "preview" database in order to have your editors publish to this preview database and verify the results before actually publishing to the live environment. 
Creating a publishing target has been documented already, so I will be very brief on this. Sitecore describes on their documentation how you can use this in combination with workflows. You need to add an actual extra database (tip: copy your web database) to work with and configure this one. Note that a few things changed in Sitecore 9 - as noted here on SSE.  A good start is the blog post of Peter Procházka on how to add a publish target.

Preview

For the scope of this article, I am focusing on using a publish target as a preview target. So the idea is that editors can publish to this target and see what they have created before publishing to the live site. This has some benefits over using the preview mode in Sitecore as that might not behave exactly as it will on the live environment. 
Assume our new target is called "preview" and we have created the database and all configs as mentioned in the above posts.

Indexes

One thing I am missing in a lot of posts on the subject is an extra index. Your code might rely on indexes (it should) but if you don't have an index to match that preview you will get strange results. So, create an extra index called sitecore_preview_index. You can copy the configuration from your web_index as it should act the same. Just don't forget to change the database in the crawler. 

SXA

The last step in configuring a preview target is normally to point a site to the preview database. That will be our test site. Normally I just use the ContentManagement site for this. In standard Sitecore this can be easily done in the configuration but in SXA I had to use a different approach.

First of all though: don't forget SXA also has it's own index..  so create a sitecore_sxa_preview_index as well (copy the config from your sxa_web_index, not from the other preview index as the sxa indexes have some different types).

Using SXA sites

In SXA you have a site definition in Sitecore itself (/sitecore/content/.../Settings/Site Grouping). This means you cannot have different values for ContentManagement and ContentDelivery in the same item (in config files that was easy). But there are solutions...  let's start by creating a second site item next to the one you already have (duplicate it to save time). 

The new site item should have all the same properties as the original one, except:
  • set the database to "preview"  (if your preview database is in the config, it should be in the list) 
  • add the preview index in the index config 

Routing the sites

Now we need to make sure that the CM is using the new site and the CD keeps using the original one. First thing to do is set the hostnames. The new site should not pick the CD urls and the original one doesn't need the CM url anymore. In some cases, this can be sufficient - it depends on the urls you have. In some cases however you might have wildcards and such that make this a first good step but not enough to make the CM go to the preview site.

SXA Environment parameter

There is one more parameter we can use - the "environment". By default this will be set to "*" on your site item. Meaning the site definition is valid for all environments. We can use this to distinguish between CM and CD environments! The value for the environment is not the same one as used in the Sitecore 9 configuration (the "role") - it is a value from a setting that we can patch on CM:
<settings>
  <setting name="XA.Foundation.Multisite.Environment" value="PREVIEW" role:require="ContentManagement" />
</settings>

So lets alter the values on our site items:
On the new preview site item we set the value to PREVIEW (same name as used in the setting). I noticed that I still had issues when the original one was left on *, so I changed that to LIVE - which is the default value of the environment parameter.

And yeah.. it works.
Not sure if this is the only way or the best way to do this, so if anyone has alternative ways to have publish targets combined with SXA please let me know ;)

Tuesday, April 23, 2019

Unit testing Sitecore index queries in 9.1

Unit testing Sitecore index queries

I will make a few assumptions in this post:
  • You know what unit testing is
  • You understand the benefits of unit testing
  • You are using Sitecore indexes in your code 
  • and so you would like to unit test that code 
If you don't understand the benefits yet.. consider this:
You have some code that needs to be tested with a number of variables (date in the past or in the future, positive or negative numbers ...). In order to test this properly you might need to create several items in Sitecore, run several manual tests and best of all - restart your Sitecore environment after every deploy with a bugfix if your code wasn't perfect from the start (and be honest, is that ever...)

Unit testing queries in 9.1

I have used code based on a solution by Vivian Roberts for a lot of projects to test code with queries on indexes. It works fine (as long as I don't have facets and such) and has saved me a bit of time already.

And then came Sitecore 9.1. And it didn't work anymore..  Sitecore introduced a new internal interface that is checked in the QueryableExtensions and that makes our test code fail:
if (!QueryableExtensions.IsIContentSearchQueryable<TSource>(source))
    return (SearchResults<TSource>) null;
The solution apparently was to inherit from Sitecore.ContentSearch.Linq.Parsing.GenericQueryable which is public and inherits the mentioned interface.  Thanks to my colleague Alex Dhaenens and Sitecore Support for helping us out on this one.

Let's recap with a complete overview of the code I'm using to test queries.

The code

Step 1: make the code testable

To make it all testable I create a SearchContextBuilder:
public class SearchContextBuilder : ISearchContextBuilder
{
  public virtual IProviderSearchContext GetSearchContext(string index)
  {
    return ContentSearchManager.GetIndex(index).CreateSearchContext();
  }
}
This builder will be the default that is injected in all our repositories that query indexes. In the test version however, we will be able to inject another one.

Generic Search repository

You could use the context builder in a generic search repository, something like:
public class SearchRepository : ISearchRepository
{
  private readonly ISearchContextBuilder searchContextBuilder;

  public SearchRepository(ISearchContextBuilder searchContextBuilder)
  {
    this.searchContextBuilder = searchContextBuilder;
  }

  public virtual SearchResults<T> GetResults<T>(Expression<Func<T, bool>> predicate, string index) where T : SearchResultItem
  {
    using (var context = searchContextBuilder.GetSearchContext(index))
    {
      var query = context.GetQueryable<T>().Where(predicate);
      return query.GetResults();
    }
  }

  public virtual IEnumerable<T> GetResultItems<T>(Expression<Func<T, bool>> predicate, string index) where T : SearchResultItem
  {
    var results = GetResults(predicate, index);
    foreach (var hit in results.Hits)
    {
      yield return hit.Document;
    }
  }
}

Step 2: create a custom QueryableCollection

This is the part where the code from Vivian Roberts is used. We did add the item count to make the result count work and adapted it to work with Sitecore 9.1. The QueryableCollection will be used in the test version of the search context.
public class SearchProviderQueryableCollection<TElement> : GenericQueryable<TElement,Query>, IOrderedQueryable<TElement>, IQueryProvider
{
  private readonly EnumerableQuery<TElement> innerQueryable;

  public SearchProviderQueryableCollection(IEnumerable<TElement> enumerable):base(null,null,null,null,null,null,null)
  {
    innerQueryable = new EnumerableQuery<TElement>(enumerable);
  }

  public SearchProviderQueryableCollection(Expression expression) : base(null, null, null, null, null, null, null)
  {
    innerQueryable = new EnumerableQuery<TElement>(expression);
  }

  public new Type ElementType => ((IQueryable)innerQueryable).ElementType;
  public new Expression Expression => ((IQueryable)innerQueryable).Expression;
  public new IQueryProvider Provider => this;

  public new IEnumerator<TElement> GetEnumerator()
  {
    return ((IEnumerable<TElement>)innerQueryable).GetEnumerator();
  }

  IEnumerator IEnumerable.GetEnumerator()
  {
    return GetEnumerator();
  }

  public IQueryable CreateQuery(Expression expression)
  {
    return new SearchProviderQueryableCollection<TElement>((IEnumerable<TElement>)((IQueryProvider)innerQueryable).CreateQuery(expression));
  }

  public new IQueryable<TElement1> CreateQuery<TElement1>(Expression expression)
  {
    return (IQueryable<TElement1>)new SearchProviderQueryableCollection<TElement>((IEnumerable<TElement>)((IQueryProvider)innerQueryable).CreateQuery(expression));
  }

  public object Execute(Expression expression)
  {
    throw new NotImplementedException();
  }

  public new TResult Execute<TResult>(Expression expression)
  {
    var items = this.ToList();
    object results = new SearchResults<TElement>(items.Select(s => new SearchHit<TElement>(0, s)), items.Count);
    return (TResult)results;
  }
}

Step 3: create our custom testable search builder

The testable search builder will be different for each type of tests you need as it will have the actual data to test against. Important is however the creation the context itself. Note that I am using Moq as mocking framework - you can use another if you want. An example could be:
public class TestableSearchBuilder : ISearchContextBuilder
{
  private readonly IList<SearchResultItem> items;

  public TestableSearchBuilder()
  {
    var templateId1 = new ID("{04fd3a5b-af21-49e3-9d88-25355301ab91}");
    var root = new ID("{0cbba84d-f2cd-4adb-912e-36d97cb22fe9}");
    var rootPath = new List<ID> { root };
    
    items = new List<SearchResultItem>
    {
      new SearchResultItem { Name = "Item1", Language = "en", ItemId = new ID(new Guid()), TemplateId = templateId1, Paths = rootPath },
      new SearchResultItem { Name = "Item2", Language = "nl", ItemId = new ID(new Guid()), TemplateId = templateId1, Paths = rootPath }
    };
  }

  public IProviderSearchContext GetSearchContext(string index)
  {
    // create the mock context
    var searchContext = new Mock<IProviderSearchContext>();
    var queryable = new SearchProviderQueryableCollection<SearchResultItem>(items);
    searchContext.Setup(x => x.GetQueryable<SearchResultItem>()).Returns(queryable);
    searchContext.Setup(x => x.GetQueryable<SearchResultItem>(It.IsAny<IExecutionContext>())).Returns(queryable);
    return searchContext.Object;
  }
}
Note that we are using our custom SearchProviderQueryableCollection in the context. The constructor is used to add data to the list of items that are in our fake index. You can put as many items in there as you need with all the available properties you need. For the example I used the SearchResultItem, but if you inherited from that class you can use yours here as well.

Step 4: start testing

We have everything in place now to start writing actual tests. I'm using Xunit for the tests as this one works fine with FakeDB, which I am sometimes using in combination with this code.

In the constructor of the test class (or setup method if not using Xunit) you need to create an instance of the TestableSearchBuilder. You need to get this instance in the class you are testing - see step 1 to make the code testable. So create an instance of the testable class (e.g. the SearchRepository) with the TestableSearchBuilder (new SearchRepository(TestableSearchBuilder)). Depending on your actual solution, you might need to tweak the IOC container as well to use these instances when needed in the tests.

And that's all..  you should be able to unit test queries and verify the results. 


Tuesday, April 16, 2019

Sitecore 9 Forms : translating client-side error messages

Sitecore 9 Forms : translating error messages client-side

A while ago a wrote a post on translating error messages in Sitecore Forms. In that post I mentioned that the validation done by jquery on the client-side was out of scope (of that post).

When a question on the subject appeared on Sitecore Stack Exchange, I started investigating. With a little help of Support the outcome was adding some custom scripts (and files). A better solution might be added in a future release but for now (9.1.1 and earlier) we'll have to do it like this. Note that my answer on SSE is a general one - this post addresses fixing it in a SXA environment.

First a small recap:
The messages entered here will be used for client- and server-side error messages. Which is nice. Just be aware that somewhere also jquery validation gets used and that can lead to strange situations. I noticed that the email validation can give two different error messages depending on what the error is.. and one of them is not translated as it comes from jquery. 
This is what  I wrote on the translated error messages in my first post. So let's start translating that email validation!

The issue can be easily reproduced with a translated form (non-English). Start typing in an email field and you will get the English jquery message. Once you have something like "...@x"  the jquery message will disappear and the Sitecore one will appear (which can be translated).

Adding the localized scripts

  1. Download all the jquery-validation localized messages files you need from https://github.com/jquery-validation/jquery-validation/tree/master/src/localization
  2. Place those files in a \sitecore modules\Web\ExperienceForms\scripts\localization folder. 
  3. You can change the translation in the files as needed - their structure is very straight forward
  4. Open Sitecore Form Wrapper.cshtml in the \Views\Shared folder 
  5. Add a script just after the @Html.RenderFormScripts():  (adapt the script for the languages you have in your environment)
<script type="text/javascript">
var lang = '@Sitecore.Context.Language.Name';
var script = document.createElement("script");
if (lang == 'nl')
{
    script.src = "/sitecore%20modules/Web/ExperienceForms/scripts/localization/messages_nl.js";
    $("head").append(script);
}
else if (lang == 'fr')
{
    script.src = "/sitecore%20modules/Web/ExperienceForms/scripts/localization/messages_fr.js";
    $("head").append(script);
}
...
</script>

Don't forget to add the added/changed files to your solution. And be aware when upgrading that you changed a Sitecore file...

This should do it. The messages are not translatable by the editors, but at least they are not English everywhere...

Alternatives?

It will also work if you put the messages javascript files in the scripts folder of Forms (\sitecore modules\Web\ExperienceForms\scripts) and add the name of the messages file to the Scripts field in the form (where the other jquery js files already are). Problem with this solution is that the Scripts field is shared - meaning it is the same for all languages. So, you can add one language and that will be used for all languages (including English). Unless you have a single language site that is non-English, this is not what you want.

Wednesday, April 10, 2019

Sitecore SXA custom rendering variant field for translations

The requirement

We had to display a field (from the datasource item) and some fixed labels in a html structure.

So for a first version we just created the rendering variant and added Field and Text variants to the definition. This works..  but in our multilingual environment we have to translate the labels. This can be done as the values are not shared, but I started thinking.. 
do I want content editors that translate labels changing my rendering variants?
And the answer was no..

To fix that, I created my own variant definition to support translated labels.

 Creating a custom variant field

Template

First thing to do is create a template to define the fields of the variant. In our case that was very easy as I just wanted to create an enhanced version of "Text", so I used that template (/sitecore/templates/Foundation/Experience Accelerator/Rendering Variants/VariantText) as base template and added nothing else.

If you have another use case to create a custom variant field, check the existing ones to see what base templates you need. A good start will be: 
  • /sitecore/templates/Foundation/Experience Accelerator/Rendering Variants/Base/_Rendering Variants Base
  • /sitecore/templates/Foundation/Experience Accelerator/Rendering Variants/Base/_Data Attributes
  • /sitecore/templates/Foundation/Experience Accelerator/Rendering Variants/Base/_Link Attributes
  • /sitecore/templates/Foundation/Experience Accelerator/Variants/Base Variant Field
Remember not to place your template in the SXA folders! Use your own folder structure...

Insert Options

You want your field to be available in the insert options when people create a variant definition. The best practices tell us not to change the SXA templates, so we won't do that. We'll use the rules engine to add the insert options!

Go to /sitecore/system/Settings/Rules/Insert Options/Rules and add your insert rule:
You need to check the item template to detect the templates you want to add the insert option to (the variant definition itself and all rendering variant templates in my case), and add your template as insert option.

Corey Smith wrote an excellent blog post on organizing your insert options - a good idea, you should read this ;)

The code

First class we need is a model for your template. It should inherit from Sitecore.XA.Foundation.RenderingVariants.Fields.RenderingVariantFieldBase, but as we are building upon VariantText I inherited from that one (in the same namespace).
public class DictionaryText : VariantText
{
  public DictionaryText(Item variantItem) : base(variantItem)
  {
  }

  public static new string DisplayName => "Dictionary Text";
}

Creating instances of this DictionaryText class is done with a processor class that inherits from Sitecore.XA.Foundation.Variants.Abstractions.Pipelines.ParseVariantFields.ParseVariantFieldProcessor.  In that class you need to define the supported template (the guid of the template you created earlier) and the TranslateField function that translates the variant field arguments to the model class. I checked the VariantText implementation (as we have the same model):
public class ParseDictionaryText : ParseVariantFieldProcessor
{
  public override ID SupportedTemplateId => new ID("{AAD9B54E-C7B2-4193-975E-954C0AD5A922}");

  public override void TranslateField(ParseVariantFieldArgs args)
  {
    var variantFieldArgs = args;
    var variantText = new DictionaryText(args.VariantItem);
    variantText.ItemName = args.VariantItem.Name;
    variantText.Text = args.VariantItem[Sitecore.XA.Foundation.RenderingVariants.Templates.VariantText.Fields.Text];
    variantText.Tag = args.VariantItem.Fields[Sitecore.XA.Foundation.RenderingVariants.Templates.VariantText.Fields.Tag].GetEnumValue();
    variantText.IsLink = args.VariantItem[Sitecore.XA.Foundation.RenderingVariants.Templates.VariantText.Fields.IsLink] == "1";
    variantText.LinkField = args.VariantRootItem[Sitecore.XA.Foundation.Variants.Abstractions.Templates.IVariantDefinition.Fields.LinkField];
    variantText.CssClass = args.VariantItem[Sitecore.XA.Foundation.RenderingVariants.Templates.VariantText.Fields.CssClass];
    variantFieldArgs.TranslatedField = variantText;
  }
}
Note that the ID in this example is the ID of my template - you will probably have another one ;)

Next step is to create the processor that actually renders the variant. Create a class that inherits from Sitecore.XA.Foundation.Variants.Abstractions.Pipelines.RenderVariantField.RenderRenderingVariantFieldProcessor. You need to set a few properties to define the supported template and the render mode (html/json) but the main part is the RenderField function.
In this RenderField you can add any logic you want and render the output. In our case, we stay very close to the code from the ootb VariantText but just translate the value inside:
public class RenderDictionaryText : RenderRenderingVariantFieldProcessor
{
  public override Type SupportedType => typeof(DictionaryText);

  public override RendererMode RendererMode => RendererMode.Html;

  public override void RenderField(RenderVariantFieldArgs args)
  {
    if (!(args.VariantField is DictionaryText variantField))
    {
      return;
    }

    var dictionaryRepository = ServiceLocator.ServiceProvider.GetService<IDictionaryRepository>();
    var control = (Control)new LiteralControl(dictionaryRepository.GetValue(variantField.Text));
    if (variantField.IsLink)
    {
      control = InsertHyperLink(control, args.Item, variantField.LinkAttributes, variantField.LinkField, false, args.HrefOverrideFunc);
    }

    if (!string.IsNullOrWhiteSpace(variantField.Tag))
    {
      var tag = new HtmlGenericControl(variantField.Tag);
      AddClass(tag, variantField.CssClass);
      AddWrapperDataAttributes(variantField, args, tag);
      MoveControl(control, tag);
      control = tag;
    }

    args.ResultControl = control;
    args.Result = RenderControl(args.ResultControl);
  }
}
Note that I am using a custom DictionaryRepository here.. you'll need to change those two lines of code with your own code to translate the text.

Configuration
Last step is adding the processors to the correct pipelines in a config patch - e.g.:
<sitecore>
  <pipelines>
    <parseVariantFields>
      <processor type="Foundation.Dictionary.RenderingVariants.ParseDictionaryText, Foundation.Dictionary" resolve="true"/>
    </parseVariantFields>
    <renderVariantField>
      <processor type="Foundation.Dictionary.RenderingVariants.RenderDictionaryText, Foundation.Dictionary" resolve="true"/>
    </renderVariantField>
  </pipelines>
</sitecore>


Conclusion

My editors are happy. They can change and translate all labels as they are used to and don't need to think about the variants.

My developers are happy because the editors don't mess around in the variants just to translate a label.

And I did not have to spend so much time doing this...  but it seemed worth to share the idea (and the code).

Thursday, March 14, 2019

Event Queue in Sitecore 9.1

Event queue

The event queue is a very useful mechanism in the Sitecore architecture. John Rappel wrote a very nice blogpost on it a while ago (I really do suggest you to read that). But since that day Sitecore came with version 9.1.

In one of our projects we used the event queue to send events to the ContentDelivery servers. The default/system Event Queue was used and it worked fine.  You can find the configuration on these event queues in the <eventing> section in Sitecore.config. The configuration includes a systemDatabase (which is deprecated now) and also a defaultEventQueue.

The configuration and code behind it seem to indicate that Sitecore is slowly refactoring this part and moving towards the <eventqueueprovider>  configuration instead of the <providers>.

Sitecore 9.1

After upgrading to Sitecore 9.1 however, we noticed there is a slight change of plans.. The release notes of 9.1 mention:
The Content Delivery server role does not require the Core database when the Sitecore security membership provider is not used.​​
The documentation on the core database still mentions the ContentDelivery role as one of the roles that refers to the core database, but in this case of the eventQueue you can (should) consider it gone...

Configuration

Let's have a look at the configuration in a Sitecore 9.1:
<eventing defaultProvider="sitecore">
  <providers>
    <clear />
    <!--
      Event provider attributes:
 systemDatabaseName: Deprecated. Name of the database which will hold the system event queue. 
   System event queue receives all the events that are not related to a specific database.
    -->
    <add name="sitecore" type="Sitecore.Eventing.EventProvider, Sitecore.Kernel" systemDatabaseName="core" />
    <add name="sitecore" role:require="ContentDelivery">
   <patch:attribute name="systemDatabaseName">web</patch:attribute>
    </add>
  </providers>
  <!-- EVENT QUEUE -->
  <eventQueueProvider defaultEventQueue="core">
    <eventQueue name="web" type="Sitecore.Data.Eventing.$(database)EventQueue, Sitecore.Kernel">...</eventQueue>
    <eventQueue name="master" role:require="!ContentDelivery" type="Sitecore.Data.Eventing.$(database)EventQueue, Sitecore.Kernel">...</eventQueue>
    <eventQueue name="core" role:require="!ContentDelivery" type="Sitecore.Data.Eventing.$(database)EventQueue, Sitecore.Kernel">...</eventQueue>
  </eventQueueProvider>
  <eventQueueProvider role:require="ContentDelivery">
    <patch:attribute name="defaultEventQueue">web</patch:attribute>
  </eventQueueProvider>
</eventing>

A few things have changed since the previous version here. Note the patches for the ContentDelivery role:

  • systemDatabaseName is set to web instead of core
  • defaultEventQueue is set to web instead of core
  • core and master queues are removed from the eventQueueProvider

As this is only done on the ContentDelivery and not on the other environments, one line in the (still excellent) blog post from John doesn't seem to fit anymore: "Sitecore can no longer assume all web servers in a cluster to have the same database set as the systemDatabase".

Sitecore will handle this nicely (assumption) but in our code we had to make a few small changes.

Coding for a web event queue

Subscribing to the event queue doesn't change as you subscribe to all event queues available with the EventManager.  

Queuing an event however did slightly change as we now push the event to a specific queue. Note that our events are pushed from the ContentManagent role which still has core as the default eventQueue to a ContentDelivery role that has no access to the core queue anymore. So, we need to push to the queue related to the database we want to reach:
var database = Sitecore.Configuration.Factory.GetDatabase(databaseName);
var eventQueue = database.RemoteEvents.EventQueue;
eventQueue.QueueEvent(...);

And we're good to go queue...

Monday, March 11, 2019

Working with external model data in Sitecore SXA variants

SXA and variant definitions

If you are working with SXA, I assume you heard about rendering variants (if not, you're doing it wrong).  There is some documentation on the options you get when creating your own rendering variant: https://doc.sitecore.com/developers/sxa/18/sitecore-experience-accelerator/en/create-a-rendering-variant.html

There is one variant definition option that I would like to highlight here, and that is the "Model".
Model: displays properties from the current Model. The SXA renderings model inherits from Sitecore.XA.Foundation.Mvc.Models.RenderingModelBase. In the Property path field, you can enter the path to the property that you want to display.

 You can use this Model field to display any data you want. I used it to display data from a custom database, but it could also be data from a webservice or any datasource you like. How to do this..

Create your (custom) model

You should start by creating your own component: https://doc.sitecore.com/developers/sxa/18/sitecore-experience-accelerator/en/create-a-new-sxa-module.html  (or copy an existing one and change the data, but that is another topic). What you need is a custom controller, model and probably also a view. Create the necessary items in Sitecore and create your variant definition.

To write the needed code, you can refer to my blog article on custom renderings with variants or to the official docs. The important part here is actually the model class. All (custom) properties you add there will be available in the variants. Fill the data in your repository (called from the controller).
Suppose we have this model:
public class CustomModel: VariantsRenderingModel
{
  public string CustomProperty { get; set; }
  public CustomClass CustomClassProperty { get; set; }
}
and we would like to show the CustomProperty value and a property from the CustomClassProperty as well. Let's create our variant...

Create the variant definition 

In the variant definition that was created for your custom component, you can add a "Model" field.
You can select a tag (just as in other variant fields) and set a css class or other attributes. The Property Path is the really interesting one: here you can define the property from your Model you want to show. As you can see in the screenshots, you just need to add the name of the property as it is in your model class. And you can also get properties from a property..  (like the "Name" property from the CustomClass that was a property of my model).

You will notice that the variant field does not have all properties like a "Field" variant field - e.g. you cannot create a link (which makes sense of course). 

Model data

I am struggling with a few things still..  example: my model has a url and I would like to show that in the variant..  but I can only show the value of the model property - I can't seem to embed this in a "a href..".  I guess there will always be such edge cases that go just that one step further than what is possible ootb. I could go for a quick fix and have my model give me the html for the link..  and if I find a cleaner solution, I'll update the blog :)

Conclusion

Displaying all sorts of external data with variant in SXA was actually easier than expected. Not the first time the product has amazed me in a good way 😃