Delivering instant data to a high traffic Sitecore site

The challenge

We had to deliver data to 12 to 15.000 concurrent users on 2 Azure servers running on Sitecore (7).
Most of the data was coming from an external source (restful services) and could (should) not be cached for longer than 1 second because it was real time (sports) scoring information. That data was merged with extra information from Sitecore.

In this post I will describe what we did to achieve this, knowing that we could not "just add some servers". We had an architecture with 2 content delivery servers, 1 content management server and a database server. That could not be changed.

All things described here had some impact, some more than others or on different levels of the application but I hope it might give you some ideas when facing a similar challenge. The solution was build on Sitecore 7 with webforms and webapi.

Coding

Using context classes

Our most visited pages had up to 10 controls showing data (so not counting controls for creating grids and so). All of these controls needed a same set of data fetched from the url, page, ...  The worst solution would be to have all that logic in every control. A better (and actually faster) way could have been to put that logic in a class and use that as base class or call it from all controls.

But in order to prevent all these controls going through that process and possible doing request to back-end services we took another approach and created a "context" class, injected per request (using Autofac in our case). The "context" class was called before any control was initialized and prepared all necessary context data for the page - without having to know what controls are on the page because that would break the whole idea of Sitecore renderings.

Example

We had a "team pages". Each team had several pages with different controls on them (player list, statistics, coach info, pictures ... ) and as it should in a Sitecore environment the editors decide what controls they want on each page. But: on each team page we need to know the team and we could already fetch the global team information as this was one object in the back-end systems. Depending on the control this could cover up to 50% of the data needed, but for each control it was one less request. If you have 10 controls on a page, this matters.. (even if the data would be cached).

Caching

Some of the data from Sitecore could be cached. At least, until it was changed in Sitecore. There are lots of post already out there on how to clear your cache when a Sitecore publish is done, but then you might clear your cache too often. So we created a more granular caching system. Maybe I should write a blog post on this one alone but in a nutshell it comes down to this: each entry in the cache has a delegate that determines whether the cache should be cleared (and maybe even immediately refilled) and after Sitecore publish, for each cache entry the delegate is called. 

This way each cache entry is responsible for clearing itself. In the delegate we can check language, published item, ..  If a lot of Sitecore publishes are happening this mechanism can prevent quite some unneeded cache clearances. Of course, one badly written delegate method could kill your publish performance, but if you keep them simple (try to get out of the method as soon as possible) it's worth it.

Threads

As mentioned before we could not cache the data coming from the external back-end system for any longer than 1 second. The API to the system was build with restful services. Calling the services when we needed the data was not an option if we wanted to serve that many users. Our solution was threading. We created a first long running thread that called the back-end every 5 minutes to see whether there was data to be fetched (this timing was fine as data was starting to show at least a day before the actual live games started). When we detected live data coming in we would start a new thread that fetched the actual data constantly and kept it in memory available for the whole application (until we detected the end of the data stream and let the thread stop). With the constant loop that fetched the data, mapped it to our own business objects (Automapper to the rescue) and sometimes even performed some business logic on it, we were able to keep the "freshness" of the data always under the required 1 second.

So the data was available on our site -in memory- at all time and we the threads for the web application were not harmed as the retrieval itself had it's own threads. A monitoring system was put in place to detect the status of the running threads and we included quite some logging to know what was going on, but in the end it all went well and was stunningly fast.

WebAPI / AngularJS

To deliver the live data on the pages we used AngularJS and WebApi to change the data on the pages without extra page request. For some of the controls this also enabled us to provide at least some content immediately to the users while fetching the rest.
Other tricks like progressive loading of images managed to get the amount of data that is initially loaded by the page down to a minimum.

Tuning

Pagespeed optimizations

This is something you should consider on every site no matter what the traffic will be like. I am talking about bundling and minifying css and javascript, optimizing images (for Sitecore images, use the parameters to define width and/or height), enabling gzip compressing and so on.. Mostly quite easy tasks that can give your site that extra boost.

IIS tuning

Probably a bit less know and for many sites not necessary, but also your IIS webservice can be tuned.

There is a good article by Stuart Brierly on the topic here.

What we did in particular is change the configs to adapt to the number of processors allowing more threads and connections. When doing this you need to make sure off course that your application can handle those extra simultaneous requests. 

We also adapted the number of connections that were allowed to the external webservice (by allowing more parallel connections to the IP-address).

These changes made sure that IIS did not put connections on hold while we still had some resources left on the server.

This looks like a small change, but with quite some impact and you will need to perform some load tests to test your changes.

Infrastructure

Servers

As said in the beginning we could not add more servers, but we did upscale the servers as far as we could.

Caching

The last step to the solution was adding caching servers (Varnish) in front of the solution. This could free the webservers from a lot of request that could easily be cached. Here the use of WebApi to load some data also helped to get quite some requests cached: this can be resources like javascript files or images, but also complete pages. If you can serve these request without them going all the way to your webserver, your server has a smaller request queue and more resources to handle the remaining requests. 

This last step does not come for free, but it had a huge impact once configured properly.

Sitecore wildcard items to the max

Wildcard items

There are enough blogs already describing what a wildcard item in Sitecore is, so a very short intro should do here: 

A Sitecore wildcard item is a regular item in Sitecore with "*" as name. The item has all the fields of a normal item.
The item will match on any url in its tree on its level that is not matched on any of its siblings. Wildcard items can have children.

Some Sitecore versions have a bug when using a display name in the path up to a wildcard item (hotfix available).

The case

We had a case for a sports organisation with a few requirements:

• create a website for a competition that can host a few overall pages and 'subsites' for all 20 events in the competition
• editors do not want to create 20 trees in Sitecore
• all data about the competitions comes live from a backend system and is not copied into Sitecore
• the competition is yearly so the site needs to be ready to be copied next year
• each event has around 120 matches 
• each event can have a mens and/or a womens tournament, each with around 60 competitors

The solution

We used some wildcards..  

The first wildcard is right below the general home page, and defines a sub-homepage for the event. Underneath we have the complete set of items for each event. 

More wildcards

Even more wildcards appear. In the (wo)mens section we have a wildcard to display a team, and underneath another one to display a player. This way we can have pages (seo friendly) for all teams and players in each event.

We also have wildcards for each game (under that wildcard we have subitems for the subpages of a match).

Navigation

The navigation is in 2 parts: first of all we have navigation that displays all events. That is not Sitecore related (data comes from backend system) but will point to the wildcard item under the home item - off course indicating the right event in the url.

The navigation inside an event can be set once. The links are automatically transferred into the correct url for that event (see part about code). Some pages will not have content, but that is handled on the page itself.

Content 

Sitecore content is shown based on datasources - as it should. But here the datasource will not go to a specific item but to an item container. In that container we look for content that can be displayed for the current event. If that is not found we display a message that the content is not yet available.

The code

Events and tournaments have a unique code. That code will be used in the urls on the spot of the main wildcard. We will use the eventcode as much as possible, as soon as a selection is made for the men/women tournament, we use the tournament code. That is because we need the correct code to go to the backend system.

To accomplish this we wrote a context class (instance per request) that detects the "context" - the current event and or tournament - from the url and exposes this to everyone who needs it. 

LinkProvider

The nifty stuff is in our custom LinkProvider. We wrote some code that determines the requested tournamentcode, applying some business rules and based on the current context. In the end, we replace the first wildcard in the url with the determined code (only the first one, as we might have multiple) with string manipulation.

Other wildcards

The other wildcards (team, player, match) are handled in their business objects. Each business class has it's own logic to create the urls and grab the information needed back from the url. As all our code passes through the business objects and nothing is done straight on the base Sitecore items, we are good here. Off course, it's not possible to put a url to a team or so from within the content but that was not a requirement anyway (actually it is possible as a 'external' url).

Components

The event homepage contains several components that need to be activated based on the status of the event (not started, ongoing, finished). This is also done automatically based on the event in our context. All components are placed on the page and the ones that are not compliant will be hidden.
We could have used a custom condition here and use the standard hide functionality to create conditional rendering rules, but we kept it simple and put the logic in a base function for all homepage components.

Conclusion

We had a first version (poc) quite fast but it took us a few days and a lot of testing to get everything working as it should - especially the small and tricky parts like the multilingual aspect, canonical urls, links between tournaments, links from within event components showing data about more than one tournament...

I did not put any of the code in here as it is quite specific based on our business logic and the backend system used, but if there are questions or you want to see some code or config - just ask ;)

Do NOT use Sitecore.Data.Database constructor

How to kill your website - part x

This post could indeed also be called "How to kill your website" because that is what using the constructor of the Sitecore.Data.Database class in Sitecore will do. 

If you set your log level to INFO and see something like this in the log files:

4184 08:30:17 INFO  Cache created: 'web[blobIDs]' (max size: 500KB, running total: 624MB)

4184 08:30:17 INFO  Cache created: 'web[blobIDs]' (max size: 500KB, running total: 624MB)

4184 08:30:17 INFO  Cache created: 'web[blobIDs]' (max size: 500KB, running total: 625MB)

4184 08:30:17 INFO  Cache created: 'web[blobIDs]' (max size: 500KB, running total: 625MB)

4184 08:30:17 INFO  Cache created: 'web[blobIDs]' (max size: 500KB, running total: 626MB)

4184 08:30:17 INFO  Cache created: 'web[blobIDs]' (max size: 500KB, running total: 626MB)

4184 08:30:17 INFO  Cache created: 'web[blobIDs]' (max size: 500KB, running total: 627MB)

This means you have a memory leak...  you're site is dying. Slowly (maybe), but surely.
Look into your code and see if you have ever used new Database(). And then get rid of that code.

How to fetch a database

The wrong way:

var database = new Database("web");

The right way(s):

var database = Sitecore.Data.Database.GetDatabase("web");

or

var database = Sitecore.Configuration.Factory.GetDatabase("web")

Security in ComputedIndexFields

Custom ComputedIndexField

How to create a custom ComputedIndexField in Sitecore can be found on several other location on the net. But what if you want to check the security of your item when populating this field? 

A simple function could check if your item was readable by an anonymous user:

const string anonymous = @"extranet\Anonymous";
var account = Account.FromName(anonymous, AccountType.User);
return AuthorizationManager.IsAllowed(item, AccessRight.ItemRead, account);

This will work in a normal environment, but if you try this during population of your ComputedIndexField you will always get true. Why?

SitecoreItemCrawler

The responsible crawler - Sitecore.ContentSearch.SitecoreItemCrawler - is designed in such a way to wrap the calls for your ComputedIndexField in a SecurityDisabler. So, the true value in our security check is by design.

And maybe for the best.. it seems like a much better approach to perform security checks indexed results are being accessed. And how do I do this?

Security and index results

Well, you don't. Sitecore does this for you. When you query your index, Sitecore has a Security check build in to the pipeline:  <processor type="Sitecore.Pipelines.Search.SecurityResolver, Sitecore.Kernel" /> . This checks the security of the item involved, not of any related items that you might have used to fill your ComputedIndexField. If you also want a security check on those, write a processor to extend the pipeline :)

ComputedIndexField with extra attributes

ComputedIndexField attributes

In the code that populates my ComputedIndexField I needed a reference to a Sitecore item. I needed to know a "context" item from which I had t retrieve data. Apparently you can extend Sitecore to do this (off course).

My configuration look like this (in a separate config file):

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
 <sitecore>
  <contentSearch>
   <configuration type="Sitecore.ContentSearch.LuceneProvider.LuceneSearchConfiguration, 
                                  Sitecore.ContentSearch.LuceneProvider">
    <indexes hint="list:AddIndex">
     <index id="MyIndexName" 
         type="Sitecore.ContentSearch.LuceneProvider.LuceneIndex,Sitecore.ContentSearch.LuceneProvider">
      <configuration type="Demo.Search.LuceneIndexConfiguration, Demo">
       <documentOptions type="Sitecore.ContentSearch.LuceneProvider.LuceneDocumentBuilderOptions">
        <ProcessDependencies>true</ProcessDependencies>
       </documentOptions>
       <fields hint="raw:AddComputedIndexField">
        <field fieldName="DemoContent" label="{1174978C-47CA-405D-13FE-4C808F3A85E7}">
          Demo.Search.DemoField, Demo
        </field>
       </fields>
      </configuration>
     </index>
    </indexes>
   </configuration>
  </contentSearch>
 </sitecore>
</configuration>

 So I added an attribute "label" in my field definition. Now we need a custom IndexConfiguration, and we will start with the one from Sitecore (reverse engineer that one and you will see we only added a few lines).

Custom IndexConfigration

public class LuceneIndexConfiguration : Sitecore.ContentSearch.LuceneProvider.LuceneIndexConfiguration
{
  public override void AddComputedIndexField(XmlNode configNode)
  {
    Assert.ArgumentNotNull(configNode, "configNode");
    string fieldName = XmlUtil.GetAttribute("fieldName", configNode, true);
    string type = XmlUtil.GetValue(configNode);            
    if (string.IsNullOrEmpty(fieldName) || string.IsNullOrEmpty(type))
    {
      throw new InvalidOperationException("Could not parse computed index field entry: " + configNode.OuterXml);
    }
    var field = ReflectionUtil.CreateObject(type) as IComputedIndexField;

    // here starts custom code
    var constructableField = field as IConstructable;
    if (constructableField != null)
    {
     constructableField.Constructed(configNode);
    }
    // end custom code

    if (field != null)
    {
      field.FieldName = fieldName.ToLowerInvariant();
      DocumentOptions.ComputedIndexFields.Add(field);
    }
  }
}

The extra stuff is based on the IConstuctable interface. This allows you to created a Constructed method that can read your xml node. You can use your own interfaces as well surely, but this one is already in the box..

And then we adapt the code for our ComputedIndexField:

IConstructable

public class DemoField : Sitecore.ContentSearch.ComputedFields.IComputedIndexField, IConstructable
{
  private Item MyItem { get; set; }

  public void Constructed(XmlNode configuration)
  {
    var label = XmlUtil.GetAttribute("label", configuration, true);
    if (string.IsNullOrEmpty(label))
    {
      ....
      return;
    }

    Guid id;
    if (Guid.TryParse(label, out id))
    {
      MyItem = Factory.GetDatabase("master").GetItem(new ID(id));
    }
  }

  public object ComputeFieldValue(IIndexable indexable)
  {
    ...
  }
}

We implement the Constructed method and read the xml node. The GetAttribute function is very useful here. We set the MyItem that will be used later on in the ComputedFieldValue function called by the crawler.

Multiple datasources for Associated Content

Associated Content

In the definition of a sublayout in Sitecore you can select a Datasource location and a Datasource template. This makes it easier for your editors to pick the right datasource when using this sublayout. But what if you want to allow more than one datasource template? The answer was found somewhere well hidden on the Sitecore developers site:

Please refer to the following Release note, which describes the officially recommended way to implement such behavior: http://sdn.sitecore.net/Products/Sitecore%20V5/Sitecore%20CMS%206/ReleaseNotes/ChangeLog.aspx
The "Select Associated Content" dialog has been made more extensible. (339396)
Developers can now specify multiple allowed templates by hooking into the <getRenderingDatasource> pipeline and populating the new TemplatesForSelection property.
The default behavior is to use value of the "Datasource Template" field on the rendering item.

How to use multiple datasource templates?

So, all you need to do is create a config file and place it in the include folder (do not alter the web.config):

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
 <sitecore>
   <!-- PIPELINES -->
   <pipelines>
     <getRenderingDatasource>
      <processor patch:after="*[@type='Sitecore.Pipelines.GetRenderingDatasource.GetDatasourceTemplate, 
              Sitecore.Kernel']" type="Demo.SitecorePipelines.GetMultipleDataSourceTemplates, Demo"/>
     </getRenderingDatasource>
   </pipelines>
 </sitecore>
</configuration>

and create your class:

public class GetMultipleDataSourceTemplates
{
    public void Process(GetRenderingDatasourceArgs args)
    {
        Assert.ArgumentNotNull(args, "args");
        if (args.Prototype != null)
        {
            return;
        }

        var data = args.RenderingItem["Datasource Template"];
        if (string.IsNullOrEmpty(data))
        {
            return;
        }

        var templates = data.Split('|');
        foreach (var name in templates)
        {
            var item = args.ContentDatabase.GetItem(name);
            if (item == null)
            {
                continue;
            }
            var template = TemplateManager.GetTemplate(item.ID, args.ContentDatabase);
            if (template != null)
            {
                args.TemplatesForSelection.Add(template);
            }
        }
    }
}

Note that we use "|" as separator here. That is just as an example.
Ok, that's it. Now we can use it like this in Sitecore:

When saving your sublayout item, Sitecore will complain because the field in the datasource template cannot be linked to an item. Don't worry about this. Your editors won't notice and it works.

Template updates

One drawback is that Sitecore is not able to link the template to the field anymore. This means that if you move or rename the template the field will not be updated (as a normal datasource template field will).

Configure the Media Browser default view

Sitecore Media Browser

When you browse for an image from an Image field or Rich Text field in Sitecore, you get to see the Media Browser. In version 6 you would just see a tree view, from version 7 you get a search view as default. The tree view is also there, but you need to switch tabs..  Some people prefer the tree view though and want that as default.

Sitecore 7.0

 In Sitecore 7.0 you can alter the default tab in the Media Browser by editing the \sitecore\shell\Applications\Media\MediaBrowser\MediaBrowser.xml. You will see a Tabstrip with 2 tabs ("MediaLibraryTab" and "MediaTab"). The first tab is the default, so by switching them around you can set the MediaTab ("browse") as default.
The best way to handle your new file is to place it in the "\sitecore\shell\Override" folder.

The result will look like this:

Sitecore 7.1

In Sitecore 7.1 they changed the Media Browser. The search pane even appears to have a bug (the datasource if provided is not taken into account), but that's not our focus here.
The top of the Media Browser now looks like:

Switching the panes can be done with the buttons on the right (above the search field). Setting the default to the tree view however is again possible: in order to use the tree view as a default view you need to change the definition of the Media Browser in the "\App_Config\Include\Sitecore.Speak.config" file like this:

Default config:

<overrideXmlControls>
    <override xmlControl="Sitecore.Shell.Applications.Media.MediaBrowser" 
           with="/sitecore/client/Sitecore/Common/Dialogs/SelectMediaDialog" />
</overrideXmlControls>

Tree view config:

<overrideXmlControls>
    <override xmlControl="Sitecore.Shell.Applications.Media.MediaBrowser" 
            with="/sitecore/client/Sitecore/Common/Dialogs/SelectMediaViaTreeDialog" />
</overrideXmlControls>

The result will look like this:

This view takes your datasource into account (as it should) and displays it as the topfolder of the tree.

Sitecore 8

In Sitecore 8, the "overrideXmlControls" element can be found in "App_config\Include\Sitecore.Speak.Applications.config". The rest should stay the same.

Patching

A good remark was made that it is better to patch this value. It is actually always a good idea to use patches to alter Sitecore configs. The patch should look something like this:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/" >
  <sitecore>
    <overrideXmlControls>
      <override xmlControl="Sitecore.Shell.Applications.Media.MediaBrowser">
   <patch:attribute name="with">/sitecore/client/applications/Dialogs/SelectMediaViaTreeDialog</patch:attribute>
      </override>
    </overrideXmlControls>    
  </sitecore>
</configuration>