A Sitecore 9 upgrade story

Sitecore 9 upgrade

I was asked to do an upgrade of a Sitecore 8.2 site to the new Sitecore 9. The site includes

Sitecore 9 - © @jammykam

• xDB with custom facets, outcomes, ...
• WFFM with custom save actions (e.g. write data to the custom facets)
• custom indexes (on Lucene)
• extra publishing target
• Sitecore Powershell Extensions
• ...

For the upgrade we decided to use the default upgrade tools as explained in the Sitecore Upgrade guide which can be found on the developer network site.

We did bump into a few issues - some of them our fault, others maybe not.. :)

But I though it might be a good idea to share the experience - the upgrade is not fully finished yet so I might be adding stuff later.

First tip of the day:  stay awake and follow the install guide thoroughly. Apparently during the process we sometimes forgot a (small) step which included in errors later on - easily fixed by doing that step but taking you much longer because you need to figure out what's wrong..

The upgrade package

The analysis of the upgrade package gave us a lot of warnings. Most of them could be ignored, some of them addressed code: our code and the Powershell Extensions. We decided to ignore this as it was a test and we wanted to know if the upgrade would succeed without removing that code. Of course, we had removed all configs that might break stuff (like the ones regarding custom WFFM stuff - as you do need to disable the standard WFFM configs as well). We had no complaints about configs - those were clean :)

The installation of the package went well.. almost. At the very end it gave an error:

"An error occured while copying files. Some files might not have been updated."

Inspection of the logfiles showed us that a file could not be deleted and we had to finish manually - luckily this is was in the post-installation steps so we can assume that the installation/upgrade was done:
ERROR:System.Exception: Could not find a part of the path '...\sitecore\shell\Applications\Social\Wizards'.
ERROR:An error occured while copying files. Some files might not have been updated.
You must manually run the following batch file to complete the installation ...\temp\__Upgrade\Upgrade_20171121T163720573\process.bat
Details: [s]Sitecore.Update.Installer.Exceptions.PostStepInstallerException: Error has occurred during file installation. at Sitecore.Update.Installer.Items.PostStepInstaller.Process(IProcessingContext entry, IProcessingContext context)

We examined the other logs and it appears that this path in the Social folder was already deleted during the installation so why delete it again? Anyway, we ran the batch file and proceeded.

WFFM

Our site includes WFFM so we have to update this as well. This is done after the installation of the upgrade package and before the installation of xConnect. We did make a small mistake though.. The upgrade guide does mention updating your Solr setup prior to updating any modules. But as we didn't have any solr to start with, an assumption was made (or we just weren"t thinking) that we could upgrade wffm and still keep our search provider to Lucene - this is still supported on a standalone instance without xDB so that should work...

But we got: 

System.AggregateException: One or more errors occurred. Sitecore.Update.Installer.Exceptions.CriticalInstallationException: Critical installation exception occurred. System.AggregateException: One or more exceptions occurred while processing the subscribers to the 'packageinstall:items:ended' event

We do have a custom index though.. and the syntax for Lucene indexes changed slightly (so we learned here). This caused our upgrade process to fail miserably. After changing the search provider to Solr (including our custom index configuration) the upgrade worked.

Tip: fix your (custom) indexes when the upgrade guide tells you to "update" Solr

Solr

When switching to Solr, do test your queries. This seems obvious of course. But we noticed that Solr does behave slightly different in some cases compared to Lucene. This is not really an upgrade issue but as more people will be switching to Solr when upgrading to Sitecore 9 this might be worth mentioning. One issue we had was a query that retrieves lots of entries.. worked fine with Lucene and terribly slow with Solr - getting them in batches resolved this.

Installing xConnect

xConnect is new, so it has to be installed rather than upgraded. We had done this before so that would be a piece of cake. The prerequisites on the server(s) were ok and all was set to run SIF (the Sitecore Install Framework). All went fine and the webdeploy was running and.. boom.  The database deployment went wrong because the SQL user we had defined in the json configuration already existed.

Hmz.. indeed. One of the steps during the upgrade is deploying new databases (ExperienceForms and Processing.tasks). And I assumed (yes, assumptions.. it was not in the document) I had to add the SQL user to those databases. Which probably is indeed the case. I first thought that would be the issue, but it's not. That still was a good idea. But apparently we also had to remove the user from the Marketingautomation, ReferenceData and Processing.pools databases.

Tip: make sure your SQL user is not attached to the 3 mentioned databases before installing xConnect

Yes, we are good to go! The webdeploy managed to finish this time.

Well, almost..  this time because I did something really silly (just before holidays people do silly things) trying to install xConnect in my "current" folder. This will cause the installer to fail as it can't access it's own log files anymore as they are "in use". So:

Tip: do not install xConnect in your "current" folder

Starting windows services

We also had an issue starting the windows services installed with xConnect. This means we had to re-run the installation with a custom json, just to perform the tasks after starting the services. The reason the services didn't start are probably related to our environment but I'll mention them in case it helps someone: the user "local service" which is running the services had no access rights to the folders where the jobs are located (..\App_data\jobs\continuous\..). Just add those rights and test by starting the services manually.

After care

We have a running site now. Time to take a look at the logs ;)

Issue 1 : Sql Exceptions

Exception: System.Data.SqlClient.SqlException

Message: Could not obtain information about Windows NT group/user 'DOMAIN\...', error code 0x5.

Source: .Net SqlClient Data Provider

This was another silly one [https://sitecore.stackexchange.com/questions/8986/sql-errors-in-xconnect-logs-after-installation-in-upgrade-scenario]. 

We restored our databases on a SQL 2016 instance before the upgrade so we already matched the required version. After asking Sitecore, this is not necessary - you can upgrade and switch afterwards as well. The issue is not that we switched before, but we restored the databases on the sql server with our windows accounts and that was not ok. We had to change the db owner to a dedicated sql user to fix this.

Issue 2 : Path analyzer errors

Path analyzer errors. Quite a few of those in the logs...  We checked the /sitecore/admin/PathAnalyzer.aspx page and noticed indeed that we had issues - some maps did not get deployed. We checked those in Sitecore, and noticed that our marketer friends for some reason deleted some standard goals. One of them was "Login" and this was now required. Packaging the missing goals from another instance, installing them and deploying the maps made the issues disappear. 

Remember that the upgrade process starts with running scripts that restore deleted Marketing Taxonomies and Marketing Definitions but that does not include any missing goals..

Publishing target

https://sitecore.stackexchange.com/questions/9207/publishing-target-in-sitecore-9/

For our extra publishing target we had to add some (new) configuration as well.

xDB Data Migration

This one is still under investigation... we succesfully ran the migration tool but are facing some aggregation errors in the logs now. And seem to be missing some data...  Will update if applicable ;)

Next steps

• Rewriting our custom facet code, ...  might be another post.
• Try this all over again when update-1 is released :)
• Try this all over again on another project with all our lessons learned...

Sitecore 9 Forms : translating error messages

Sitecore 9 Experience Forms

Sitecore released a successor for Web Forms for Marketers (WFFM) with the new Sitecore 9 version, called Sitecore Forms.

In a multilingual environment you want to translate your forms, which is easy as the form editor itself has a language switch.  But what about the error messages - those things a visitor gets when (s)he fills in the form not as expected...

Translating error messages

Find the validators in the Sitecore Content Editor: /sitecore/system/Settings/Forms/Validations. 

Each validator has a Message field. Create versions of the validator item in all the languages needed and fill (or update) the message field as desired. Just check that the message is valid: the parameters in the text need to match the original value. As in the example (image) for email there is "{0}". A developer will recognize this syntax - all translations do need this token as it will be replaced by the field name when displaying the error.

Client-side messages

The messages entered here will be used for client- and server-side error messages. Which is nice. Just be aware that somewhere also jquery validation gets used and that can lead to strange situations. I noticed that the email validation can give two different error messages depending on what the error is.. and one of them is not translated as it comes from jquery. 

Translating the jquery validation is out of scope here as that is documented elsewhere and is not related to the Sitecore solution. Update: https://ggullentops.blogspot.com/2019/04/sitecore-9-forms-translating-client-error.html ;)

Required field

You might notice that the most used validation - required field - is not in the list. That is because this is a special case.. also on a Forms field, you would not select a required validator, but just tick the required checkbox. 

The code for the required field validation works in a different way and the message will come from the Sitecore dictionary. If you want to update or translate it, you need to add a key to that Dictionary. This can be done in the master database - just add an item in the /system/Dictionary - the name doesn't really matter. The key however needs to be exactly "{0} is required.". The phrase can be anything you want, just remember to add the {0} token. 

Create versions and translate in all languages needed - and don't forget to publish.

Update - password/email confirmation fields

As new fields got added in newer versions (9.3) - some new translation requirements were found. AS asked in the comments, the password & email confirmation fields seem to be special cases as well. How to translate them was mentioned on Sitecore StackExchange (https://sitecore.stackexchange.com/q/24129/237): just add "The {0} and {1} do not match." to the dictionary as well.

Sitecore 9 Forms: custom submit action

Sitecore 9 Experience Forms

Sitecore released a successor for Web Forms for Marketers (WFFM) with the new Sitecore 9 version, called Sitecore Forms.

Submit actions - save actions

Sitecore Forms comes with a few submit actions (or save actions as we used to call them, or how they are still called on the submit button) out of the box like Trigger Goal, Redirect to Page, Save Data ... but is also missing a few and developers will probably -just as with wffm- need to make custom ones.

Custom redirect

I wrote a post on Sitecore 8 about customizing the redirect url. In Sitecore 9 with Forms, that article is no longer valid as we should use a different approach and use a custom submit action. So lets try to do that: create a submit action that redirects to a page and adds the value of a field (e.g. email) to the querystring to be processed on the results page.
Could become useful as there is no ootb option for a success message anymore.

The code

Let's go straight to the good stuff. We'll create a class derived from SubmitActionBase<>  and as we want to create redirect functionality we can reuse the ActionData class from the existing redirect so it becomes derived from SubmitActionBase<RedirectActionData>.

public class CustomRedirectAction : SubmitActionBase<RedirectActionData>
{
  public CustomRedirectAction(ISubmitActionData submitActionData) : base(submitActionData)
  {
  }

  protected override bool Execute(RedirectActionData data, FormSubmitContext formSubmitContext)
  {
    Assert.ArgumentNotNull(formSubmitContext, "formSubmitContext");
    if (data == null || !(data.ReferenceId != Guid.Empty))
        return false;
    var item = Sitecore.Context.Database.GetItem(new ID(data.ReferenceId));
    if (item == null)
        return false;

    var email = string.Empty;
    var postedFormData = formSubmitContext.PostedFormData;
    var field = postedFormData.Fields.FirstOrDefault(f => f.Name.Equals("Email"));
    if (field != null)
    {
        var property = field.GetType().GetProperty("Value");
        var postedEmail = property.GetValue(field);
        email= postedEmail.ToStringOrEmpty();
    }
            
    var defaultUrlOptions = LinkManager.GetDefaultUrlOptions();
    defaultUrlOptions.SiteResolving = Settings.Rendering.SiteResolving;
    formSubmitContext.RedirectUrl = new UrlString(LinkManager.GetItemUrl(item, defaultUrlOptions)) + "?email=" + email;
    formSubmitContext.RedirectOnSuccess = true;
    formSubmitContext.Abort();
    return true;
  }
}

We start by fetching the item to redirect to - which is found in the reference id. If this is not available, we get out.

Fetching data from the submit context

Next thing to do is get the value of the email field. That was a bit tricky and I'm not sure whether I used the best approach here - I did get this idea from the Save Data code so it won't be all bad :).

The good news is that we can use the item names (of the "field" item of the form) so we shouldn't have issues with translations. And it will work for other forms if they keep the name identical (the name is not the label shown on the form by the way). Finding the field is easy as they all are a property of the posted form data available in the form submit context. But getting the data from that field was not so trivial - but the "property" stuff works.

Finalizing

As we have all our data now, we van trigger the redirect. Based on the redirect item we can construct the base url - and then add the email to it in the querystring. 

FormSubmitContext

Once the final url is known, we can set it in the form submit context. We also set the redirect property to true and abort the context. Aborting the context will prevent other actions to be executed. Our redirect should be the last.

Configuring Sitecore Forms

Once we have our code, we need to configure Sitecore Forms to use it. The configuration is done in Sitecore.

Open a content editor and go to /sitecore/system/Settings/Forms/Submit Actions. Create a new item of type Submit Action. Fill in the Model Type (e.g. Forms.CustomRedirectAction) and an error message. 

Editor

The editor field will define how your editor can configure your action. In this case we want them to be able to select an item from the content tree. We could use the existing one from the ootb redirect action, but lets create a new one..

Move to the core database and locate /sitecore/client/Applications/FormsBuilder/Components/Layouts/Actions. You will see the existing editor options there (including the RedirectToPage which has a wrong and confusing display name -Trigger Goal- at the time of writing).  As these are all editor actions that will open a content tree at a certain location, the easiest way to accomplish our needs is copy an existing tree and update the items needed - which is actually just the ItemTreeView. The StaticData field defines the root item available for your editors. You might have an issue getting the correct data in here, as the droptree on this field will show you the core database. And you want data from the master database.. switching to raw values and entering the desired guid will do the trick!

Once this is set, we can go back to the master database and finalize our submit action by selecting our newly created editor option.

That's all, your submit action should be available on your forms in the "add" list of any submit button. 

And don't forget to publish ;) 

Sitecore 9 Forms : custom validation

Sitecore 9 Experience Forms

Sitecore released a successor for Web Forms for Marketers (WFFM) with the new Sitecore 9 version, called Sitecore Forms.

Validators

Sitecore Forms comes with a few validators out of the box like email, string length, ... and of course the required field but that one is kinda special.

What if you want to validate fields with your own custom business rules?  

Lets see how we can create a validator that checks the length of the string input (based on parameters) and combines that with a regular expression validation that all characters are numeric.

Custom validator

We start by creating a new class and derive from RegularExpressionValidation. This will give us some functionality concerning the regular expression validation.

We define our class properties that we'll need to setup the validation:

public override string RegularExpression => "^[0-9]+$";
private int MinLength { get; set; }
private int MaxLength { get; set; }

Three functions need to be overridden:

• Initialize: initializes the validator based on the validationModel object
• Validate : server side validation based on the value
• GetClientValidationRules : registers the client side validation rules

Code

The resulting code will look like this:

public class CustomValidation : RegularExpressionValidation
{
  public override string RegularExpression => "^[0-9]+$";

  private int MinLength { get; set; }
  private int MaxLength { get; set; }

  public CustomValidation(ValidationDataModel validationItem) : base(validationItem)
  {
  }

  public override void Initialize(object validationModel)
  {
    base.Initialize(validationModel);
    var stringInputViewModel = validationModel as StringInputViewModel;
    if (stringInputViewModel == null) return;
    MinLength = stringInputViewModel.MinLength;
    MaxLength = stringInputViewModel.MaxLength;
    Title = stringInputViewModel.Title;
  }

  public override ValidationResult Validate(object value)
  {
    if (value == null) return ValidationResult.Success;
 
    var num = value.ToString().Length;
    if (num > 0 && (num < MinLength || num > MaxLength))
    {
      return new ValidationResult(FormatMessage(Title));
    }

    var regex = new Regex(RegularExpression, RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture | RegexOptions.Compiled);
    var input = (string)value;
    if (string.IsNullOrEmpty(input) || regex.IsMatch(input)) return ValidationResult.Success;
    return new ValidationResult(FormatMessage(Title));
  }

  public override IEnumerable<ModelClientValidationRule> GetClientValidationRules()
  {
    if (MaxLength > 0)
        yield return new ModelClientValidationStringLengthRule(FormatMessage(Title), MinLength, MaxLength);

    yield return new ModelClientValidationRegexRule(FormatMessage(Title), new Regex(RegularExpression, RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture | RegexOptions.Compiled).ToString());
  }
}

We compile the code and publish the dll to the website, where we will add the validator to the configuration in Sitecore.

Sitecore Forms configuration

Open the Content Editor and go to the Forms section in Settings: /sitecore/system/Settings/Forms/Validations. Create a new item of type Validation (or copy an existing and update the fields).

You need to fill in the type (e.g. "Forms.CustomValidation,Forms") and a message. The message has to contain "{0}" as this will be replaced with the title of the field in the code - so e.g. {0} is not a valid entry for this field.
Also, do not forget to create a version in every language of your site and translate the message.

Now we need to attach the validator to the field types in order to make it visible when selecting a validator. Go to the Field Types section and let's take the single line text field: /sitecore/system/Settings/Forms/Field Types/Basic/Single-Line Text. Find the Allowed Validations field and your custom validator to the selected ones.

That's it..  if you now go to a form and select a single-line text field, you will be able to select your newly created validator.


ps: don't forget to publish ;)

How to add a custom rendering in all your future Sitecore SXA sites

Sitecore SXA and custom renderings

There are already blogs about how to create custom renderings in Sitecore SXA. The official Sitecore documentation site has an article on how to create your rendering and add it to the toolbox. You can read on my own blog how to do this with variants.

In short, the steps are:

• Create a rendering (controller, action method, repository ...)
• Create a controller rendering in Sitecore
• Add a new rendering to the toolbox

Once you completed these steps you will have a new component in the toollbox for your site. But a Sitecore instance with SXA will probably host multiple sites, so what if you created a rendering that you want to use on all your future SXA sites? I found some information on that and would like to share it, using an existing component as example to show you how to create your own. This will also show that it is possible to add your rendering to the existing component groups in SXA, but that might be not such a good idea considering that upgrades of SXA might remove your additions. And of course, you don't want to mix features.

Adding your rendering to the available components of new SXA sites

First step is to open the Sitecore content editor. I will use the "Page Content" renderings group to show what you can do as this is one with some nice examples.

Branch Template

First thing to do is to create a branch template that will create the item(s) needed to make our component(s) available when a new site is generated. Go to /sitecore/templates/Branches/Feature/Experience Accelerator and you will find already quite some folders with branch templates for the existing components. As we are using Page Content as example, lets check that folder:

The folder contains a few branches, but we'll focus on the Available Page Content Renderings now. This is the branch template that will be used to create the item with the renderings that are included in the "Page Content" rendering group - you'll find all of them in the $name item in the branch. 

To create your own you need to define a new branch template just like it with a proper name. If you just want to add your rendering to an existing group, you can simply add it to the proper branch template and your done - you don't need any further steps in that case but be aware that updates might delete your addition (of course, this is easily checked and re-added if needed).

Site Setup

If you created your own branch to have your own rendering group, go to /sitecore/system/Settings/Feature/Experience Accelerator. You will recognize the folders there for all the default rendering groups and you should create your own here.

First item in the folder is a "Site Setup" item, where we can define the name of the group and whether it is checked by default (or installed by default).

The Site Setup has several children - steps that will be taken during setup. The first ones you can see in the Page Content example are all of type "AddItem" (/sitecore/templates/Foundation/Experience Accelerator/Scaffolding/Actions/Site/AddItem)

AddItem

Let's take the first one as example as this is the most interesting one:

Add Available Renderings

In the AddItem item, you choose a location and a template (and a name). To add the available renderings -what is our main purpose here-, we take the Available Renderings parent as location and use our branch template as Template - the one you created in the first step.

This will tell the SXA site creation script to create a new item based on the branch template at the set location. 

Data items

You can use the same AddItem to create other items as well, like a data folder (including data items if you want). The Page Content has examples on that using other branch templates. 

Rendering variants

If your rendering can make use of variants, you should also create the items for that. The Page Content has examples on empty variant folders (like Field Editor) or variant folders with a default variant (like Page List) where again a branch template is used to create all the necessary items.

Other instances

But what about other instances? Yes, indeed - we have been talking about SXA sites within the same instance here. That covers quite some scenarios, but if you want to have this in other instances with SXA as well you can easily package your stuff and tranfer it - after all, we did nothing more than manipulating items in Sitecore...

Conclusion

Adding your own components is quite powerful but still fairly easy if you know which steps to take. Just adding the renderings can be done with the steps provided here, but I'm sure they are more possibilities to discover by looking at the existing features (e.g. adding to the theme, which you can check in the search feature). And that is the best advice I can give you: take a look at the existing features and you'll discover the fancy world of SXA ;)

Sitecore SXA: PageList component with an Item Query

Sitecore Experience Accelerator (SXA) PageList component & Item Queries

Our requirement was simple: 

Display a set of fields from a list of items, based on a single template under a root item.

We checked the available components in SXA to see if we could do this in a decent way (and without any custom code) and found the PageList component:

Displays lists of pages by predefined or composed queries.

Item Queries as from SXA 1.3

We are using SXA 1.3 and in that version we can define Item Queries in the Settings of the site:

As you can see here we created an Item Query called "Speakers".
We used the "Build query" option on the Query field to create a query to select the desired template at a location. We could extend this with more clauses if needed.

The query build tool looks very familiar to editors who are used to work with the search tools in Sitecore and creates a search index query. You can see the resulting query in the Crawling log file.

Page List

The Page List component has a few options to determine the resulting items. 

Datasource

The "datasource" field also has a "Build query" option. Our first attempt was actually to use this to build our entire query but that failed as we did get multiple results and Sitecore does not like that in the Datasource field. So you have the options to simply select a single item as source or use a query to determine the datasource (but make sure that query does not return more than 1 item or the control will fail). You can combine the item datasource with a query (like the predefined "Children"). 

But in our case we didn't need a datasource as we had set everything on the query - not exactly sure how everything is handled behind the scenes, but as far as I could see they used a combination of the datasource and the query so a single index query seems a better option here.

Source Type

This is the option we did use - by creating our "item query" we added a new option to this dropdown (next to the standard ones):

We select our item query "Speakers" as source type. Save, publish, ..  and there is our list with the exact items we wanted.

Fields - Variants

In order to display the fields we want in the list, we can use Variants. How to create a variant is explained on the Sitecore doc site. In short:

1. Go to /sitecore/content/[tenant folder/tenant/site/...]/Presentation/Rendering Variants/Page List
2. Create a new Variant Definition
3. Add all the VariantFields you want (mark them as link if you want them to link to the underlying item - for a page list you probably want at least one of them to do so)
4. Select your newly created variant in the Styling properties of the control on your page

Credits

Thanks to Dawid (daru) & Adam (adamnaj) for the support on Sitecore Slack - the SXA channel there is a real good place to get help!  Next to Sitecore Stack Exchange of course, already having 58 questions on the SXA tag as we speak (and apparently even one where item queries are covered - d'oh).

Sitecore SXA: using variants to create a list with images and links

A small post with an example on how to tweak SXA to get what you need without any code..

The requirement

We had to create a list of images that had a link on them to an internal or external page.

We couldn't use the default "Link List", as that had no image options..  We couldn't use the "Gallery" as that has no links..

Our solution

Template

We started by creating a new template for datasource items, inheriting from the existing "Link" template (/sitecore/templates/Feature/Experience Accelerator/Navigation/Datasource/Link) in the Navigation feature.

We called it ImageLink and added one field: Image (placed it in the Link section so it would be combined with the existing Link field on our base -Link- template. 

Variant definition

For the Link List component, we added a new Variant definition.

We called it "Imaged" and added just one VariantField for the image with properties:

• Field name: "Image"
• Is link: checked 

In order to get the link using the link field as source, we filled in the name of the field containing the link ("Link") in the field "Field used as link target" on the variant definition itself.

Don't forget to select the allowed templates in the variant definition ;)

Link List variant

Now we can add a Link List on our page and select the "Imaged" variant. And that is it actually.. save, publish, ...  and magic!

Here is our output:

A list with links and images - just what we needed. We can also use the (optional) title from the Link List and other inherited features.

And we didn't write any code. 

Scary...