Implementing Sitecore EXM with External Data Sources

There are some common pitfalls to avoid when installing Sitecore’s Email Experience Manager (EXM), and a few easy ways to boost performance and work seamlessly with external data sources. I’ve covered everything from basic installation to performance optimization in the whitepaper we’re releasing at Sitecore Symposium today, but below are some tips for working with external data sources in particular, plus a link to some sample code to pay with.

There are a few ways to go about integrating external data sources. I’ll start by going through a few options and then walk through my opinion of the overall best approach. But first, it makes sense to point out that it is important that the contact always gets created in the contact database inside Sitecore, which is stored inside the xDB (MongoDB). If the contact is not in Sitecore’s contact database, then Sitecore analytics will not work properly so you will lose all the analytics data for those contacts. With that, let’s jump into the integration solutions.

Solution One: Build a Sync Tool

The first solution is to build a sync tool that scans the external data source on an interval and adds all the contacts into the contact database. This solution is quite simple really. Sitecore already has scheduled tasks built in, so you can run the task in the background directly from within Sitecore based on whatever schedule you would like. The process that runs would perform the following tasks:

  1. Load all the contacts from the external data source. This should be done in pages if the external data source is very large.
  2. For each of the contacts, try to load the contact from Sitecore’s contact repository. If you find the contact, use that contact as the contact to update; if you do not find the contact, create the contact in the contact repository.
  3. With the contact that you now have, update all the values on the contact. You can import custom values using facets, or use the facets that come built into Sitecore. These facets can be used in the list manager to create segmented lists.
  4. Save the contact back into the contact repository. This triggers it to be saved into xDB and also get added or updated in the Lucene index.

You can run this task as frequently as you need to, but remember that it’s likely to affect performance as it is fairly time consuming to load a large number of contacts from the contact database and save them back to the list. There would also be a huge performance gain from keeping track of the contacts that have changed and only syncing the updated contacts.

Solution Two: Updating the Pipeline

The second option is to update the pipeline that is used to load the contacts for the contact lists in order to write our own code to get the contacts. This method is quite a bit more complicated, but the general process is detailed below.

  1. Extend the List Manager pipeline that is used to load the contacts for a list.
  2. Write a new implementation of the IOrderedQueryable object and return that in the pipeline. It is very important that your new implementation is built to handle performance since Sitecore will load that object many, many times while you are using the list manager and EXM. It should be built to only return the contacts that are requested since loading the entire dataset each time would be very time consuming.
  3. Confirm contacts in the repository. As each contact is being loaded from the external data source, you must also confirm that the contact is in the contact repository in Sitecore in order for analytics to work properly.

Preferred Solution & Why

Of these two options, I generally prefer the first, for the following reasons:

  1. It is a far simpler solution.
  2. It is less likely to break while upgrading Sitecore in the future.
  3. Performance problems are less likely. However, there are still some reasons to go with the second option, including:
    1. You are having an issue with the Lucene / SOLR index and need to have a way to rebuild the contact lists from scratch.
    2. The external data source updates frequently and you can’t wait for the sync process to complete.

Next: learn how to do more with Sitecore with our guides to designing for Sitecore, integrating Sitecore and Salesforce Marketing Cloud, and improving Sitecore’s startup time.