Curious case of the LocationsDictionaryCache

We have some cache monitoring in place for some enterprise Sitecore customers and that system has found one particular cache being evicted roughly 40 times per hour for one particular customer. It’s curious, as this cache isn’t covered in the standard documentation for Sitecore’s caches. Sitecore support had to point me in the right direction on this . . . and it’s an interesting case.

The LocationsDictionaryCache cache is defined in Sitecore.Analytics.Data.Dictionaries.LocationsDictionary and there’s a hard coded 600 second expiration in that object definition:

Expires

There’s also a maxCacheSize set of 0xf4240 hex (1,000,000 or 976 KB). You can’t alter these settings through configuration, you’d have to compile a new DLL to alter these values.

It’s not clear to me that the quick eviction/turn-over of this cache is a perf issue to worry about . . . I think, at this point, it’s working as expected and indicative of a busy site with lots of Sitecore Analytics and Tracking behaviour. Reviewing the decompiled Sitecore code that uses this class (like in Sitecore.Analytics.Tracking.CurrentVisitContext or Sitecore.Analytics.Pipelines.CommitSession.ProcessSubscriptions), it appears that this cache serves as a short-term lookup along the same lines as devices, user-agents, etc. Why this particular cache is active in ways UserAgentDictionaryCache and others is not, besides the obvious 600 second life span, is something we need to dig further into — but I don’t know that it’s a perf bottleneck for our given scenario.

 

How to use Redis in place of Sitecore in-memory caches

In this previous post I discussed the case for separating functions typically combined into a single out-of-the-box Sitecore instance into distinct parts, supporting better scalability, performance, availability, etc.  I concluded that piece with a reference to using Redis cache for Sitecore caches, instead of the default IIS in-memory cache.  Prior to Sitecore 8.2, this wasn’t entirely possible (although one could alter some Sitecore caching aspects through extensive customization of certain Sitecore assemblies).  Implementations looking to free up IIS resources, to get more out of their limited Sitecore IIS licenses, might consider swapping the in-memory caches for Redis.

Major caveat: This specific blog post doesn’t analyze the performance implications of doing this; my focus here is to cover how to accomplish the swap of the in-memory cache instead of examining all the nuances of why or why not to do it.  That being said, I do expect the use of Redis for Sitecore caches to be a surgical decision made based on the particulars of each project.  It’s not a one-size-fits-all solution.  There are also questions of which caches to put into Redis, and where Redis should be running (considerations of connectivity, Redis perf, and the whole landscape of Redis can also be brought into this conversation).  This will make for future material I cover, probably in conjunction with the big time Redis pros at ObjectRocket (who, conveniently, are part of my family at Rackspace).  I hope and expect others in the greater Sitecore community will expand on this topic, too!

Customizing Cache Configuration with Sitecore 8.2 +

Sitecore 8.2 includes a new .config file at App_Config\Include\CacheContainers.config — as the name suggests, this is where one can identify cache containers that will replace the default ones Sitecore uses.  For my example, I’m going to specify a custom container for the website[html] cache; any cache can be included here (I refer you to the old reliable /sitecore/admin/cache.aspx page — the name column enumerates all the Sitecore caches):

cache

With my customization, my CacheContainers.config file looks like this:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
 <sitecore>
  <cacheContainerConfiguration>
   <cacheContainer name="website[html]" type="Rackspace.Sc.Caching.ReferenceRedisCache, Rackspace.Sc.Caching">
    <param desc="name">$(0)</param>
    <param desc="maxSize">$(1)</param>
    <cacheSizeCalculationStrategy type="Sitecore.Caching.DefaultCacheSizeCalculationStrategy, Sitecore.Kernel" />
   </cacheContainer>
  </cacheContainerConfiguration>
 </sitecore>
</configuration>

I’m not going to regurgitate the comments from the default CacheContainers.config file, but it’s certainly worth reviewing.

Once the CacheContainers.config is pointing to the custom class and assembly (in my case type=”Rackspace.Sc.Caching.ReferenceRedisCache, Rackspace.Sc.Caching”) it’s necessary to create the custom assembly to provide the Redis cache behaviour.  The foundation for this is to implement the new Sitecore.Caching.ICache interface (which requires Sitecore.Caching.ICacheInfo and Sitecore.Caching.Generics.ICache<string> interfaces. It can look like a lot of work, but it actually isn’t so bad.

One way I found to make it easier was to reference the out-of-the-box Sitecore Sitecore.Caching.MemoryCacheAdapter in the Sitecore.Kernel assembly.  One can consider the MemoryCacheAdapter for an example of the implementation of these interfaces:

cache2

Reflector, or whatever your preferred decompilation tool of choice, for the win!

In my case, I create a ReferenceRedisCache for this. . .

public class ReferenceRedisCache : Sitecore.Caching.ICache

. . . and the nuts and bolts come down to swapping Redis in place of the System.Runtime.Caching.MemoryCache object from the MemoryCacheAdapter implementation.  There are other considerations, of course, but this is the big picture.

I’m not going to post all the code here as mine is still a bit of a work in progress, and I need to evaluate it in terms of performance etc.  You’re much safer using the decompiled Sitecore source as a guide, at this point.  With this post, I did want to share how one can approach this powerful Sitecore customization; you can selectively change from Sitecore’s in-memory caching to just about any option you’d like this way.

There is another avenue that I’ve not explored with this.  Sitecore provides a Sitecore.Caching.BaseCacheConfiguration class that can be overridden to introduce customizations.  Sitecore.Caching.DefaultCacheConfiguration is the standard approach you could use as an example; this is an alternative way of substituting your own caching logic, but I haven’t dug into it yet.

I’ll clean up my code, do some profiling and evaluations, and come back with the full details of my custom Rackspace.Sc.Caching.ReferenceRedisCache implementation soon.

The Sitecore Pie: strategic slicing for better implementations

The Sitecore Pie

pie
At some point I want to catalog the full set of features a Sitecore Content Delivery and Content Management server can manage in a project.  My goal would be to identify all the elements that can be split apart into independent services.  This post is not a comprehensive list of those features, but serves to introduce the concept.

Think of Sitecore as a big blueberry pie that can be sliced into constituent parts.  Some Sitecore sites can really benefit from slicing the pie into small pieces and letting dedicated servers or services manage that specific aspect of the pie.  Too often, companies don’t strategize around how much different work their Sitecore solution is doing.

An example will help communicate my point: consider IIS and how it serves as the execution context for Sitecore.   Many implementations will run logic for the following through the same IIS server that is handling the Sitecore request for rendering a web page.  These are all slices of the Sitecore pie for a Sitecore Content Delivery server:

  1. URL redirection through Sitecore
  2. Securing HTTP traffic with SSL
  3. Image resizing for requests using low-bandwidth or alternative devices
  4. Serving static assets like CSS, JS, graphics, etc
  5. Search indexing and query processing (if one is using Lucene)

If you wanted to cast a broader net, you could include HTTP Session state for when InProc mode is chosen, Geo-IP look-ups for certain CD servers, and others to this list of pie slices.  Remember, I didn’t claim this was an exhaustive list.  The point is: IIS is enlisted in all this other work besides processing content into HTML output for Sitecore website visitors.

Given our specific pie slices above, one could employ the following alternatives to relieve IIS of the processing:

  1. URL Redirection at the load balancer level can be more performant than having Sitecore process redirects
  2. Apply SSL between the load balancer and the public internet, but not between the IIS nodes behind your load balancer — caled “SSL Offloading” or “SSL Termination”
  3. There are services like Akamai that fold in dynamic image processing as part of their suite of products
  4. Serving static assets from a CDN is common practice for Sitecore
  5. Coveo for Sitecore is an alternative search provider that can take a lot of customer-facing search aspects and shift it to dedicated search servers or even Coveo’s Cloud.  One can go even further with Solr for Sitecore or still other search tiers if you’re really adventurous

My point is, just like how we hear a lot this election season about “let Candidate X be Candidate X” — we can let Sitecore be Sitecore and allow it to focus on rendering content created and edited by content authors and presenting it as HTML responses.  That’s what Sitecore is extremely valuable for.

Enter the Cache

I’ve engaged with a lot of Sitecore implementations who were examining their Sitecore pie and determining what slices belong where . . . and frequently we’d make the observation that the caching layer of Sitecore was tightly coupled with the rest of the Sitecore system and caching wasn’t a good candidate for slicing off.  There wasn’t a slick provider model for Sitecore caches, and while certain caches could be partially moved to other servers, it wasn’t clean, complete, or convenient.

That all changed officially with the initial release of Sitecore 8.2 last month.  Now there is a Sitecore.Caching.DefaultCacheManager class, a Sitecore.Caching.ICache interface, and other key extension points as part of the standard Sitecore API.  One can genuinely add the Sitecore cache to the list of pie slices one can consider for off-loading.

In my next post, I will explore using this new API to use Redis as the cache provider for Sitecore instead of the standard in-memory IIS cache.

Sitecore Data Cache Tuning Crash Course

I’ve been out of the blogging space for a while, due to a number of reasons that aren’t too relevant here . . . suffice it to say that I’ve never left the Sitecore space, just got invested in some very demanding projects.  I’m feeling the itch to write-up some more of my notes about Sitecore, and the fact is it helps me to process, retain, and better understand the subject when I shape it into a few paragraphs on this blog.

In my time at Sitecore, every couple weeks I would encounter an implementation intent on running with their cache setting Caching.DisableCacheSizeLimits configured to true.  It is a fact that the Sitecore documentation suggests this is a good idea (they say so in the Sitecore CMS Tuning Guide) for 64-bit environments with plenty of memory.  Unfortunately, this nugget of guidance can bring about a lot of problems as the “one size fits all” cache tuning logic within Sitecore leaves something to be desired.  I ran into such a situation this week, which prompted me to write this post.

I’ve seen higher CPU usage, publishing errors,  and out of memory exceptions caused by setting Caching.DisableCacheSizeLimits to true.  The cache sizes could regularly exceed the memory of the environment (out-of-memory exceptions!), or a high memory machine allowed so much memory to be set aside for the cache, that it would take a really long time to evict it . . . there is no substitute for a well-tuned Sitecore cache, tailored to the needs of each organization.  This is a case where the quick fix (letting Sitecore manage your cache automatically courtesy of the Caching.DisableCacheSizeLimits setting) can be vastly inferior to a well thought out solution.  The tortoise can beat the hare, in this case.

So, I almost always recommend setting Caching.DisableCacheSizeLimits to false and tuning the Sitecore data caches according to the CMS Tuning guide (https://sdn.sitecore.net/Reference/Sitecore%207/CMS%20Performance%20Tuning%20Guide.aspx).  Those instructions aren’t as detailed as I would like, so let me share some additional notes on how to do this:

    1. Work on a back-up of your real Sitecore databases and use a hardware profile that matches your production environment. . . running against a fraction of the real content will give you incorrect results, as will tuning a cache on a 4 GB machine when your prod servers are 32 GB!
    2. Be sure to disable the configuration setting Query.MaxItems so you’re not artificially limiting how many items the Sitecore API returns
    3. If it’s inconvenient to run a load generator, as the CMS Tuning guide suggests you should do as part of the testing, you can just run the “give me the kitchen sink” XPath query in the old XPath Builder tool.  It’s located at http://%5BServerName%5D/sitecore/shell/default.aspx?xmlcontrol=IDE.XPath.Builder –this used to be a core part of developer life with Sitecore, but now this XPath builder is a vestigial tail that doesn’t get too much use.  It’s very handy for this cache flooding exercise!  Run an XPath query like /sitecore/content//* that should flood your Sitecore caches with the entire content tree.
    4. This video is old, but technically an excellent resource for Sitecore cache tuning: http://mediacontent.sitecore.net/wmv/CacheRecordingWebinar.mp4.
      1. Among other things, the video covers what to prefetch cache, and a methodology for determining settings such as Caching.AverageItemSize using System.GC.GetTotalMemory() method calls.  It’s all about how to calculate the right values for your implementation — and I can’t recommend this video more highly.

With all that said, there is a situation when setting Caching.DisableCacheSizeLimits to true makes sense.  The most common rationale I can think of is when one encounters a production Sitecore system with no changes to Sitecore out-of-the-box in terms of data caching.  I’ve seen this situation due to an implementation team that didn’t know how important this can be for perf, for example.  In one case, I was in this situation in mid November just a few days before Thanksgiving — and the customer needed a quick caching solution that didn’t require careful analysis etc.  In this case, as a temporary measure to improve the data cash landscape for the client, I suggested setting Caching.DisableCacheSizeLimits to true until we could assist them with  a thorough cache tuning effort.  This got their system beyond the very limited data caching  configuration Sitecore comes with out of the box (probably dating back to the Win32 days!).  This wasn’t in place long term, but it was sufficient to improve their performance ahead of a busy online shopping period.