I carried forward my Azure PaaS benchmarking work from earlier this month (see this post on the indexing side of the equation for the start of the story).
For a quick refresher, I’ve used an ARM template based deployment of Sitecore to get a system resembling the following:
The element I’m exercising in the benchmarks is how Sitecore’s web servers work with the “Search” icon in the diagram above. I tackled the document ingestion side (how data gets into the search indexes) in my earlier post. This post addresses the querying side of things (how data gets out of the search indexes).
By default, Azure PaaS search with Sitecore is configured to use Azure Search. Solr is another viable option.
Here’s where I’ll interject that Coveo also has an excellent search technology for Sitecore. There are specific use-cases where Coveo is a strong fit, however, and in my indexing the sitecore_core_index evaluations in the earlier post Coveo would not be considered a good fit. This changes, however, for the set of benchmarks I’ve run in this post. I am in the process of testing the Coveo approach in Azure PaaS for Sitecore . . . it’s hot off the presses, so there are still rough edges to work around . . . but Coveo is not part of this write-up for the time being. I will post an update here once I’ve completed the analysis involving Coveo.
In considering Azure Search vs Solr, I used a methodology with JMeter laid out in a great KB article from Sitecore at https://kb.sitecore.net/articles/398589. I have a LaunchSitecore site running and I use JMeter to automate visits to the site, simulating simple user behaviour. I don’t go too crazy with this, because I’m more interested in exercising a basic Sitecore work load than doing a deep-dive in xDB traffic simulation.
My first post showed a clear advantage to Solr for the indexing side of search, but for the querying side I can say there is very little variance between Azure Search and Solr. Sitecore does a good job of protecting data repositories with layers of data and html caches, but even with those those features disabled (we’re talking cacheHtml=”false”on the site definition, <cacheSizes> configuration all set to a heretical zero (“0”), etc) there isn’t a significant difference between the two technologies.
I’m not going to put up a graph of it, because the throughput as measured by JMeter for tests of 20, 50, 100, 200, or more visitors performed almost the same.
I could develop a more search heavy set of benchmarks, performing a random dictionary of searches against a large custom index that Sitecore responds to but must bypass all caches etc, but that feels like overkill for what I’m looking to achieve. Maybe that’s appropriate once I bring Coveo into the benchmarking fun.
For this, I wanted to get a sense for the relative performance between Azure Search and Solr as it relates to Sitecore PaaS and I think I’ve done that. Succinctly:
- Solr is considerably faster at search indexing (courtesy of the search provider implementation in Sitecore)
- both Azure Search and Solr perform about the same when it comes to querying a basic Sitecore site like LaunchSitecore (again, courtesy of the search provider implementation in Sitecore)
This isn’t the definitive take on the topic. It’s more like the beginning. Azure Search is native to Azure, so there are significant advantages there. There is a lot of momentum around Azure and Sitecore in general, so that story will continue to evolve.
There are Solr as a service options out there that make Solr for Sitecore much easier (such as www.measuredsearch.com which I’ll blog about in the next few days), but Solr can be a lot for corporate IT departments to take on, so it isn’t a simple choice for everyone.