The Solr *Optimize Now* Button for Sitecore Use Cases

If you’ve worked with Sitecore and Solr, you’re no stranger to the Solr Admin UI.  There are great aspects to that UI, and some exciting extension points with Sitecore implications too, but I know one element of that Solr UI that causes some head-scratching . . . the “optimize now” feature:

OptimizeNow.JPG

 

The inclusion of the badcauses people to think “something is wrong . . . red bad . . . must click now!”

What is this Optimize?

I’m writing this for the benefit of Sitecore developers who may not be coming at this from a deep search background: do not worry if your Solr cores show this badicon encouraging you to optimize now.  Fight that instinct.  For standard Sitecore Solr cores that are frequently updating, such as the sitecore_core_index, sitecore_master_index, sitecore_analytics_index, and — depending on your publishing strategy — sitecore_web_index, one may notice these cores almost always appear with this “optimize now” button in the Solr Admin UI.  Other indexes, too, may be heavily in use depending on how your Sitecore implementation is structured.  If you choose the optimize now option and then reload the screen, you’ll see the friendly green check mark next to Optimized and you should notice the Segment Count drops to a value of 1:

segmentcount1.JPG

Segments are Lucene’s (and therefore Solr’s) file system unit.  On disk, “segments” are where data is durably stored and organized for Lucene.  In Solr version 5 and newer, one can visualize Segment details for each Solr Core via the Solr Admin UI Segments Info screen.  This shows 2 Segments:

segmentinfo

If your Segment count is greater than 1, the Solr Admin UI will report that your Solr Core is in need of Optimization (with that somewhat alarmingbadicon).  The Optimize operation re-organizes all the Segments in a Core down to just one single Segment . . . and for busy Sitecore indexes this is not something to do very often (or at all!).

To track an optimize operation through at the file system level, consider this snapshot of the /data/index directory for a sitecore_master_index before performing optimization; note the quantity of files:optimizebefore

After the optimization, consider the same file system:

optimizeafter

When in doubt, don’t optimize

Solr’s optimize now command is like cleaning up a house after a party.  It reduces the clutter and consolidates the representation of the Solr Core on disk to a minimal footprint.  The problem, is, however, optimizing takes longer the larger the index is — so the act of optimizing may produce very non-optimal performance while it’s doing the work.  Solr has to read a copy of the entire index and restructure the copy into a single Segment.  This can be slow.  Caches must be re-populated after an optimization, too, compounding the perf impact.  To continue the analogy of the optimize now being like cleaning after a party, imagine cleaning up during a party; maybe you pause the music and ask everyone to leave the house for 20 minutes while you straighten everything up.  Then everyone returns and the partying resumes, with the cleaning being a mostly useless delay.

To draw from the official Solr documentation at https://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations:

“Optimizing is very expensive, and if the index is constantly changing, the slight performance boost will not last long. The trade-off is not often worth it for a non static index.”

For those Sitecore indexes in Solr that are decidedly non-static, then, ignore that “optimize now” feature of the Solr Admin UI.  It’s better to pay attention to Solr “Merge Policies” for a rules based approach to maintaining Segments; this is a huge topic, one left for another time.

When to consider optimizing

Knowing more about the optimization process in Solr, then, we can think about when it may be appropriate to apply the optimize command.  For external data one is pulling into Solr, for example, a routine optimization could make sense.  If you have a weekly product data load, for instance, where 10,000 items are regularly loaded into a Solr Core and then they remain un-changed, optimization after the load completes makes a lot of sense.  That data in the Core is not dynamic.  When the data load completes, you could include an API call to Solr that triggers the optimize.

An API call to trigger an optimize in Solr is available through an update handler call : http://solr-server:8983/solr/sitecore_product_catalog/update?stream.body=<optimize><query>*:*</query></optimize&gt;

Sitecore search has a very checkered past with the Lucene Optimize operation.  I’ve worked on big projects that were crippled by too frequent optimizing work like that discussed in Uli Weltersbach’s post.  We ended up customizing the Optimize methods to be no-op statements, or another variation like that.  For additional validation, check out the Lucene docs on the optimize method:

“This method has been deprecated, as it is horribly inefficient and very rarely justified.”

Since Solr sits on top of Lucene, the heritage of Lucene’s optimize is still relevant and — in the Solr Admin UI — we see a potential performance bottleneck button ripe for clicking . . . fight that instinct!

now

 

Solr Configuration for Integration with Sitecore

I’ve got a few good Solr and Sitecore blogs around 75% finished, but I’ve been too busy lately to focus on finishing them.  In the meantime, I figure a picture can be worth 1,000 words sometimes so let me post this visual representation of Solr strategies for Sitecore integrations.  One Solr core per index is certainly the best practice for production Sitecore implementations, but now that Solr support has significantly matured at Sitecore a one Solr core for all the Sitecore indexes is a viable, if limited, option:

draft

There used to be a bug (or two?) that made this single Solr core for every Sitecore index unstable, but that’s been corrected for some time now.

More to follow!

Sitecore and Solr Diagnostics: Pings and Logs

Like I mentioned last week in this post about Sitecore + Solr – Lucene, I wanted to share some other posts about running Sitecore with Solr.  While 2017 is shaping up to be the year of Azure for Sitecore, I also expect Solr to continue to grow in prominence across Sitecore implementations.

Solr exposes lots of information about it’s current condition and state, but it’s not that easy to access.  I’ll take a run at shedding light on some of the monitoring basics and let’s see where this goes:

Ping

A proof-of-life for Solr is usually part of an implementation, whether it’s for a load-balancer in front of Solr nodes or some other form of a health check.  This is typically done via the Solr /admin/ping endpoint so a request to http://localhost:8983/solr/sitecore_master_index/admin/ping?wt=json would result in something like:

{
  "responseHeader": {
    "status": 0,
    "QTime": 0,
    "params": {
      "q": "{!lucene}*:*",
      "distrib": "false",
      "df": "text",
      "rows": "10",
      "wt": "json",
      "echoParams": "all"
    }
  },
  "status": "OK"
}

I chose the JSON response writer for Solr (more about this at https://wiki.apache.org/solr/CoreQueryParameters).  Solr’s default response writer is XML and returns a lot more formatting text around the exact same information.

There is actually a lot of flexibility around this ping RequestHandler for Solr (check out the documentation).  I’m scratching the surface here since, for a plain vanilla Sitecore install, knowing that ping will either be “OK” or return the failing HTTP error code is usually sufficient.

Logging

So the ping is easy, but only reports OK or NOT OK.  It doesn’t provide any real granularity around that determination . . . so let’s assume you’ve got a Solr server that is not returning an “OK” status from the ping.  If you’ve restarted Solr on the server, and Solr still isn’t responding “OK” then it’s probably time to dig into the Solr logs.  Solr defaults to using a variation of Log4j to capture trace logs, so the documentation on Log4j can be helpful (just note that currently Log4j is on version 2.x, while the Solr versions I’ve worked on [up to Solr 6.2] are all using the earlier Log4j 1.2 version [log4j-1.2.17.jar]).

Solr will use the file server\resources\log4j.properties as the basis for Log4j configuration, but this can vary depending on how you’ve installed Solr.   If you need to determine where this file is configured in your implementation, you can check the Dashboard of the Solr web admin tool.  This shows the runtime configuration settings for Solr.  I’ve highlighted the two log related settings in my example:

solrlogsettings

The -Xloggc parameter is for Java Garbage Collection (GC) logging and usually not something you would consider Solr specific, but good to know it exists if that becomes a topic to investigate.

Inside a standard log4j.properties file one finds the following:

#  Logging level
solr.log=logs
log4j.rootLogger=INFO, file

log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender

log4j.appender.CONSOLE.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%-4r %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n

#- size rotation with log cleanup.
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.MaxFileSize=50MB
log4j.appender.file.MaxBackupIndex=9

#- File to log to and log format
log4j.appender.file.File=${solr.log}/solr.log
log4j.appender.file.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m\n

log4j.logger.org.apache.zookeeper=WARN
log4j.logger.org.apache.hadoop=WARN

# set to INFO to enable infostream log messages
log4j.logger.org.apache.solr.update.LoggingInfoStream=OFF

That configuration file is fairly self-explanatory.  Adjust the rootLogger value off of INFO if you’d prefer DEBUG or WARN or some other level.  I think I have just one other note on the above configuration file:

the MaxFileSize setting of older Solr installations was set to 4 MB; this is pretty small for a production instance, so the example above (using a Solr 6 default) of using 50 MB makes more sense for production.

If one peeks into a solr log file, there’s a couple common record types by default.  Here’s what a query logged for Solr looks like:

webapp=/solr path=/select params={df=text&distrib=false&fl=_uniqueid&fl=score&shards.purpose=4&start=0&fsv=true&fq=_indexname:(sitecore_testing_index)&shard.url=http://10.225.240.243:8983/solr/sitecore_testing_index_shard2_replica2/|http://10.225.240.72:8983/solr/sitecore_testing_index_shard2_replica1/&rows=2147483647&version=2&q=__is_running_b:(True)&NOW=1481640759343&isShard=true&wt=javabin} hits=0 status=0 QTime=0

After the query parameters, the log reports the number of hits for the query, the status (0 means “OK”) and the duration for this request in milliseconds.  Note that Solr queries are logged after they complete . . . so if you have a run-away query you may not see it in the log with the standard configuration.  More about this at a later time, perhaps.

Updates to Solr are logged using a similar pattern, they will start with webapp=/solr path=/update.

It’s possible to  setup Log4j to create more narrowly targeted logs.  For example, you could have queries and updates written to different logs for easier troubleshooting (see this example for the nitty gritty on targeting Solr logs).

If your project doesn’t want all this “noise” about each update and query to Solr, or there’s a concern that all this logging could become a bottleneck, you can set the log level for Solr to WARN and only see warnings and errors in your logs.  You will be sacrificing that query performance data, however, so you’ll not have a record to review for slow performing queries . . . so a recommended practice is to trigger a “WARN” condition when a query exceeds a certain threshold.  In solrconfig.xml add a <slowQueryThresholdMillis> element to the query section of solrconfig.xml:

<query>
<slowQueryThresholdMillis>2000</slowQueryThresholdMillis>

Any queries that take longer than 2 seconds (2000 milliseconds defined above) will be logged as “slow” queries at the WARN level.  Note that this setting would need to be duplicated for every core that’s defined in Solr (meaning, with Sitecore’s one core per index  approach you’ll have several solrconfig.xml files to touch).

Besides these fairly routine entries in Solr logs, there will also be exceptions and sometimes using the Logging section of the Solr web admin provides a convenient way to see the most recent set of these exceptions.  This is a convenient way of viewing Log4j items for your Solr implementation in near real-time; note if you make changes to the Level setting from this web interface, it is not persistent after a Solr restart.

Example of a Solr logging section in the web admin interface:

loggingsolradmin

There are a number of other ways to review the Log4j data, old-school Chainsaw is still around, some I know some prefer Otros log viewer, but I’ve had success with Splunk (see screen shot below):

splunkforSolr.JPG

I’ll get into other methods for diagnostic work with Solr in the next post.

Sitecore Gets Serious About Search, by Leaving the Game

Sitecore is steadily reducing the use-cases for using Lucene as the search provider for the Sitecore product.  Sitecore published this document on “Solr or Lucene” a few days ago, going so far as to state this one criteria where you must use Solr:

  • You have two or more content delivery servers

Read that bullet point again.  If you’ve worked with Sitecore for a while, this should trigger an alarm.

Sound the alarm: most implementations have 2 or more CD servers!  I’d say 80% or more of Sitecore implementations are using more than a single CD server, in fact.  Carrying this logic forward, 80% of Sitecore implementations should not be running on Lucene!  This is a big departure for Sitecore as a company, who would historically dodge conversations about what is the right technology for a given situation.  I think the Sitecore philosophy was that advocating for one technical approach over another is risky and means a Sitecore endorsement could make them accountable if a technology approach fails for one reason or another.  For as long as I’ve known Sitecore, it’s a software company intent on selling licenses, not dictating how to use the product.  Risk aversion has kept them from getting really down in the weeds with customers.  This tide has been turning, however, with things like Helix and now this more aggressive messaging about the limitations of Lucene.  I think it’s great for Sitecore to be more vocal about best practices, it’s just taken years for them to come around to the idea.

As a bit of a search geek, I want to state for the record that this new Solr over Lucene guidance from Sitecore is not really an indictment of Lucene.  The Apache Lucene project, and it’s cousin the .Net port Lucene.net that Sitecore makes use of out-of-the-box, was ground breaking in many ways.  As a technology, Lucene can handle enormous volumes of documents and performs really well.  Solr is making use of Lucene underneath it all, anyway!  This recent announcement from Sitecore is more acknowledgement that Sitecore’s event plumbing is no substitute for Solr’s CAP-theorem straddling acrobatics.  Sitecore is done trying to roll their own distributed search system.  I think this is Sitecore announcing that they’re tired of patching the EventQueue, troubleshooting search index update strategies for enterprise installations, and giving up on ensuring clients don’t hijack the Sitecore heartbeat thread and block search indexing with some custom boondoggle in the publishing pipeline.  They’re saying: we give up — just use Solr.

Amen.  I’m fine with that.

To be honest, I think this change of heart can also be explained by the predominant role Azure Search plays in the newest Sitecore PaaS offering.  Having an IP address for all the “search stuff” is awful nice whether you’re running in Azure or anywhere else.  It’s clear Sitecore decided they weren’t keen to recreate the search wheel a few years ago, and are steadily converging around these technologies with either huge corporate backing (Azure Search) or a vibrant open source community (Solr).

I should also note, here, that Coveo for Sitecore probably welcomes this opportunity for Sitecore teams to break away from the shackles of the local Lucene index.  I’m not convinced that long-term Coveo can out-run the likes of Azure Search or Solr, but I know today if your focus is quickly ramping up a search heavy site on Sitecore you should certainly weight the pros/cons of Coveo.

With all this said, I want to take my next few posts and dig really deep into Solr for Sitecore and talk about performance monitoring, tuning, and lots of areas that get overlooked when it comes to search with Sitecore and Solr.   So I’ll have more to say on this topic in the next few days!

 

Slice & Dice Solr Indexes for Sitecore with Banana UI

I’m fresh off a week at the Lucene-Solr Revolution conference and training, and it was a real eye-opener for me!  I have many pages of notes, so many potential blog posts bridging the Solr/Sitecore divide, and saw up-close the vibrancy of the Solr community.  There’s so much innovation going on there . . . it was really inspiring.

One take away from the experience was how Solr has a variety of tools for index analysis that could be really useful for Sitecore scenarios.  I know there’s some basic Sitecore diagnostic pages that will shed light on some index internals for Lucene or Solr, and the Solr admin UI has lots of powerful features, but I wanted to share in this blog post a bit about Banana UI and what it can do for us in the Sitecore world.

What Banana UI Can Do

Before explaining how Banana UI works, it’s probably most useful to show a screen shot of it in action:

banana0

The above shows 3 custom dashboard widgets (Banana calls them panels) that are processing against my sitecore_master_index for a LaunchSitecore 8.1 demo instance I have.  I quickly contrived the above 3 visualizations, using just some sample data, but they represent a bit of the possible options using this free tool.

The first dashboard charts parsedcreatedby_s values for the LaunchSitecore site (I set it to exclude the 15,000 items authored by the Sitecore admin using a Solr filter of q=-parsedupdatedby_s:sitecoreadmin).

banana2.JPG

The second dashboard is a basic word cloud derived from title_t in the LaunchSitecore site index.  Banana UI makes it easy to inspect the underlying query for a visualization, so I could find the baseline query executed against Solr for this was q=*%3A*&wt=json&rows=0&fq=__smallcreateddate_tdt:[2011-10-17T19:49:57.000Z%20TO%202016-10-17T19:49:57.000Z]&facet=true&facet.field=title_t&facet.limit=20).

banana3.JPG

The third dashboard shows a heat map of culture_s and menu_title_t.  I wanted to show the heatmap since it’s a great way of visualizing data correlations, and in this case it illustrates how heavily LaunchSitecore relies on the en culture for item content.

banana4

The crux of the Banana UI is time series of data, although it has powerful applications for other data too.  For my example dashboards, I setup a time window spanning 5 years so it included all the origin data for LaunchSitecore (has it really been 5 years since that project got off the ground?).  Like I said above, I wanted to exclude the 15,000+ pieces of sitecore\admin content since it would dominate the chart in dashboard 1.  With Banana UI, it’s all driven through the single page application:

banana1.JPG

I’m just getting started with Banana as a tool for examining customer Solr implementations for Sitecore . . . I can envision a lot of interesting angles to this, whether using the sitecore_analytics_index for xDB visualizations or metadata analysis with canned searches of the sitecore_web_index.  There’s probably a use case for everyone: business users wanting to keep tabs on a few key terms in the corpus, super-users doing site-wide analysis, certainly for technical users curious about term frequencies or looking to test out some Solr faceting with zero coding and a nice visual interface for feedback.  Banana UI belongs on the radar of every Sitecore implementation making use of Solr.

How To Put Banana UI To Use

Quoting from the Lucidwork’s GitHub repo for Banana:

The Banana project was forked from Kibana, and works with all kinds of time series (and non-time series) data stored in Apache Solr. It uses Kibana’s powerful dashboard configuration capabilities, ports key panels to work with Solr, and provides significant additional capabilities, including new panels that leverage D3.js.

The goal is to create a rich and flexible UI, enabling users to rapidly develop end-to-end applications that leverage the power of Apache Solr.

Here’s a key reference on how to generally configure Banana for Solr.

I’m not going to repeat the reference above, but the gist of it is to copy the Banana UI source into your solr-webapp directory and customize a couple settings (like the name of the Solr collection you want to analyze), then you’re ready to begin slicing and dicing the Solr information:

banana

One can access the Banana UI  via http://your solr server:your solr port/solr/banana/ (or solr/banana/src/index.html if the redirects aren’t setup).  I found Chrome to be the best browser for visualizing the data.  You can save pre-built dashboards for ease of sharing, or build dashboards out ad-hoc.

While Banana (and the original, Kibana) is designed for viewing event and log data in real-time, it’s charting components work great as a facade in front of Solr and it deploys as a simple stand-alone directory as part of the Solr web app.  I’m looking forward to seeing how I can further shape Sitecore specific dashboards with Banana, as currently I’m really just scratching the surface.

The Sitecore Pie: strategic slicing for better implementations

The Sitecore Pie

pie
At some point I want to catalog the full set of features a Sitecore Content Delivery and Content Management server can manage in a project.  My goal would be to identify all the elements that can be split apart into independent services.  This post is not a comprehensive list of those features, but serves to introduce the concept.

Think of Sitecore as a big blueberry pie that can be sliced into constituent parts.  Some Sitecore sites can really benefit from slicing the pie into small pieces and letting dedicated servers or services manage that specific aspect of the pie.  Too often, companies don’t strategize around how much different work their Sitecore solution is doing.

An example will help communicate my point: consider IIS and how it serves as the execution context for Sitecore.   Many implementations will run logic for the following through the same IIS server that is handling the Sitecore request for rendering a web page.  These are all slices of the Sitecore pie for a Sitecore Content Delivery server:

  1. URL redirection through Sitecore
  2. Securing HTTP traffic with SSL
  3. Image resizing for requests using low-bandwidth or alternative devices
  4. Serving static assets like CSS, JS, graphics, etc
  5. Search indexing and query processing (if one is using Lucene)

If you wanted to cast a broader net, you could include HTTP Session state for when InProc mode is chosen, Geo-IP look-ups for certain CD servers, and others to this list of pie slices.  Remember, I didn’t claim this was an exhaustive list.  The point is: IIS is enlisted in all this other work besides processing content into HTML output for Sitecore website visitors.

Given our specific pie slices above, one could employ the following alternatives to relieve IIS of the processing:

  1. URL Redirection at the load balancer level can be more performant than having Sitecore process redirects
  2. Apply SSL between the load balancer and the public internet, but not between the IIS nodes behind your load balancer — caled “SSL Offloading” or “SSL Termination”
  3. There are services like Akamai that fold in dynamic image processing as part of their suite of products
  4. Serving static assets from a CDN is common practice for Sitecore
  5. Coveo for Sitecore is an alternative search provider that can take a lot of customer-facing search aspects and shift it to dedicated search servers or even Coveo’s Cloud.  One can go even further with Solr for Sitecore or still other search tiers if you’re really adventurous

My point is, just like how we hear a lot this election season about “let Candidate X be Candidate X” — we can let Sitecore be Sitecore and allow it to focus on rendering content created and edited by content authors and presenting it as HTML responses.  That’s what Sitecore is extremely valuable for.

Enter the Cache

I’ve engaged with a lot of Sitecore implementations who were examining their Sitecore pie and determining what slices belong where . . . and frequently we’d make the observation that the caching layer of Sitecore was tightly coupled with the rest of the Sitecore system and caching wasn’t a good candidate for slicing off.  There wasn’t a slick provider model for Sitecore caches, and while certain caches could be partially moved to other servers, it wasn’t clean, complete, or convenient.

That all changed officially with the initial release of Sitecore 8.2 last month.  Now there is a Sitecore.Caching.DefaultCacheManager class, a Sitecore.Caching.ICache interface, and other key extension points as part of the standard Sitecore API.  One can genuinely add the Sitecore cache to the list of pie slices one can consider for off-loading.

In my next post, I will explore using this new API to use Redis as the cache provider for Sitecore instead of the standard in-memory IIS cache.

Sitecore RemoteRebuild Strategy Best Practices

I spent entirely too long troubleshooting a customer’s approach to the RemoteRebuild indexing strategy for Sitecore.  The official documentation is fairly straight-forward, but there are some significant gaps left up to the reader to infer or figure out.

I think the heading “Best Practice” on that documentation page is great, and I hope Sitecore continues to expand those notes to be as thorough as possible.  That being said, I would include the following example patch configuration showing how to apply the change without manually editing Sitecore base install files:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <contentSearch>
      <configuration type="Sitecore.ContentSearch.ContentSearchConfiguration, Sitecore.ContentSearch">
        <indexes hint="list:AddIndex">
           <index id="sitecore_web_index" type="Sitecore.ContentSearch.LuceneProvider.LuceneIndex, Sitecore.ContentSearch.LuceneProvider">
            <strategies hint="list:AddStrategy">
              <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/remoteRebuild" />
            </strategies>
          </index>
        </indexes>
      </configuration>
    </contentSearch>
  </sitecore>
</configuration>

This patch should be applied on the Sitecore CD servers where you want to perform the Lucene re-indexing operations.  There are no configuration changes necessary to the Sitecore CM servers.

Speaking of CM server, one might think the posted Sitecore prerequisites cover it:

  • The name of the index on the remote server must be identical to the name of the index that you forced to rebuild.
  • You must enable the EventQueue.
  • The database you assign for system event queue storage (core by default) must be shared between the Sitecore instance where the rebuild takes place and the other instances.

But the biggest addition to this I found was that the Indexing Manager feature in the Control Panel does not call the proper API to trigger the RemoteRebuild activity, so this screen is not where one initiates the remote rebuild:

Won’t work:

remoterebuild

The only way to properly activate the RemoteRebuild is via the Developer ribbon in the Sitecore Content Editor

This works:

remoterebuild2

See this quick video on how to enable this Developer area in the Content Editor, in case this is new to you.

Apparently this dependence on the Developer ribbon is a bug in Sitecore and scheduled to be corrected in a future release.  I know I spent several hours troubleshooting configuration changes and attempted many permutations of the configuration for the RemoteRebuild strategy before Sitecore support confirmed this fact (as did the ever-helpful Ivan Sharamok).

The only other detail I’ll share on the Sitecore RemoteRebuild strategy is that one should verify if the Lucene index adding RemoteRebuild to is using Sitecore.ContentSearch.LuceneProvider.SwitchOnRebuildLuceneIndex or just Sitecore.ContentSearch.LuceneProvider.LuceneIndex.  The patch .config file one uses should reference the appropriate type attribute.

A final note I’ll add about rationale for the RemoteRebuild . . . this customer has several Sitecore Content Delivery servers spread in different data centers, and is currently reliant on the Lucene search provider for Sitecore.  One could make a strong case the customer should be using Solr instead, and indeed this is on their implementation road map, but in the meantime we suggested the RemoteRebuild as a way for them to centrally manage the state of indexes on all their disparate CD servers.  The alternative would be depending on the onPublishEndAsync strategy (which works well, but has limited applications in certain cases), or doing some administrative connection to each CD server (via browser or PowerShell etc) and doing something along the lines of the following when they need to rebuild specific indexes:

public void Page_Load(object sender, EventArgs e)
{
        Server.ScriptTimeout = 3600; //one hour in case this takes a while
        ContentSearchManager.GetIndex("your_index_name").Rebuild();
}

This quickly becomes unwieldy and hard to maintain, however, so getting RemoteRebuild wired up in this case is going to be a valuable addition to the implementation.

Search User Group Talk

I did a talk for the New England Search Technology (NEST) user group yesterday.  Even though the meetings in Boston are a good 90+ minutes away for me, I try to make the trip there a couple times a year since the topics are usually very relevant to what I’m up to with Sitecore.   I offered to do a talk bridging the Sitecore and Search domains, and they took me up on it.  The audience is typically serious Solr and ElasticSearch technologists, some regular committers to those projects, so it was fun to combine those domains with Sitecore’s relative immaturity when it comes to the platform of search.

I don’t want to just post the powerpoint presentations and say “have at it” since the presentations require context to make sense (and it is 2 different powerpoints to sift through).  I’m a non-conventional guy, and I try to avoid powerpoint with bullet points galore and hyper-structured material.  The talk was more a conversation about search and the challenges unique to search with Sitecore (and other CMS platforms that build on Lucene).

My premise relied heavily on Plato’s Allegory of the Cave where I was a “prisoner” experiencing the world of search through the “cave” of Sitecore.  In reality, search is an enormous space with lots of complexity and innovation across the technology . . . but in terms of Sitecore, we experience a filtered reality (the shadows on the cave wall).  This graphic represents a traditional Allegory of the Cave concept:

cave

I’m not going to summarize the entire talk, but I want to state for the record this isn’t a particular criticism of Sitecore — it’s just the nature of working with a product that is built on top of other products.  In Sitecore’s case with Lucene, for example, there’s a .Net port of Lucene (Lucene.Net) and a custom .Net event pipeline in Sitecore used to orchestrate Lucene.Net activities via an execution context of IIS, and so on.

Understanding Search technology with Sitecore is always going to be filtered through the lens of Sitecore.  My talk was addressing that fact, and how Search (with a capital “S”) is a far broader technology than what is put to use in Sitecore.  Understanding the world of Search beyond the confines of Sitecore can be very revealing, and opens up a lot of opportunity for improving a Sitecore implementation.

I should also note that my talk benefited from insightful input from Sitecore-Coveo Jeff, Tim “Sitecore MVP” Braga, and Al Cole from Lucidworks and others.  The host, Monster.com, shared a great meeting space for us and opened the evening with a quick survey of their specific experience with search using both Solr, Elasticsearch, and their own proprietary technology.

While not specific to Sitecore, one link I wanted to share in particular was the talk about Bloomberg’s 3-year journey with Solr/Lucene; it’s a talk from the Berlin Buzzwords conference a couple weeks ago and thoroughly worth watching.  Getting search right can take persistence and smart analysis, regardless of the platform.  With Sitecore, too often implementations assume Search will just work out of the box and not appreciate that it’s a critical set of features worthy of careful consideration.

I’ll have a few follow-up posts covering more of the points I made in my talk; some lend themselves to distinct blog posts instead of turning this into a sprawling re-hash of the entire evening.

Adding Custom Fields to Sitecore Search Indexes in Solr vs Lucene

We’re doing a lot with Solr lately, and I wanted to make a note of the difference in how one defines custom fields to include in your Sitecore indexes. I’m not including how one defines a full custom index, but just the field definition; if one has a “Description” and “Conversation” field to include in a search index for Sitecore, this summarizes the basics of how to make that happen.

Lucene

One adds <field> nodes to the fieldNames and fieldMap sections of the XML configuration.  Again, this example alters the defaultLuceneIndexConfiguration for Sitecore which isn’t a best-practice; it’s generally better to define your own index that is laser-focused just on the content you want to use for your implementation, but I wanted as succinct an example as possible to note the caveat!

 <sitecore>  
   <contentSearch>  
    <indexConfigurations>  
      <defaultLuceneIndexConfiguration type="Sitecore.ContentSearch.LuceneProvider.LuceneIndexConfiguration, Sitecore.ContentSearch.LuceneProvider">  
       <fieldMap type="Sitecore.ContentSearch.FieldMap, Sitecore.ContentSearch">  
         <fieldNames hint="raw:AddFieldByFieldName">  
          <field fieldName="Description" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider">  
            <analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />  
          </field>  
          <field fieldName="Conversation" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider">  
            <analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />  
          </field>  
         </fieldNames>  
       </fieldMap>  
      </defaultLuceneIndexConfiguration>  
    </indexConfigurations>  
   </contentSearch>  
 </sitecore>  

Solr

One adds <fieldReader> nodes to the mapFieldByTypeName and FieldReaders sections of the XML configuration.  Same caveat applies here for Solr as with Lucene — it’s generally recommended to define your own Solr index and work from that, but I wanted a minimal example:

 <sitecore>  
  <contentSearch>    
   <indexConfigurations>  
    <defaultSolrIndexConfiguration type="Sitecore.ContentSearch.SolrProvider.SolrIndexConfiguration, Sitecore.ContentSearch.SolrProvider">  
     <FieldReaders type="Sitecore.ContentSearch.FieldReaders.FieldReaderMap, Sitecore.ContentSearch">  
      <mapFieldByTypeName hint="raw:AddFieldReaderByFieldTypeName">  
       <fieldReader fieldTypeName="Description"  fieldReaderType="Sitecore.ContentSearch.FieldReaders.RichTextFieldReader, Sitecore.ContentSearch" />  
       <fieldReader fieldTypeName="Conversation"  fieldReaderType="Sitecore.ContentSearch.FieldReaders.RichTextFieldReader, Sitecore.ContentSearch" />  
      </mapFieldByTypeName>  
     </FieldReaders>  
    </defaultSolrIndexConfiguration>  
   </indexConfigurations>  
  </contentSearch>  
 </sitecore>  

 

This is really just the tip of the iceburg, as fieldReaderType definitions for Solr and Lucene analyzers — just two examples — open up so many possibilities.  I struggled to find this side-by-side info for Lucene and Solr as it applies to Sitecore (Sitecore 8.1 update-1, specifically), so I wanted to share it here.

Sitecore Config for Publishing to Remote Data Centers

A couple months back, I shared our Rackspace publishing strategy for Sitecore in multiple data centers.  It’s fast, reliable, and we’ve had good experiences taking that approach.

This diagram shows what I’m talking about:2016-02-23-Sitecore-Enterprise-Architecture-For-Global-Publishing

That post at developer.Rackspace.com is only part of the picture, however, and I wanted to return to this topic since I recently had a chance to revisit our Sitecore configuration for a customer running Sitecore 7.2.  Referring to the diagram above, we needed to patch the client Sitecore configuration to accommodate the addition of the proxyWebForHongKong and proxyWebForDallas databases; we were finishing up the provisioning of a clean environment, and had replication running, but we needed to connect the two remote data centers so they fully participated with publishing.

Wading through the Sitecore documentation to find these key architectural pieces is tough, and you would need to synthesize at least the Search Scaling Guide, the Scaling Guide, and the Search and Indexing Guide to get to all this material.  For my own benefit, probably more than for any potential reader, I’m consolidating this here.

It’s important when one does a publish in Sitecore that it triggers index updates on the remote servers, as well as move the content to the appropriate Web database.  This is the crux of what I’m up to here.

At a high level, we had to complete these steps:

  1. Add the database connection strings for the two proxy databases serving our two remote data centers
  2. Add the two proxy database definitions under the <sitecore> node so they can be referenced as Sitecore Publishing Targets etc
  3. Add an index update strategy specific for each proxy database
    • This only needs to be on the environments where the index is used; for example, sitecore_web_index configurations would only be relevant on the CD servers
  4. Connect these new indexing strategies to indexes that need to be kept consistent between data centers
    • This customer is making some use of Lucene, so there is a sitecore_web_index that’s relevant here
    • This customer is using an alternative search provider, too, but I’m focusing on the out-of-the-box configuration as it relates to Lucene here

To implement all of the above, we would typically steer customers to a Sitecore patch configuration strategy such as this one by Bryan “Flash” Furlong.  For simplicity, however, we at Rackspace created a subdirectory App_Config/Include/Rackspace and placed a single config patch file in it.  We don’t own the solution structure, and work to isolate our changes as much as possible from a customer implementation.  So having a single file we can reference is handy for our use case — but not what I’d consider an absolute best practice.

Here’s a nice summary on using subdirectories for Sitecore configurations, in case that’s new to you; here’s Sitecore’s official configuration patch documentation if you’d like to review that: https://sdn.sitecore.net/upload/sitecore6/60/include_file_patching_facilities_sc6orlater-usletter.pdf.

Unfortunately, the <connectionStrings> definitions are not under the Sitecore node and so we can’t patch those the same way, but everything else is in the Sitecore configuration patch below.  I’m not going to detail the following XML, but summarize by saying steps 2 through 4 above are handled here.  Other relevant settings, such as EnableEventQueues etc, are set elsewhere as the Sitecore scaling guide has been broadly applied to this setup already . . . the following is just the changes we needed to make in this particular case, but I’m hoping the example is helpful:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <!-- 
  We cannot patch in a connection string, so be sure to explicitly add connectionstrings
  pointing to the appropriate proxy databases
  <add name="ProxyWebForHongKong" connectionString="Trusted_Connection=True;Data Source=SQLCluster;Database=ProxyWebForHongKong" />
  <add name="ProxyWebForDallas" connectionString="Trusted_Connection=True;Data Source=SQLCluster;Database=ProxyWebForDallas" />

-->
  <sitecore>

    <!-- Add the database configurations that match the connectionstring entries
    (The id of the <database> node must be equal to the name of your connection string node)-->
    <databases>
      <database id="ProxyWebForHongKong" singleInstance="true" type="Sitecore.Data.Database, Sitecore.Kernel">
        <param desc="name">$(id)</param>
        <icon>Network/16x16/earth.png</icon>
        <securityEnabled>true</securityEnabled>
        <dataProviders hint="list:AddDataProvider">
          <dataProvider ref="dataProviders/main" param1="$(id)">
            <disableGroup>publishing</disableGroup>
            <prefetch hint="raw:AddPrefetch">
              <sc.include file="/App_Config/Prefetch/Common.config" />
              <sc.include file="/App_Config/Prefetch/Webdb.config" />
            </prefetch>
          </dataProvider>
        </dataProviders>
        <proxiesEnabled>false</proxiesEnabled>
        <proxyDataProvider ref="proxyDataProviders/main" param1="$(id)" />
        <archives hint="raw:AddArchive">
          <archive name="archive" />
          <archive name="recyclebin" />
        </archives>
        <Engines.HistoryEngine.Storage>
          <obj type="Sitecore.Data.SqlServer.SqlServerHistoryStorage, Sitecore.Kernel">
            <param connectionStringName="$(id)" />
            <EntryLifeTime>30.00:00:00</EntryLifeTime>
          </obj>
        </Engines.HistoryEngine.Storage>
        <Engines.HistoryEngine.SaveDotNetCallStack>false</Engines.HistoryEngine.SaveDotNetCallStack>
        <cacheSizes hint="setting">
          <data>20MB</data>
          <items>10MB</items>
          <paths>500KB</paths>
          <itempaths>10MB</itempaths>
          <standardValues>500KB</standardValues>
        </cacheSizes>
      </database>
      <database id="ProxyWebForDallas" singleInstance="true" type="Sitecore.Data.Database, Sitecore.Kernel">
        <param desc="name">$(id)</param>
        <icon>Network/16x16/earth.png</icon>
        <securityEnabled>true</securityEnabled>
        <dataProviders hint="list:AddDataProvider">
          <dataProvider ref="dataProviders/main" param1="$(id)">
            <disableGroup>publishing</disableGroup>
            <prefetch hint="raw:AddPrefetch">
              <sc.include file="/App_Config/Prefetch/Common.config" />
              <sc.include file="/App_Config/Prefetch/Webdb.config" />
            </prefetch>
          </dataProvider>
        </dataProviders>
        <proxiesEnabled>false</proxiesEnabled>
        <proxyDataProvider ref="proxyDataProviders/main" param1="$(id)" />
        <archives hint="raw:AddArchive">
          <archive name="archive" />
          <archive name="recyclebin" />
        </archives>
        <Engines.HistoryEngine.Storage>
          <obj type="Sitecore.Data.SqlServer.SqlServerHistoryStorage, Sitecore.Kernel">
            <param connectionStringName="$(id)" />
            <EntryLifeTime>30.00:00:00</EntryLifeTime>
          </obj>
        </Engines.HistoryEngine.Storage>
        <Engines.HistoryEngine.SaveDotNetCallStack>false</Engines.HistoryEngine.SaveDotNetCallStack>
        <cacheSizes hint="setting">
          <data>20MB</data>
          <items>10MB</items>
          <paths>500KB</paths>
          <itempaths>10MB</itempaths>
          <standardValues>500KB</standardValues>
        </cacheSizes>
      </database>
    </databases>


    <!-- Add the specific update strategies for each data center-->
    <contentSearch>
      <indexUpdateStrategies>
        <onPublishEndAsyncProxyHongKong type="Sitecore.ContentSearch.Maintenance.Strategies.OnPublishEndAsynchronousStrategy, Sitecore.ContentSearch">
          <param desc="database">ProxyWebForHongKong</param>
          <CheckForThreshold>true</CheckForThreshold>
        </onPublishEndAsyncProxyHongKong>
        <onPublishEndAsyncProxyDallas type="Sitecore.ContentSearch.Maintenance.Strategies.OnPublishEndAsynchronousStrategy, Sitecore.ContentSearch">
          <param desc="database">ProxyWebForDallas</param>
          <CheckForThreshold>true</CheckForThreshold>
        </onPublishEndAsyncProxyDallas>
      </indexUpdateStrategies>




      <!-- Add the additional indexing strategies to the sitecore_web_index, so our additional regions get the messages to update Lucene-->
      <configuration type="Sitecore.ContentSearch.ContentSearchConfiguration, Sitecore.ContentSearch">
        <indexes hint="list:AddIndex">
          <index id="sitecore_web_index" type="Sitecore.ContentSearch.LuceneProvider.LuceneIndex, Sitecore.ContentSearch.LuceneProvider">
            <strategies hint="list:AddStrategy">
              <strategy ref="contentSearch/indexUpdateStrategies/onPublishEndAsyncProxyHongKong" />
              <strategy ref="contentSearch/indexUpdateStrategies/onPublishEndAsyncProxyDallas" />
            </strategies>
          </index>
        </indexes>
      </configuration>
      
      
    </contentSearch>
  </sitecore>
</configuration>