“It works on my machine” has to mean it also works perfectly in production, right? As a developer, we of course always think our app works perfectly. Without real data to back up our own perceptions, the reality could be very different. This blog post is about a change we made that seemed to improve performance, but actually tanked it!
We use Elasticsearch for storing errors & logs data. We receive billions of log messages a month, and we are always working on ways to improve performance in various ways. We decided to disable the default index warmers because they seemed to require a lot of memory and we were trying to optimize memory usage.
After the setting change, it actually felt like running search queries against Elasticseach was faster. There are a ton of different types of elastic queries lumped in the reporting charts below. As a whole, the performance didn’t seem to change much. In a couple specific test queries it felt faster though. We thought we had improved performance, w00t!
In reality, searches were marginally faster at best and we started having some new problems. We started getting some monitor alerts about new incoming data not coming in and other issues that we always monitor for (Read more: How we monitor Elasticsearch). Indexing took a serious turn for the worst as you can see in these charts below. “Average load time” for the indexing spiked, request volume went down, and our satisfaction scoring went to hell.
Our server monitoring also shows that CPU spiked for some reason.
It’s hard to know exactly how Elasticsearch is using memory for various things, but memory overall ended up looking about the same.
When doing any type of performance tuning for an application, you need to have lots of good before and after metrics to compare. You also have to test with real world load in a test environment or by using a load testing tool with a simulated workload. It is really easy to fall for the “works on my machine” syndrome and have a false perception about the performance.
Tools like Stackify APM+ are extremely helpful when it comes to doing performance tuning. We talk to clients all the time that think a certain request in their app is slow and are not surprised when our reporting clearly confirms it for them. Click here to use our APM+ tool with your own data today…for free!
- The 7 Ps of High Performance Cloud Apps - December 23, 2015
- App Performance: Measuring Perception vs Reality - December 22, 2015
- How to Monitor Noisy Cloud Neighbors & Your Web Apps Using Stackify’s Retrace for APM - December 4, 2015
- What Microsoft Azure Got Right for Developers - December 1, 2015
- Stackify Gets PCMag’s Editors’ Choice Award for APM (Application Performance Management) - November 16, 2015