Last week I was in Madrid to attend the Elasticsearch’s european conference. We had the opportunity to talk to the experts and see the last features of the search engine.
Seems that one of the main focuses for 2019 is machine learning. Since the system has evolved to a log store system, one of the main products will be APM which it will be capable of detecting anomalies in your performance monitoring stats and detect problems in your infrastructure.
Another feature on last versions are frozen indices. With this capabilities we will be able to store more data in less space. This will be specially useful for information that you don’t need to query every day, but you have to keep it just in case. Also, Elastic has relased ElasticSearch SQL which looks very promising.
From the technical point of view, there was a very interesting panel discussion with Matias Cascallares where he explained how does it look like a cluster with more than 100 nodes following the Hot-Warm architecture. With this system, you have some parts of your data in high availability while the older data will be available but at other access speed organizing it in different data-tiers. Another handful pieces of advice: try to keep your shards between 10GB and 40GB of size and do not choose a number of shards that could not be shrank or rolled over i.e. from 2 to 1 or from 2 to 4. But that does not work for 5, 7 or 9.
Right now we are in the process of migration to the last version of Elasticsearch so in the next weeks we will see which of the new features are more handy.
Hope you are all doing well!