Enhancing Effectivity Of Goku Time Collection Database at Pinterest (Half 2) | by Pinterest Engineering | Pinterest Engineering Weblog | Mar, 2024

Monil Mukesh Sanghavi | Software program Engineer, Actual Time Analytics Crew; Xiao Li | Software program Engineer, Actual Time Analytics Crew; Ming-Could Hu | Software program Engineer, Actual Time Analytics Crew; Zhenxiao Luo | Software program Engineer, Actual Time Analytics Crew; Kapil Bajaj | Supervisor, Actual Time Analytics Crew

At Pinterest, one of many pillars of the observability stack gives inner engineering groups (our customers) the chance to watch their providers utilizing metrics knowledge and arrange alerting on it. Goku is our in-house time collection database offering price environment friendly and low latency storage for metrics knowledge. Beneath, Goku just isn’t a single cluster however a set of sub-service elements together with:

• Goku Brief Time period (in-memory storage for the final 24 hours of information, known as GokuS)
• Goku Lengthy Time period (ssd and hdd primarily based storage for older knowledge, known as GokuL)
• Goku Compactor (time collection knowledge aggregation and conversion engine)
• Goku Root (sensible question routing)

You’ll be able to learn extra about these elements within the weblog posts on GokuS Storage, GokuL (long run) storage, and Price Financial savings on Goku, however loads has modified in Goku since these have been written. Now we have carried out a number of options that elevated the effectivity of Goku and improved the person expertise. On this 3 half weblog publish collection, we are going to cowl the effectivity enhancements in 3 main elements:

1. Enhancing restoration time of each GokuS and GokuL (that is the whole time a single host or cluster in Goku takes to return up and begin serving time collection queries)
2. Enhancing question expertise in Goku by reducing latencies of high-priced and excessive cardinality queries
3. Lowering the general price of Goku at Pinterest

We’ll additionally share some learnings and takeaways from utilizing Goku for storing metrics at Pinterest.

This 2nd weblog publish focuses on how Goku time collection queries have been improved. We’ll present a short overview of Goku’s time collection knowledge mannequin, question mannequin, and structure. We’ll comply with up with the advance options we added together with rollup, pre-aggregation, and pagination.

The information mannequin of a time collection in Goku is similar to OpenTSDB’s (which Goku changed) knowledge mannequin. Yow will discover extra particulars here. Right here’s a fast overview of the Goku TimeSeries knowledge mannequin.

A time collection metadata or key consists of the next:

The information a part of a time collection, which we consult with as time collection stream, consists of knowledge factors which are time worth pairs, the place time is in unix time and worth is a numerical worth.

A number of hosts can emit time collection for a singular metric title. For instance: cpu,reminiscence,disk utilization or some software metric. The host-specific info is a part of one of many tags talked about above. For instance: tag- key == host and worth == host title.

A cardinality of a metric (i.e. metric title) is outlined as the whole variety of distinctive timeseries for that metric title. A singular time collection has a singular mixture of tag keys and values. You’ll be able to perceive extra about cardinality here.

For instance, the cardinality of the metric title “proc.stat.cpu” within the above desk is 5, as a result of the mixture of tag worth pairs together with the metric title of every of those 5 timeseries don’t repeat. Equally, the cardinality of the metric title “proc.stat.mem” is 3. Be aware how we symbolize a selected string (be it metric title or tag worth) as a singular shade. That is to indicate {that a} sure tag worth pair will be current in a number of time collection, however the mixture of such strings is what makes a time collection distinctive.

Goku makes use of apache thrift for Question RPC. The question mannequin of Goku is similar to OpenTSDB’s question mannequin specified here. To summarize, a question to Goku Root is just like the request specified under:

Let’s go over the essential choices within the request construction above:

• metricName — metric title with out the tag combos
• record<Filter> — filters on tag values like sample match, wildcard, embody/ exclude tag worth (will be a number of), and many others.
• Aggregator — sum/ max/ min/ p99/ rely/ imply/ median and many others. on the group of timeseries
• Downsample — person specified granularity in time returned in outcomes
• Rollup aggregation/ interval — downsampling at a time collection degree. This selection turns into necessary in lengthy vary queries (you will notice the rationale under in Rollup).
• startTime, endTime — vary of question

The question response appears as follows:

The monitoring and alerting framework at Pinterest (internally known as statsboard) question shopper sends QueryRequest to Goku Root, which forwards it to the leaf clusters (GokuS and/ or GokuL) primarily based on the question time vary and the shards they host. The leaf clusters do the mandatory grouping (filtering), interpolation, aggregation, and downsampling as wanted and reply to the Goku Root with QueryResponse. The Root will once more do the aggregation if vital and reply to the statsboard question shopper with QueryResponse.

Let’s now take a look at how we improved the question expertise.

Goku helps the bottom time granularity of 1 second within the time collection stream. Nevertheless, having such fantastic granularity can affect the question efficiency because of the following causes:

• An excessive amount of knowledge (too many knowledge factors) over the community for a non downsample uncooked question
• Costly computation and therefore cpu price whereas aggregating due to too many knowledge factors
• Time consuming knowledge fetch, particularly for GokuL (which makes use of SSD, HDD for knowledge storage)

For outdated metric knowledge residing in GokuL, we determined to additionally retailer rolled up knowledge to spice up question latency. Rolling up means lowering the granularity of the time collection knowledge factors by storing aggregated values for the determined interval. For instance: A uncooked time collection stream

when aggregated utilizing rollup interval of 5 and rollup aggregators of sum, min, max, rely, common may have 5 shorter time collection streams as follows:

The next desk explains the tiering and rollup technique:

Rollup benefitted the GokuL service in 3 methods:

• Diminished the storage price of plentiful uncooked knowledge
• Decreased the info fetch price from ssd, lowered the cpu aggregation price, and thus lowered the question latency
• Some queries that might day out from the OpenTSDB supporting HBase clusters would return profitable question outcomes from GokuL.

The rollup aggregation is finished within the Goku compactor (defined right here) earlier than it creates the sst information containing the time collection knowledge to be saved within the rocksDB primarily based GokuL situations.

In manufacturing, we observe that p99 latency of queries utilizing rolled up knowledge is nearly 1000x lower than queries utilizing uncooked knowledge.

At question time, Goku responds with an exception stating “cardinality restrict exceeded” if the variety of time collection the question would choose/ learn from publish filtering exceeds the pre-configured restrict. That is to guard the Goku system sources because of noisy costly queries. We noticed queries for prime cardinality metrics hitting timeouts, chewing up the system sources, and affecting the in any other case low latency queries. Usually, after analyzing the excessive cardinality or timing out queries, we discovered that the tag(s) that contributed to the excessive cardinality of the metric weren’t even wanted by the person within the remaining question end result.

The pre-aggregation function was launched with the purpose of eradicating these undesirable tags within the pre-aggregated metrics, thus, lowering the unique cardinality, lowering the question latency, and efficiently serving the question outcomes to the person with out timing out or consuming a variety of system sources. The function creates and shops aggregated time collection by eradicating pointless tags that the person mentions. The aggregated time collection has tags that the person has particularly requested to protect. For instance:

If the person asks to allow pre-aggregation for the metric “app.some_stat” and desires to protect solely the cluster and az info, the pre-aggregated time collection will appear to be this:

Be aware how the cardinality of the pre-aggregated metric is lowered from 5 to three.

The pre-aggregated metrics are new time collection created inside Goku that don’t change the unique uncooked time collection. Additionally for the sake of simplicity, we determined to not introduce these metrics again into the standard ingestion pipeline that we emit to Kafka.

Here’s a movement of how enabling pre-aggregation works:

1. Customers experiencing excessive latency queries or queries hitting cardinality restrict exceeded timeout determine to allow pre-aggregation for the metric.
2. The Goku workforce gives the tag mixture distribution of the metric to the person. For instance:

3. Customers determine on the tags they wish to protect within the pre-aggregated time collection. The “to be preserved” tags are known as grouping tags. There may be additionally an elective provision supplied to pick a selected tag key == tag worth mixture to be preserved and discard all different tag worth combos for that tag key. These provisions are known as conditional tags.

4. Consumer is notified of the lowered cardinality and pre-aggregation is enabled for the metric which the person finalizes.

Write path change:

After consuming an information level for a metric from Kafka, the Goku Brief Time period host checks if the time collection qualifies to be pre-aggregated. If the time collection qualifies, the worth of the datapoint is entered in an in reminiscence knowledge construction, which data the sum, max, min, rely, and imply of the info seen up to now. The information construction additionally emits 5 aggregated knowledge factors (aggregations talked about above) for the time collection with an internally modified Goku metric title each minute.

Learn Path change:

Within the question request to Goku Root, the observability statsboard shopper sends a boolean, which determines if the pre-aggregated model of the metric must be queried. Goku Root does the corresponding metric title change to question the fitting time collection.

Success story: One manufacturing metric (within the instance supplied above) saved in Goku on which alerts have been set was seeing excessive cardinality exceptions (cardinality ~32M throughout peak hours).

We reached out to the person to assist perceive the use case and advised enabling pre-aggregation for his or her metric. As soon as we enabled pre-aggregation, the queries efficiently accomplished with latencies under 100ms.

Now we have onboarded greater than 50 use circumstances for pre-aggregation.

Throughout launch to manufacturing, a question timeout function needed to be carried out in Goku Lengthy Time period to keep away from an costly question consuming the server sources for a very long time. This, nevertheless, resulted in customers of high-priced queries seeing timeouts and wastage of server sources even when it was for a brief time period (i.e. configured question timeout). To confront this difficulty, the pagination function was launched, which might promise a non timed out end result to the tip person of an costly question, regardless that it might take longer than normal. It could additionally break/ plan the question in such a manner that useful resource utilization on the server is managed.

The workflow of the pagination function is:

1. Question shopper sends a PagedQueryRequest to Goku Root if the metric is within the record of pagination supported metrics.
2. Goku Root plans the question primarily based on time slicing.
3. Goku Root and Question shopper have a collection of request-response exchanges with the foundation server. This gives the question shopper with a touch of what must be the following begin and finish time vary of the question and its personal IP tackle in order that the site visitors managing envoy can route the question to the fitting server.

Now we have included ~10 use circumstances in manufacturing.

The next are concepts we’ve got to additional enhance question expertise in Goku:

Tag-based aggregation in Goku

Throughout compaction, generate pre-aggregated time collection by aggregating on the excessive cardinality contributing tags like host, and many others. Work with the shopper workforce to establish such tags. This can generate time collection and enhance the storage price, however not by a lot. Within the queries, if the excessive cardinality tags will not be current, the leaf server will robotically serve utilizing the pre-aggregated time collection.

Presently, the shopper observability workforce already has a function in place to take away the excessive cardinality contributing host tag from a set of long run metrics. Sooner or later, this could make use of the tag-based aggregation help in Goku, or Goku can present the tips that could the observability workforce primarily based on the question evaluation above to incorporate extra long run metrics of their record.

Publish-query processing help in Goku

Many customers of statsboard use the tscript publish question processing to additional course of their outcomes. The pushing of this processing layer into Goku can present the next advantages:

1. Leverages further compute sources obtainable at Goku Root and Goku Leaf (GokuS and GokuL) clusters
2. Much less knowledge over the community resulting in doable decrease question latencies

Some examples of publish question processing help embody discovering the highest N time collection, summing of the time collection, and many others.

Backfilling help in pre-aggregation

We at present don’t help pre-aggregated queries for a metric for a time vary that falls earlier than the time the metric was configured for pre-aggregation. For instance: if a metric was enabled for pre-aggregation on 1st Jan 2022 00:00:00, customers received’t be capable of question pre-aggregated knowledge for time earlier than thirty first Dec 2021 23:59:59. By supporting pre-aggregation throughout compaction, we are able to take away this restrict and slowly however steadily (as bigger tier buckets begin forming), customers will begin seeing pre-aggregated knowledge for older time ranges.

SQL help

Presently, Goku is queryable solely by utilizing a thrift interface for RPC. SQL is extensively used worldwide as a querying framework for knowledge, and having SQL help in Goku would considerably assist analytical use circumstances. We’re beginning to see an growing demand for this and are exploring options.

Learn from S3

A capability to retailer and skim from S3 would assist Goku prolong the ttl of uncooked knowledge, and even prolong the ttl of queryable metrics knowledge. This might additionally show price useful to retailer metrics which are sometimes used.

Particular because of Rui Zhang, Hao Jiang, and Miao Wang for his or her efforts in supporting the above options. An enormous because of the Observability workforce for his or her assist and help for these options on the person going through aspect.

Within the subsequent weblog, we are going to deal with how we introduced down the price of the Goku service(s).

To study extra about engineering at Pinterest, take a look at the remainder of our Engineering Weblog and go to our Pinterest Labs website. To discover and apply to open roles, go to our Careers web page.