Mobius.RRD (Mobius v0.8.0)

Copy Markdown View Source

A round robin database for Mobius

This is the RRD used by Mobius to store historical metric data.

A round robin database (RRD) is a data store that has a circular buffer to store information. As time moves forward the older data points get overwritten by newer data points. This type of data storage is useful for a consistent memory footprint for time series data.

The Mobius.RRD implementation provides four time resolutions. These are: seconds, minutes, hours and days. Each resolution can be configured to allow for as many single data points as you see fit. For example, if you want to store three days of data at an hour resolution you can configure the RRD like so:

RRD.new(hours: 72)

The above will configure the hour resolution to store 72 hours worth of data points in the hour archive.

The default resolutions are:

  • 60 days (each day for about 2 months)
  • 48 hours (each hour for 2 days)
  • 120 minutes (each minute for 2 hours)
  • 120 seconds (each second for 2 minutes)

For more information about round robin databases, RRD tool is a great resource to study.

Summary

Types

Options for the RRD

Identity of a histogram-enabled metric definition: name plus sorted tag keys.

Options for loading a persisted RRD

Resolution name

Options for saving RRD into a binary

Snapshot payload stored at each RRD timestamp.

t()

Functions

Return all items in order

Insert an item for the specified time

Insert an item for the specified time, recovering from backwards clock steps

Load persisted data back into a TimeLayerBuffer

Create a new RRD

Return all items with timestamps equal to or after the specified one

Return all items within the specified range

Whether the RRD may have discarded its oldest scrapes.

Serialize to an iolist

Types

create_opt()

@type create_opt() :: {resolution(), non_neg_integer()}

Options for the RRD

For resolution options you specify which resolution and the max number of metric data to keep for that resolution.

For example, if the RRD were to track seconds up to five minutes it would need to track 300 seconds. Also, if the same RRD wanted to track day resolution for a year, it would need to contain 365 days.

Mobius.RRD.new(seconds: 300, days: 365)

histogram_config_key()

@type histogram_config_key() :: {Mobius.metric_name(), [atom()]}

Identity of a histogram-enabled metric definition: name plus sorted tag keys.

load_opt()

@type load_opt() :: {:histogram_configs, %{required(histogram_config_key()) => map()}}

Options for loading a persisted RRD

  • :histogram_configs - the current resolved sketch configuration per histogram-enabled metric. When given, v3 files are checked series by series and persisted histogram data whose recorded configuration does not match the current one is dropped — bin indices are meaningless under a different configuration, so reinterpreting them would produce silently wrong quantiles. When omitted, no compatibility check is performed.

resolution()

@type resolution() :: :seconds | :minutes | :hours | :days

Resolution name

save_opt()

@type save_opt() ::
  {:histogram_configs, %{required(histogram_config_key()) => map()}}
  | {:compression_level, 0..9}

Options for saving RRD into a binary

  • :histogram_configs - the resolved sketch configuration per histogram-enabled metric, recorded in v3 files so load/3 can detect histogram data produced under a different configuration
  • :compression_level - the zlib level (0..9) used to compress the payload, defaults to 9. 0 disables compression.

snapshot()

@type snapshot() ::
  {[Mobius.metric_record()], Mobius.Scraper.histograms()}
  | [Mobius.metric_record()]

Snapshot payload stored at each RRD timestamp.

A {records, histograms} tuple where records is the list of regular Mobius.metric_record/0 tuples and histograms is the compact per-metric histogram payload map from Mobius.Scraper (one :erlang.term_to_binary/1-encoded sketch snapshot per metric series).

Plain [Mobius.metric_record()] is also accepted for backwards compatibility with older callers; it is treated as records with no histogram data.

t()

@opaque t()

Functions

all(rrd)

@spec all(t()) :: [{Mobius.timestamp(), snapshot()}]

Return all items in order

insert(rrd, ts, item)

@spec insert(t(), integer(), snapshot()) :: t()

Insert an item for the specified time

insert_with_regression_recovery(rrd, ts, item)

@spec insert_with_regression_recovery(t(), integer(), snapshot()) :: t()

Insert an item for the specified time, recovering from backwards clock steps

Behaves like insert/3 unless ts is more than 60 seconds below the second high-water mark. The marks only move forward, so after the wall clock steps backwards that far insert/3 would silently drop every scrape until the clock catches back up — and since the RRD is persisted, the condition survives reboots. A trusted clock that steps backwards also means everything recorded above ts was stamped by a clock now known to have been ahead, so those entries are pruned (and the high-water marks reset) before the item is inserted. Backwards jitter within the margin keeps the insert/3 drop behavior.

Callers are expected to only use this with timestamps from a synchronized clock: a pre-sync timestamp (e.g. 1970-era before NTP) would prune valid history.

load(rrd, binary, opts \\ [])

@spec load(t(), binary(), [load_opt()]) ::
  {:ok, t()} | {:error, Mobius.DataLoadError.t()}

Load persisted data back into a TimeLayerBuffer

The rrd that's passed in is expected to be a new one without any entries.

new(opts \\ [])

@spec new([create_opt()]) :: t()

Create a new RRD

The default resolution values are:

  • 60 days (each day for about 2 months)
  • 48 hours (each hour for 2 days)
  • 120 minutes (each minute for 2 hours)
  • 120 seconds (each second for 2 minutes)

query(rrd, from)

@spec query(t(), from :: integer()) :: [{integer(), snapshot()}]

Return all items with timestamps equal to or after the specified one

query(rrd, from, to)

@spec query(t(), from :: integer(), to :: integer()) :: [{integer(), snapshot()}]

Return all items within the specified range

rolled_off?(rrd)

@spec rolled_off?(t()) :: boolean()

Whether the RRD may have discarded its oldest scrapes.

The day archive holds the oldest retained data, so history starts rolling off once it has filled to capacity. Window queries use this to distinguish "the metric started inside retention" (an absent pre-window snapshot means an empty cumulative baseline) from "the metric's earlier history rolled off" (an absent pre-window snapshot means the baseline was lost, and the window must be truncated to the retained data instead).

save(rrd, opts \\ [])

@spec save(t(), [save_opt()]) :: iolist()

Serialize to an iolist

Always writes the current serialization format; load/3 handles older formats on the way back in.