A round robin database for Mobius
This is the RRD used by Mobius to store historical metric data.
A round robin database (RRD) is a data store that has a circular buffer to store information. As time moves forward the older data points get overwritten by newer data points. This type of data storage is useful for a consistent memory footprint for time series data.
The Mobius.RRD implementation provides four time resolutions. These are: seconds,
minutes, hours and days. Each resolution can be configured to allow for as
many single data points as you see fit. For example, if you want to store three
days of data at an hour resolution you can configure the RRD like so:
RRD.new(hours: 72)The above will configure the hour resolution to store 72 hours worth of data points in the hour archive.
The default resolutions are:
- 60 days (each day for about 2 months)
- 48 hours (each hour for 2 days)
- 120 minutes (each minute for 2 hours)
- 120 seconds (each second for 2 minutes)
For more information about round robin databases, RRD tool is a great resource to study.
Summary
Types
Options for the RRD
Identity of a histogram-enabled metric definition: name plus sorted tag keys.
Options for loading a persisted RRD
Resolution name
Options for saving RRD into a binary
Snapshot payload stored at each RRD timestamp.
Functions
Return all items in order
Insert an item for the specified time
Insert an item for the specified time, recovering from backwards clock steps
Load persisted data back into a TimeLayerBuffer
Create a new RRD
Return all items with timestamps equal to or after the specified one
Return all items within the specified range
Whether the RRD may have discarded its oldest scrapes.
Serialize to an iolist
Types
@type create_opt() :: {resolution(), non_neg_integer()}
Options for the RRD
For resolution options you specify which resolution and the max number of metric data to keep for that resolution.
For example, if the RRD were to track seconds up to five minutes it would need
to track 300 seconds. Also, if the same RRD wanted to track day resolution
for a year, it would need to contain 365 days.
Mobius.RRD.new(seconds: 300, days: 365)
@type histogram_config_key() :: {Mobius.metric_name(), [atom()]}
Identity of a histogram-enabled metric definition: name plus sorted tag keys.
@type load_opt() :: {:histogram_configs, %{required(histogram_config_key()) => map()}}
Options for loading a persisted RRD
:histogram_configs- the current resolved sketch configuration per histogram-enabled metric. When given, v3 files are checked series by series and persisted histogram data whose recorded configuration does not match the current one is dropped — bin indices are meaningless under a different configuration, so reinterpreting them would produce silently wrong quantiles. When omitted, no compatibility check is performed.
@type resolution() :: :seconds | :minutes | :hours | :days
Resolution name
@type save_opt() :: {:histogram_configs, %{required(histogram_config_key()) => map()}} | {:compression_level, 0..9}
Options for saving RRD into a binary
:histogram_configs- the resolved sketch configuration per histogram-enabled metric, recorded in v3 files soload/3can detect histogram data produced under a different configuration:compression_level- the zlib level (0..9) used to compress the payload, defaults to9.0disables compression.
@type snapshot() :: {[Mobius.metric_record()], Mobius.Scraper.histograms()} | [Mobius.metric_record()]
Snapshot payload stored at each RRD timestamp.
A {records, histograms} tuple where records is the list of regular
Mobius.metric_record/0 tuples and histograms is the compact per-metric
histogram payload map from Mobius.Scraper (one
:erlang.term_to_binary/1-encoded sketch snapshot per metric series).
Plain [Mobius.metric_record()] is also accepted for backwards compatibility
with older callers; it is treated as records with no histogram data.
@opaque t()
Functions
@spec all(t()) :: [{Mobius.timestamp(), snapshot()}]
Return all items in order
Insert an item for the specified time
Insert an item for the specified time, recovering from backwards clock steps
Behaves like insert/3 unless ts is more than 60 seconds
below the second high-water mark. The marks only move forward, so after the
wall clock steps backwards that far insert/3 would silently drop every
scrape until the clock catches back up — and since the RRD is persisted, the
condition survives reboots. A trusted clock that steps backwards also means
everything recorded above ts was stamped by a clock now known to have been
ahead, so those entries are pruned (and the high-water marks reset) before
the item is inserted. Backwards jitter within the margin keeps the insert/3
drop behavior.
Callers are expected to only use this with timestamps from a synchronized clock: a pre-sync timestamp (e.g. 1970-era before NTP) would prune valid history.
@spec load(t(), binary(), [load_opt()]) :: {:ok, t()} | {:error, Mobius.DataLoadError.t()}
Load persisted data back into a TimeLayerBuffer
The rrd that's passed in is expected to be a new one without any entries.
@spec new([create_opt()]) :: t()
Create a new RRD
The default resolution values are:
- 60 days (each day for about 2 months)
- 48 hours (each hour for 2 days)
- 120 minutes (each minute for 2 hours)
- 120 seconds (each second for 2 minutes)
Return all items with timestamps equal to or after the specified one
Return all items within the specified range
Whether the RRD may have discarded its oldest scrapes.
The day archive holds the oldest retained data, so history starts rolling off once it has filled to capacity. Window queries use this to distinguish "the metric started inside retention" (an absent pre-window snapshot means an empty cumulative baseline) from "the metric's earlier history rolled off" (an absent pre-window snapshot means the baseline was lost, and the window must be truncated to the retained data instead).
Serialize to an iolist
Always writes the current serialization format; load/3 handles older
formats on the way back in.