Lower your storage prices with Amazon OpenSearch Service index rollups

Amazon OpenSearch Service is a completely managed service to help search, log analytics, and generative AI Retrieval Increase Era (RAG) workloads within the AWS Cloud. It simplifies the deployment, safety, and scaling of OpenSearch clusters. As organizations scale their log analytics workloads by repeatedly gathering and analyzing huge quantities of knowledge, they typically wrestle to keep up fast entry to historic info whereas managing prices successfully. OpenSearch Service addresses these challenges by means of its tiered storage choices: sizzling, UltraWarm, and chilly storage. These storage tiers are nice choices to assist optimize prices and provide a steadiness between efficiency and affordability, so organizations can handle their information extra effectively. Organizations can select between these completely different storage tiers by conserving information in costly sizzling storage for fast entry or transferring it to cheaper chilly storage with restricted accessibility. This trade-off turns into significantly difficult when organizations want to investigate each current and historic information for compliance, development evaluation, or enterprise intelligence.

On this submit, we discover the right way to use index rollups in Amazon OpenSearch Service to deal with this problem. This function helps organizations effectively handle their historic information by mechanically summarizing and compressing older information whereas sustaining its analytical worth, considerably lowering storage prices in any storage tier with out sacrificing the power to question historic info successfully.

Index rollups overview

Index rollups present a mechanism to combination historic information into summarized indexes at specified time intervals. This function is especially helpful for time sequence information the place the granularity of older information could be diminished whereas sustaining significant analytics capabilities.

Key advantages embrace:

• Lowered storage prices (varies by granularity stage), for instance:
  • Bigger financial savings when aggregating from seconds to hours
  • Reasonable financial savings when aggregating from seconds to minutes
• Improved question efficiency of historic information
• Maintained information accessibility for long-term analytics
• Automated information summarization course of

Index rollups are a part of a complete information administration technique. The true price financial savings come from correctly managing your information lifecycle together with rollups. To realize significant price reductions, you have to take away or transfer the unique information to a lower-cost storage tier after creating the rollup.

For purchasers already utilizing Index State Administration (ISM) to maneuver older information to UltraWarm or chilly tiers, rollups can present vital further advantages. By aggregating information at greater time intervals earlier than transferring it to lower-cost tiers, you possibly can dramatically cut back the quantity of knowledge in these tiers, resulting in additional price financial savings. This technique is especially efficient for workloads with giant quantities of time sequence information, usually measuring in terabytes or petabytes. The bigger your information quantity, the extra impactful your financial savings shall be when implementing rollups accurately.

Index rollups could be carried out utilizing ISM insurance policies by means of the OpenSearch Dashboards UI or the OpenSearch API. Index rollups require OpenSearch or Elasticsearch 7.9 or later.

The choice to make use of completely different storage tiers requires cautious consideration of a corporation’s particular wants, balancing the will for price financial savings with the requirement for information accessibility and efficiency. As information volumes proceed to develop and analytics grow to be more and more vital, discovering the correct storage technique turns into essential for companies to stay aggressive and compliant whereas managing their budgets successfully.

On this submit, we think about a state of affairs with a big quantity of time sequence information that may be aggregated utilizing the Rollup API. With rollups, you have got the pliability to both retailer aggregated information within the sizzling tier for speedy entry or combination and put it up for sale to more cost effective tiers comparable to UltraWarm or chilly storage. This strategy permits for environment friendly information and index lifecycle administration whereas optimizing each efficiency and price.

Index rollups are sometimes confused with index rollovers, that are automated OpenSearch Service operations that create new indexes when specified thresholds are met, for instance by age, dimension, or doc depend. This function maintains uncooked information whereas optimizing cluster efficiency by means of managed index progress. For instance, rolling over when an index reaches 50 GB or is 30 days previous.

Use circumstances for index rollups

Index rollups are perfect for eventualities the place it’s essential to steadiness storage prices with information granularity, comparable to:

• Time sequence information that requires completely different granularity ranges over time – For instance, Web of Issues (IoT) sensor information the place real-time precision issues just for the newest information.
  • Conventional strategy – It is not uncommon for customers to maintain all information in costly sizzling storage for fast accessibility. Nonetheless, this isn’t optimum for price.
  • Beneficial – Retain current (per second) information in sizzling storage for rapid entry. For older durations, retailer aggregated (hourly or each day) information utilizing index rollups. Transfer or delete the higher-granularity previous information from the new tier. This balances accessibility and cost-effectiveness.
• Historic information with cost-optimization wants – For instance, system efficiency metrics the place total developments are extra precious than exact values over time.
  • Conventional strategy – It is not uncommon for customers to retailer all efficiency metrics at full granularity indefinitely, consuming extreme cupboard space. We don’t suggest storing information indefinitely. Implement a knowledge retention coverage primarily based in your particular enterprise wants and compliance necessities.
  • Beneficial – Keep detailed metrics for current monitoring (final 30 days) and combination older information into hourly or each day summaries. This preserves the development evaluation functionality whereas considerably lowering storage prices.
• Log information with rare historic entry and low worth – For instance, utility error logs the place detailed investigation is primarily wanted for current incidents.
  • Conventional strategy – It is not uncommon for customers to maintain all log entries at full element, no matter age or entry frequency.
  • Beneficial – Protect detailed logs for an lively troubleshooting interval (for instance, 1 week) and preserve summarized error patterns and statistics for older durations. This allows historic sample evaluation whereas lowering storage overhead.

Schema design

A well-planned schema is essential for profitable rollup implementation. Correct schema design makes certain your rolled-up information stays precious for evaluation whereas maximizing storage financial savings. Contemplate the next key features:

• Determine fields required for long-term evaluation – Fastidiously choose fields that present significant insights over time, avoiding pointless information retention.
• Outline aggregation varieties for every discipline, comparable to min, max, sum, and common – Select acceptable aggregation strategies that protect the analytical worth of your information.
• Decide which fields could be excluded from rollups – Scale back storage prices by omitting fields that don’t contribute to long-term evaluation.
• Contemplate mapping compatibility between supply and goal indexes – Present profitable information transition with out mapping conflicts. This includes:
  • Matching information varieties (for instance, date fields stay as date in rollups)
  • Dealing with nested fields appropriately
  • Making certain all required fields are included within the rollup
  • Contemplating the impression of analyzed vs. non-analyzed fields
  • Incompatible mappings can result in failed rollup jobs or incorrect information aggregation.

Useful and non-functional necessities

Earlier than implementing index rollups, think about the next:

• Knowledge entry patterns – When implementing information rollup methods, it’s essential to first analyze information entry patterns, together with question frequency and utilization durations, to find out optimum rollup intervals. This evaluation ought to result in particular granularity metrics, comparable to deciding between hourly or each day aggregations, whereas establishing clear thresholds primarily based on each information quantity and question necessities. These choices needs to be documented alongside particular aggregation guidelines for every information sort.
• Knowledge progress fee – Storage optimization begins with calculating your present dataset dimension and its progress fee. This info helps quantify potential house reductions throughout completely different rollup methods. Efficiency metrics, significantly anticipated question response occasions, needs to be outlined upfront. Moreover, set up monitoring KPIs specializing in latency, throughput, and useful resource utilization to verify the system meets efficiency expectations.
• Compliance or information retention necessities – Retention planning requires cautious consideration of regulatory necessities and enterprise wants. Develop a transparent retention coverage that specifies how lengthy to maintain various kinds of information at numerous granularity ranges. Implement systematic processes for archiving or deleting older information and preserve detailed documentation of storage prices throughout completely different retention durations.
• Useful resource utilization and planning – For profitable implementation, correct cluster capability planning is important. This includes precisely sizing computing assets, together with CPU, RAM, and storage necessities. Outline particular time home windows for executing rollup jobs to attenuate impression on common operations. Set clear useful resource utilization thresholds and implement proactive capability monitoring. Lastly, develop a scalability plan that accounts for each horizontal and vertical progress to accommodate future wants.

Operational necessities

Correct operational planning facilitates clean ongoing administration of your rollup implementation. That is important for sustaining information reliability and system well being:

• Monitoring – You could monitor rollup jobs for his or her accuracy and desired outcomes. This implies implementing automated checks that validate information completeness, aggregation accuracy, and job execution standing. Arrange alerts for failed jobs, information inconsistencies, or when aggregation outcomes fall outdoors anticipated ranges.
• Scheduling hours – Schedule rollup operations during times of low system utilization, usually throughout off-peak hours. Doc these upkeep home windows clearly and talk them to all stakeholders. Embody buffer time for potential points and set up clear procedures for what occurs if a upkeep window must be prolonged.
• Backup and restoration – OpenSearch Service takes automated snapshots of your information at 1-hour intervals. However you possibly can outline and implement complete backup procedures utilizing snapshot administration performance to help your Restoration Time Goal (RTO) and Restoration Level Goal (RPO).

Your RPO could be personalized by means of completely different rollup schedules primarily based on index patterns. This flexibility helps you outline various information loss tolerance ranges in line with your information’s criticality. For mission-critical indexes, you possibly can configure extra frequent rollups, whereas sustaining much less frequent schedules for analytical information.

You possibly can tailor RTO administration in OpenSearch per index sample by means of backup and replication choices. For important rollup indexes, implementing cross-cluster replication maintains up-to-date copies, considerably lowering restoration time. Different indexes would possibly use customary backup procedures, balancing restoration pace with operational prices. This versatile strategy helps you optimize each storage prices and restoration targets primarily based in your particular enterprise necessities for various kinds of information inside your OpenSearch deployment.

Earlier than implementing rollups, audit all purposes and dashboards that use the information being aggregated. Replace queries and visualizations to accommodate the brand new information construction. Take a look at these modifications completely in a staging atmosphere to verify they proceed to supply correct outcomes with the rolled-up information. Create a rollback plan in case of sudden points with dependent purposes.

Within the following sections, we stroll by means of the steps to create, run, and monitor a rollup job.

Create a rollup job

As mentioned in earlier sections, there are some issues when selecting good candidates for index rollup utilization. Constructing on this idea, establish your indexes to roll up their information and create the roles.The next code is an instance of making a primary rollup job:

PUT /_plugins/_rollup/jobs/sensor_hourly_rollup
{
  "rollup": {
    "rollup_id": "sensor_1_hour_rollup",
    "enabled": true,
    "schedule": {
      "interval": {
        "start_time": 1746632400,        
        "interval": 1,
        "unit": "hours",
        "schedule_delay": 0
      }
    },
    "description": "Rolls up sensor information 1 hourly per device_id",
    "source_index": "sensor-*",           
    "target_index": "sensor_rolled_hour",
    "page_size": 1000,
    "delay": 0,
    "steady": true,
    "dimensions": [
      {
        "date_histogram": {
          "fixed_interval": "1h",
          "source_field": "timestamp",
          "target_field": "timestamp",
          "timezone": "UTC"
        }
      },
      {
        "terms": {
          "source_field": "device_id",
          "target_field": "device_id"
        }
      }
    ],
    "metrics": [
      {
        "source_field": "temperature",
        "metrics": [
          { "avg": {} },
          { "min": {} },
          { "max": {} }
        ]
      },
      {
        "source_field": "humidity",
        "metrics": [
          { "avg": {} },
          { "min": {} },
          { "max": {} }
        ]
      },
      {
        "source_field": "strain",
        "metrics": [
          { "avg": {} },
          { "min": {} },
          { "max": {} }
        ]
      },
      {
        "source_field": "battery",
        "metrics": [
          { "avg": {} },
          { "min": {} },
          { "max": {} }
        ]
      }
    ]
  }
}

This rollup job processes IoT sensor information, aggregating readings from the sensor-* index sample into hourly summaries saved in sensor_rolled_hour. It maintains device-level granularity whereas calculating common, minimal, and most values for temperature, humidity, strain, and battery ranges. The job executes hourly, processing 1,000 paperwork per batch.

The previous code assumes that the device_id discipline is of sort key phrase; observe that aggregation can’t be carried out on the textual content discipline.

Begin the rollup job

After you create the job, it can mechanically be scheduled primarily based on the job’s configuration (confer with the schedule: a part of the job instance code within the earlier part). Nonetheless, you may as well set off the job manually utilizing the next API name:

POST _plugins/_rollup/jobs/sensor_hourly_rollup/_start

The next is an instance of the outcomes:

Monitor progress

Utilizing Dev Instruments, run the next command to observe the progress:

GET _plugins/_rollup/jobs/sensor_hourly_rollup/_explain

The next is an instance of the outcomes:

{
  "sensor_hourly_rollup": {
    "metadata_id": "pCDjMZcBgTxYF90dWEfP",
    "rollup_metadata": {
      "rollup_id": "sensor_hourly_rollup",
      "last_updated_time": 1749043472416,
      "steady": {
        "next_window_start_time": 1749043440000,
        "next_window_end_time": 1749043560000
      },
      "standing": "began",
      "failure_reason": null,
      "stats": {
        "pages_processed": 374603,
        "documents_processed": 390,
        "rollups_indexed": 200,
        "index_time_in_millis": 789,
        "search_time_in_millis": 402202
      }
    }
  }
}  

The GET _plugins/_rollup/jobs/sensor_hourly_rollup/_explain command exhibits the present standing and statistics of the sensor_hourly_rollup job. The response exhibits vital statistics such because the variety of processed paperwork, listed rollups, time spent on indexing and looking out, and data of any failures. The standing signifies whether or not the job is lively (began) or stopped (stopped) and exhibits the final processed timestamp. This info is essential for monitoring the effectivity and well being of the rollup course of, serving to directors monitor progress, establish potential points or bottlenecks, and make sure the job is working as anticipated. Common checks of those statistics can assist in optimizing the rollup job’s efficiency and sustaining information integrity.

Actual-world instance

Let’s think about a state of affairs the place an organization collects IoT sensor information, ingesting 240 GB of knowledge per day to an OpenSearch cluster, which totals 7.2 TB per thirty days.

The next is an instance file:

"_source": {
          "timestamp": "2024-01-01T10:00:00Z",
          "device_id": "sensor_001",
          "temperature": 26.1,
          "humidity": 43,
          "strain": 1009.3,
          "battery": 90
}

Assume you have got a time sequence index with the next configuration:

• Ingest fee: 10 million paperwork per hour
• Retention interval: 30 days
• Every doc dimension: Roughly 1 KB

The entire storage with out rollups is as follows:

• Per-day storage dimension: 10,000,000 docs per hour × ~1 KB × 24 hours per day = ~240 GB
• Per-month storage dimension: 240 GB × 30 days = ~7.2 TB

The choice to implement rollups needs to be primarily based on a cost-benefit evaluation. Contemplate the next:

• Present storage prices vs. potential financial savings
• Compute prices for working rollup jobs
• Worth of granular information over time
• Frequency of historic information entry

For smaller datasets (for instance, lower than 50 GB/day), the advantages may be much less vital. As information volumes develop, the fee financial savings grow to be extra compelling.

Rollup configuration

Let’s roll up the information with the next configuration:

• From 1-minute granularity to 1-hour granularity
• Aggregating common, min, and max, grouped by device_id
• Decreasing 60 paperwork per minute to 1 rollup doc per minute

The brand new doc depend per hour is as follows:

• Per-hour paperwork: 10,000,000/60 = 166,667 docs per hour
• Assuming every rollup doc is 2 KB (additional metadata), whole rollup storage: 166,667 docs per hour × 24 hours per day × 30 days × 2KB ˜= 240 GB/month

Confirm all required information exists within the new rolled index, then delete the unique index to take away uncooked information manually or by utilizing ISM insurance policies (as mentioned within the subsequent part).

Execute the rollup job following the previous directions to combination information into the brand new rolled up index. To view your aggregated outcomes, run the next code:

GET sensor_rolled_hour/_search
{
  "dimension": 0,
  "aggs": {
    "per_device": {
      "phrases": {
        "discipline": "device_id",
        "dimension": 200,
        "shard_size": 200
      },
      "aggs": {
        "temperature_avg": {
          "avg": {
            "discipline": "temperature"
          }
        },
        "temperature_min": {
          "min": {
            "discipline": "temperature"
          }
        },
        "temperature_max": {
          "max": {
            "discipline": "temperature"
          }
        }
      }
      }
    }
  } 

The next code exhibits the instance outcomes:

"aggregations": {
    "per_device": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "sensor_001",
          "doc_count": 98,
          "temperature_min": {
            "value": 24.100000381469727
          },
          "temperature_avg": {
            "value": 26.287754603794642
          },
          "temperature_max": {
            "value": 27.5
          }
        },
        {
          "key": "sensor_002",
          "doc_count": 98,
          "temperature_min": {
            "value": 20.600000381469727
          },
          "temperature_avg": {
            "value": 22.192856146364797
          },
          "temperature_max": {
            "value": 22.799999237060547
          }
        },...]

This doc represents the rolled-up information for sensor_001 and sensor_002 throughout a 1-hour interval. It aggregates 1 hour of sensor readings right into a single file, storing minimal, common, and most values for temperature ranges. The file consists of metadata in regards to the rollup course of and timestamps for information monitoring. This aggregated format considerably reduces storage necessities whereas sustaining important statistical details about the sensor’s efficiency throughout that hour.

We are able to calculate the storage financial savings as follows:

• Authentic storage: 7.2 TB (or 7200 GB)
• Publish-rollup storage: 240 GB
• Storage financial savings: ((7.2 TB – 240 GB)/7.2 GB) × 100 = 96.67% financial savings

Utilizing OpenSearch rollups as demonstrated on this instance, you possibly can obtain roughly 96% storage financial savings whereas preserving vital combination insights.

The aggregation ranges and doc sizes could be personalized in line with your particular use case necessities.

Automate rollups with ISM

To totally understand the advantages of index rollups, automate the method utilizing ISM insurance policies. The next code is an instance that implements a rollup technique primarily based on the given state of affairs:

PUT _plugins/_ism/insurance policies/sensor_rollup_policy
{
  "coverage": {
    "description": "Roll up sensor information and delete unique",
    "default_state": "sizzling",
    "ism_template": {
      "index_patterns": ["sensor-*"],
      "precedence": 100
    },
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          {
            "state_name": "rollup",
            "conditions": {
              "min_index_age": "1d"
            }
          }
        ]
      },
      {
        "title": "rollup",
        "actions": [
          {
            "rollup": {
              "ism_rollup": {
                "target_index": "sensor_rolled_minutely",
                "description": "Rollup sensor data to minutely aggregations",
                "page_size": 1000,
                "dimensions": [
                  {
                    "date_histogram": {
                      "fixed_interval": "1m",
                      "source_field": "timestamp",
                      "target_field": "timestamp"
                    }
                  },
                  {
                    "terms": {
                      "source_field": "device_id",
                      "target_field": "device_id"
                    }
                  }
                ],
                "metrics": [
                  {
                    "source_field": "temperature",
                    "metrics": [{ "avg": {} }, { "min": {} }, { "max": {} }]
                  },
                  {
                    "source_field": "humidity",
                    "metrics": [{ "avg": {} }, { "min": {} }, { "max": {} }]
                  }
                ]
              }
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "2d"
            }
          }
        ]
      },
      {
        "title": "delete",
        "actions": [
          {
            "delete": {}
          }
        ]
      }
    ]
  }
}

This ISM coverage automates the rollup course of and information lifecycle:

• 1. Applies to all indexes matching the sensor-* sample.
  2. Retains unique information within the sizzling state for 1 day.
  3. After 1 day, rolls up the information into minutely aggregations. Aggregates by device_id and calculates common, minimal, and most for temperature and humidity.
  4. Shops rolled-up information within the sensor_rolled_minutely index.
  5. Deletes the unique index 2 days after rollup.

This technique presents the next advantages:

• Current information is offered at full granularity
• Historic information is effectively summarized
• Storage is optimized by eradicating unique information after rollup

You possibly can monitor the coverage’s execution utilizing the next command:

GET _plugins/_ism/insurance policies/sensor_rollup_policy

Bear in mind to regulate the timeframes, metrics, and aggregation intervals primarily based in your particular necessities and information patterns.

Conclusion

Index rollups in OpenSearch Service present a strong option to handle storage prices whereas sustaining precious historic information entry. By implementing a well-planned rollup technique, organizations can obtain vital price financial savings whereas ensuring their information stays obtainable for evaluation.

To get began, take the next subsequent steps:

• Assessment your present index patterns and information retention necessities
• Analyze your historic information volumes and entry patterns
• Begin with a proof-of-concept rollup implementation in a take a look at atmosphere
• Monitor efficiency and storage metrics to optimize your rollup technique
• Transfer the sometimes accessed information between storage tiers:
  • Delete information you’ll now not use
  • Automate the method utilizing ISM insurance policies

To be taught extra, confer with the next assets:

Concerning the authors