Usage Quotas

Learn how to control how much data your teams are storing in order to create predictable spend month-over-month, while still viewing, alerting on, and retaining the critical data you need.

LogDNA EnterpriseLogDNA Enterprise

Note The Usage Quotas feature is available for customers on an Enterprise plan, and is restricted to Owner and Admin user roles. Please contact your CSM or [email protected] if you expect to see this feature and do not.

About Usage Quotas

The Usage Quotas feature lets you specify a threshold value amount (daily and/or monthly) for the volume of log files to store. When the user-defined threshold (or optionally a specific percentage of the threshold) is met, then LogDNA will apply the configurations that you defined with usage quotas.

These configuration options allow for fine-grained control, providing a range of options such as stopping the storage of all incoming logs, reducing the volume of logs that are stored (based on exclusion criteria) but still viewing all logs in the live tail, or simply creating alerts to notify you that your usage quota threshold is met.

Usage Quotas enables developers and technical teams to view and set alarms for all of the important logs that they need to do their job, while also empowering account administrators to manage how much data their teams are storing and predict and meet their budget. The ideal balance is to both meet the monthly storage budget but still store all critical logs.

The Usage Quotas option is in the left-pane of the LogDNA application, under Settings -> Usage.

Levels of Usage Control

An important aspect of the Usage Quotas feature is the ability to set the specific level of control over the volume of log lines to retain (i.e. to store), and even which types of logs to exclude.

By configuring specific aspects of the usage quota, you can achieve multiple levels of control, from immediately stopping retention to simply sending alerts when the usage quota is met:

  • Hard Quotas: Stop retaining all logs as soon as the defined usage quota is met for daily or monthly quotas (or both). Logs that are not retained are not counted toward your bill at the end of the month. With this option all logs still display in live tailing and trigger any relevant alerts, even though they are not stored.
    Learn how to configure this level of control

  • Soft Quotas: To add finer-grained control to the volume of logs stored, configure one or more triggered exclusion rules, or "Discard Rules," which allow you to specify which logs to store and which to discard. With triggered exclusion rules, you first define a percentage of your overall daily or monthly quota (the "trigger"), and when that percentage of the quota is met, LogDNA stops retaining any logs matching the exclusion criteria. For the exclusion criteria, you can select to discard logs from a specified combination of applications, hosts and queries. This is considered a "soft" quota.
    Learn how to configure this level of control

  • Alerts Only: You can opt to only send alerts when the defined quota value is met, but continue to store all logs and make no changes other than sending alerts when either the quota value is met or edited.
    Learn how to configure this minimal level of control

Considerations

  • Be aware that any logs that are excluded either via exclusion rules or usage quotas will not be archived.

  • Note that Usage Quotas and their related triggered exclusion rules are separate from other defined Exclusion Rules and the Shut Off option. The triggered exclusion rules that are implemented with usage quotas are dynamically toggled on and off based on your Usage Quota settings, whereas standard exclusion rules are persistent and always either on or off.

  • Each account can have a maximum of five (5) Daily triggered exclusion rules and five (5) Monthly triggered exclusion rules.

  • To turn off this feature and disable all triggered exclusion rules, set the quota values to zero (0).

  • Be aware that if a quota value is set lower than what has already been consumed and sent to storage, all incoming logs after the rule has been set will be discarded.

  • Updating the Daily and/or Monthly Usage Quotas will automatically turn off all enabled discard rules associated with the type of usage quota that is updated. For example, if you’ve set a Monthly usage quota of 100GB, and you have three Daily triggered exclusion rules based on that quota, updating that quota to anything other than 100GB will automatically turn off those rules. You can manually turn them back on, but LogDNA does this to protect against making changes that could result in logs not being retained, so we want you to verify those triggered rules are still desired after making the update.

How To Implement Usage Quotas

This section covers how to configure Usage Quotas for the various levels of controlling log file retention and storage.

Hard Quotas: Stop Retaining All Logs When Quota is Met

For the maximum control over the amount of log files stored, use the *Discard logs" option to cease log retention as soon as you reach the volume set in the defined quota. With this option, live tailing and alerting are automatically preserved.

To configure hard quotas follow these steps:

  1. Open the Usage Quotas page by navigating to Settings -> Usage -> Usage Quotas.
  2. On the Usage Quotas page, click Set Usage Quotas. (If you have previously set the usage quota values, click Edit Usage Quotas.)
    The Usage Quotas modal appears.
  3. In the Usage Quotas modal, enter the value for the Daily Usage Quota and/or the Monthly Usage Quota, and specify the unit (MB, GB, or TB).
    4.In the section below the values, select the check box(es) for Discard logs when the Daily Usage Quota is reached and/or Discard logs when the Monthly Usage Quota is reached.
  4. Click Apply to close the modal box.

Soft Quotas: Define Daily or Monthly Triggered Exclusion Rules

Triggered Exclusions Rules allow you to implement a more granular approach, and filter out (i.e. not retain) specific non-critical logs, before you hit the "hard" ceiling of the usage quota value. This method slows the retention rate, and allows you to focus on the log data that matters the most.

For example, if you set a Daily Usage Quota threshold at 2GB, and set the percentage at 50%, then at 1GB the system will start applying the triggered exclusion rule and no longer retain logs from the specified sources/apps/queries.

This approach provides the "best of both worlds" in that you can chose to exclude specific logs from storage, but still have them live tail and set alerts for them.

To configure soft quotas follow these steps:

  1. Open the Usage Quotas page by navigating to Settings -> Usage -> Usage Quotas.
  2. On the Usage Quotas page, click Set Usage Quotas. (If you have previously set the usage quota values, click Edit Usage Quotas.)
    The Setup modal appears. (If you have previously set the usage quota values, the modal is Usage Quotas.)
  1. In the Setup modal, click Set Usage Quotas.
    The Usage Quotas modal appears.
  2. In the Usage Quotas modal, enter the value for the Daily Usage Quota and the Monthly Usage Quota, and specify the unit (MB, GB, or TB).
    Leave unselected the check boxes for Discard logs when the Daily Usage Quota is reached and Discard logs when the Monthly Usage Quota is reached.
  3. Click Apply to close the modal box.
  4. In the Alert recipients area, specify the notification channels (email and Slack) and the recipients.
  5. In the Discard Rules area, click Add daily rule or Add monthly rule depending on the type of rule you want.
    The Create a Triggered Exclusion Rule modal displays.
  1. On the Apply this rule at: percentage selector, use your cursor to select and drag the marker to the percentage point you want. This percentage indicates at what percentage point of your pre-defined Daily/Monthly Usage Quota you want to start applying the triggered exclusion rule.
  2. Set the exclusion rules based on Sources, Apps, and Query. You can use one or more of these filters. Note that at least filter is required.
  3. Set the actions for the exclusion rules:
    a. Select Alert all alert recipients via Email and/or Slack to send notifications to the defined recipients when the exclusion rule is triggered. Note that at least one alert recipient is required.
    b. Select Discard matching log lines until midnight UTC (for daily rules) or Discard matching log lines until the start of the next monthly billing cycle (midnight UTC) (for monthly rules) if you want to immediately stop retaining logs as soon as the specified percentage is met.
    c. Select Preserve for Live Tail and Alerting if you want to stop retaining all logs as soon as the specified percentage is met, but you do want to still have ingested logs appear in live tail and be able to set alerts on them. Alternatively, if you know these logs are noise then you can elect to not have them appear in Live Tail nor send alerts.
  4. Click Apply to save your settings and close the modal.

Alerts Only: Configure a Usage Quota for Alerts Only

You can configure your Usage Quotas to simply send alert notifications via Slack and/or email when your Daily and Monthly Usage Quotas are met.

To configure this setting follow these steps:

  1. Open the Usage Quotas page by navigating to Settings -> Usage -> Usage Quotas.

  2. On the Usage Quotas page, click Set Usage Quotas. (If you have previously set the usage quota values, click Edit Usage Quotas.)
    The Setup modal appears. (If you have previously set the usage quota values, the modal is Usage Quotas.)

  3. In the Setup modal, click Set Usage Quotas.
    The Usage Quotas modal appears.

  4. In the Usage Quotas modal, enter the value for the Daily Usage Quota and the Monthly Usage Quota, and specify the unit (MB, GB, or TB).

  5. Leave unselected the check boxes for Discard logs when the Daily Usage Quota is reached and Discard logs when the Monthly Usage Quota is reached.

  6. Click Apply to close the modal box.

  7. On the Setup page, in the Alert recipients area, specify the notification channels (email and Slack) and the recipients.

  8. Do not add a Discard Rule (Daily or Monthly triggered exclusion rule). Optionally, you can also create a triggered exclusion rule, and receive an alert specifically based on the defined rule(s), in addition to the alert for when you reach the overall usage quota.

  9. To close the Usage Quotas page, click the X in the upper right.

Updated 6 months ago

Usage Quotas


Learn how to control how much data your teams are storing in order to create predictable spend month-over-month, while still viewing, alerting on, and retaining the critical data you need.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.