$25,000 a Day for Data Nobody Needed

The monthly Azure bill jumped from $250k to $1M. Same users. Same workloads. Nobody could say why.

The background was ordinary. An on-premises Hadoop platform had been lifted and shifted to Azure HDInsight. The application was slow, so storage moved to the premium tier on zone-redundant storage with active sync. Sensible calls, every one.

But blob versioning was on, and nothing deleted the old versions. A new version of the data landed every single day. 8 TB quietly became 1 petabyte. No alert flagged it.

Every 24 hours without a delete rule cost about $25,000.

How it happened

Blob versioning is a useful feature. When it's on, Azure keeps the previous version of a blob each time it's overwritten or deleted, so you can recover from mistakes.

What versioning doesn't do is clean up after itself. Previous versions stay until something removes them, and they're stored and billed like any other data.

Three settings stacked up:

• Versioning was on. Each daily load wrote a new version and kept the old one.
• No lifecycle rule for previous versions. Nothing was set to delete them after a period.
• Premium, zone-redundant storage. Every retained version sat on the premium tier, replicated across availability zones.

None of these was wrong on its own. Together, with a daily data load, they compounded quietly. The workload didn't change. The amount of data being kept did.

How it was fixed

• An inventory of every subscription and resource, sorted by cost. One storage account stood out.
• A lifecycle policy to delete previous versions after a set period.
• An Event Grid event on every deletion, to confirm the policy was working.
• A ticket raised automatically on each deletion, so there was a record.

How to check your own estate

You don't need a cost spike to go looking. These checks are read-only until the last two steps.

1. Find your most expensive storage accounts

In the Azure portal, open Cost Management and go to Cost analysis. Set the scope as high as you can, such as a management group or billing account, so every subscription is included. Use the Cost by resource view, filter Service name to Storage, and sort by cost.

Then look at the trend for the top few accounts over the last several months. Steady month-on-month growth with no matching change in the workload is the pattern to look for.

2. Check whether versioning is on

For each account on that list, open the storage account and go to Data management > Data protection. Look for Enable versioning for blobs. You can also check from the Azure CLI:

az storage account blob-service-properties show \
  --account-name <account> --resource-group <resource-group> \
  --query isVersioningEnabled

If versioning is on, check whether a lifecycle rule deals with previous versions. If there isn't one, old versions are being kept indefinitely.

3. Add a lifecycle rule for previous versions

Lifecycle rules live under Data management > Lifecycle management on the storage account. A rule that deletes previous versions a set number of days after they were created looks like this:

{
  "rules": [
    {
      "enabled": true,
      "name": "delete-previous-versions",
      "type": "Lifecycle",
      "definition": {
        "actions": {
          "version": {
            "delete": { "daysAfterCreationGreaterThan": 30 }
          }
        },
        "filters": {
          "blobTypes": [ "blockBlob" ]
        }
      }
    }
  ]
}

The 30 is only an example. Pick a period that matches how far back you genuinely need to recover, and add a prefix filter if the rule should cover only some containers. Agree the period with whoever owns the data first: once a version is removed, it no longer protects you. Lifecycle policies don't act instantly, so allow time for the first run before judging whether it worked.

4. Confirm the deletions with Event Grid

A rule you can't see working is a rule nobody trusts. On the storage account, open Events and create an Event Grid subscription for Blob Deleted events. Send them somewhere that leaves a record, such as a Logic App or Azure Function that logs the event or raises a ticket in your service desk.

Event Grid also publishes a Lifecycle Policy Completed event when a policy run finishes. It's a simple daily signal that the rule actually ran.

5. Make sure something would have told you

No alert flagged this one. Set a budget with alerts on each subscription in Cost Management, and turn on cost anomaly alerts, so the next unexpected climb reaches a person before it reaches the invoice.

The lesson

Big spikes rarely come from more usage. They come from a setting nobody went back to check.

Compute gets watched. Teams right-size virtual machines, buy reservations and switch things off overnight. Storage just grows. It rarely breaks anything, so nobody looks at it until the bill does.

If you haven't looked at your storage accounts by cost recently, start there.