Elasticsearch housekeeping

Archive note, 8 October 2026. This Python 2 example uses legacy Elasticsearch _status and _optimize APIs and is not validated for current versions. It issues destructive DELETE requests. Do not run it as a current maintenance recipe: first produce a read-only target list, verify an exact naming/retention policy, and confirm recoverable backups. The URL-formatting typo is corrected below, not a claim that the script is safe or maintained.

Logstash is very cool but the underlying Elasticsearch engine can take up a lot of space, so I wrote a small cleaning up script that runs daily to either discard older than 30 days data or optimize active tables.

#!/usr/bin/python

import pycurl
import json
import StringIO
from datetime import datetime, timedelta

retentionDays = 30

c = pycurl.Curl()
b = StringIO.StringIO()

c.setopt(c.URL, 'http://127.0.0.1:9200/_status')
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.perform()

blob = json.loads( b.getvalue() )

for index in blob['indices']:
 if 'logstash' in index:
  old = datetime.now() - timedelta(days = retentionDays)
  indexDate = datetime.strptime(index, "logstash-%Y.%m.%d")
  if old > indexDate:
   print "delete", index
   c.setopt(pycurl.CUSTOMREQUEST, "DELETE")
   c.setopt(c.URL, ('http://127.0.0.1:9200/{}').format(index))
   c.perform()
  else:
   print "optimize", index
   c.setopt(c.URL, ('http://127.0.0.1:9200/{}/_optimize').format(index))
   c.perform()

Turns out there is a much better tool to do all Elasticsearch related housekeeping called curator but anyway sometimes it's nice to make your own scripts :-)