ES的索引每日都在增加的煩惱


elasticsearch本身有提供API去讓你處理index 當然我們可以自己去寫一個簡單的小程序去每日處理那些沒有用的index (e.g. 服務器日誌) 但是 Don’t reinvent the wheel Don’t reinvent the wheel 因為ES就有提供一個簡單方便的程序去做這件事 https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/fe_source.html

只要進行簡單的設定及以cronjob運行就可以了(我是用k8s中的cron job去運行,更加簡單) 設定config.yml (怎樣連接去es)

client:
      hosts:
        - http://your-es.host.com
      port: 9200
      url_prefix:
      use_ssl: False
      certificate:
      client_cert:
      client_key:
      ssl_no_validate: False
      http_auth:
      timeout: 30
      master_only: False
    logging:
      loglevel: INFO
      logfile:
      logformat: default
      blacklist: ['elasticsearch', 'urllib3']

  設定action_file.yml (決定哪一個index需要刪除)

actions:
      1:
        action: delete_indices
        description: "Clean up ES by deleting old indices"
        options:
          continue_if_exception: False
          disable_action: False
        filters:
        - filtertype: pattern
          kind: regex
          value: '^(prod-|stag-|dev-)kubernetes.*$'
        - filtertype: age
          source: name
          direction: older
          timestring: '%Y-%m-%h'
          unit: days
          unit_count: 30
          exclude: False

注意filters中間的條件是AND 而不是OR的 這邊的設定簡單來說就是只要index的名字是stag/dev/prod而且含有kubernetes字元,產生日期是30天前就會刪掉