ElasticSearch – Error – “index read-only / allow delete (api)”

Background

Ran into an error with ElasticSearch.

I was trying to add a new record.

The error reads

index read-only / allow delete (api) ( Elastic Search – Error )

 

Trouble Shooting

Elastic\Elasticsearch\logs\elasticsearch.log

Reviewed elasticsearch.log and it is reading…

 

Text


rerouting shards: [high disk watermark exceeded on one or more nodes]

high disk watermark [90%] exceeded on
[C:\ProgramData\Elastic\Elasticsearch\data\nodes\0]
free: 12.1gb[5.1%], shards will be relocated away from this node

Image

Explanation

  1. Frequency
    • Every 30 seconds, new log entries are been posted.
  2. Alert
    • high disk watermark [90%] exceeded on
  3. Causation
    • free: 12.1gb[5.1%]
  4. Response
    • Shards will be relocated away from this node

 

Storage Utilization

Operating System ( OS )

Computer Management

Image

Explanation
  1. On our data drive, C:, here is what how utilization looks like
    • Reading
      • Capacity :- 235 GB
      • Free Space :- 12 GB
      • Free % :- 5%

 

Windows Explorer

Image

Explanation
  1. On our data drive, C:, here is what how utilization looks like
    • Reading
      • Capacity :- 235 GB
      • Free Space :- 12.1 GB
    • Deducted
      • Free Ratio :- 0.051
      • Percentile :- 5.1%

 

ElasticSearch

ElasticSearch – Metadata

_nodes/_local/stats/fs
Query
  1. Method :- GET
  2. Payload :- http://localhost:9200/_nodes/_local/stats/fs
Image

 

Explanation
  1. Metrics
    • Availed
      • “total_in_bytes”: 252806361088
      • “free_in_bytes”: 14050291712
      • “available_in_bytes”: 14050291712
    • Calculated
      • Equation :- available_in_bytes / total_in_bytes
      • Number :- 14050291712 / 252806361088
      • Ratio :- 0.0555772871043747
      • % :- 5%

 

Settings

Setting – ElasticSearch

_cluster/settings

Query
  1. Method :- GET
  2. Payload :-
    http://localhost:9200/_cluster/settings?include_defaults=true
Output
Image

Textual
  1. disk
    • threshold_enabled :- true
    • Watermark
      • low :- 85%
      • flood Stage :- 95%
      • High :- 90%

 

What does the numbers mean?

 

Item Definition Implication
cluster.routing.allocation.disk.threshold_enabled Defaults to true. Set to false to disable the disk allocation decider. Determine whether disk metrics are checked
cluster.routing.allocation.disk.watermark.low Controls the low watermark for disk usage. It defaults to 85%, meaning that Elasticsearch will not allocate shards to nodes that have more than 85% disk used. It can also be set to an absolute byte value (like 500mb) to prevent Elasticsearch from allocating shards if less than the specified amount of space is available. This setting has no effect on the primary shards of newly-created indices or, specifically, any shards that have never previously been allocated. New Shards will not be allocated
cluster.routing.allocation.disk.watermark.high Controls the high watermark. It defaults to 90%, meaning that Elasticsearch will attempt to relocate shards away from a node whose disk usage is above 90%. It can also be set to an absolute byte value (similarly to the low watermark) to relocate shards away from a node if it has less than the specified amount of free space. This setting affects the allocation of all shards, whether previously allocated or not. Attempts will be made to move Current Shards
cluster.routing.allocation.disk.watermark.flood_stage Controls the flood stage watermark. It defaults to 95%, meaning that Elasticsearch enforces a read-only index block (index.blocks.read_only_allow_delete) on every index that has one or more shards allocated on the node that has at least one disk exceeding the flood stage. This is a last resort to prevent nodes from running out of disk space. The index block must be released manually once there is enough disk space available to allow indexing operations to continue. Index will marked readonly

Remediation

Outline

  1. File System
    • Make room in File-system

Options Explored

File System

Use OS Tools to make room in File System

Linux

In Linux, use du ( Disk used), df ( Disk free), rm ( remove files ), mv ( move files, and ls ( List files).

 

ElasticSearch

Review default or current disk utilization alert settings and judge whether ripe for changes.

Configuration Setting

C:\ProgramData\Elastic\Elasticsearch\config
Syntax
cluster.routing.allocation.disk.watermark.low: [ratio]
cluster.routing.allocation.disk.watermark.high: [ratio]
cluster.routing.allocation.disk.watermark.flood_stage: [ratio]
Sample
cluster.routing.allocation.disk.watermark.low: 0.99
cluster.routing.allocation.disk.watermark.high: 0.99
cluster.routing.allocation.disk.watermark.flood_stage: 0.99

Index

Index Setting – Read
Syntax
 
http://localhost:9200/index/_settings
Sample
http://localhost:9200/bible/_settings

 

Image

 

 

Index Setting – Set
Syntax
 
{
  
    "index": 
    {
        "blocks": 
        {
              "read_only_allow_delete": "false"
            , "read_only": "false"          
        }
    }
    
}

 

Sample

 

 
{
 
    "index": 
    {
        "blocks": 
        {
              "read_only_allow_delete": "false"
            , "read_only": "false"          
        }
    }
    
}
Image
  1. Method :- PUT
  2. Payload :-
     http://localhost:9200/bible/_settings

 

 

Summary

Quite a bit to go over.

In summary, monitor your disk usage.

If need to, adjust ElasticSearch to scale back by modifying the percentile or hard numbers it responds and starts making accommodation on.

 

References

  1. elastic.co
    • Cluster
      • Disk-based Shard Allocation
        • Elasticsearch Reference [6.0] » Modules » Cluster » Disk-based Shard Allocation
          Link
    • Indices
      • Read
        • Indices – Setting – Read
          Link
      • Update
        • Update Indices Settings
          Link
      • Indices – API – Get Settings
        • Elasticsearch Reference [6.3] » Indices APIs » Get Settings
          « Update Indices Settings
          Link
    • Node
      • Node Stats
        • Elasticsearch Reference [6.3] » Cluster APIs » Nodes Stats « Cluster Update Settings
          Link
    • Issues
      • How to show cluster settings “default” values ? #18645
        Link
      • cluster.routing.allocation.disk.watermark.flood_stage not updateable through cluster api #27406
        Link
  2. Stack-overflow
    • Disk Space Usage
      • ElasticSearch find disk space usage
        Link
    • High disk watermark exceeded even when there is not much data in my index
      Link