ElasticSearch – cURL – Batch Operations – Day 1

Background

In earlier posts we exclusively used Postman to perform Data Entry against Elastic Search.

Batch

Preface

In this post we will discuss using cURL, a command line client tool.

What is cURL?

Wikipedia

Link

cURL is a computer software project providing a library and command-line tool for transferring data using various protocols. The cURL project produces two products, libcurl and cURL. It was first released in 1997. The name stands for “Client URL”.

Get cURL

Please get cURL from here.

Please choose the binary specific to your OS, download and extract it.

No need to install, just reference the folder you extracted it into.

 

Use Scenario

Remove Data

Remove Data – Based on Query

Overview

  1. Command File
    • ElasticSearch
      • URL
        • Index :- bible
        • Type :- biblekjv
        • Method :- _delete_by_query
    • cURL
      • Content-Type: application/json
  2. JSON File
    • Query
      • Match on
        • book=Revelation

Command File


REM Retrieving a document
rem https://www.elastic.co/guide/en/elasticsearch/guide/current/get-doc.html

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

REM ElasticSearch - URL
set "_url=http://localhost:9200"

REM ElasticSearch - Index,Type,Operation
set "_esindex=bible"
set "_estype=biblekjv"
set "_esoperation=_delete_by_query"
set "_espath=/%_esindex%/%_estype%/%_esoperation%"

REM ElasticSearch - Jsonfile
set "_jsonfile=jsonfile\deleteByQuery_book_Revelation.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"

REM HTTP - ContentType
set "_contentType=Content-Type: application/json"

%_bin% -i -H "%_contentType%"  %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal 

JSON File

{

  "query":
  {
    "bool":
    {
      "must":
      [
        {
            "match":
            {
                "book":"Revelation"
            }
        }
      ]
    }
  }

}

Output

deleteUsingQuery_20180803_0330PM

Remove Data – Based on Bulk

Overview

  1. Command File
    • ElasticSearch
      • URL
        • Index :- bible
        • Type :- biblekjv
        • Method :- _bulk
    • cURL
      • Content-Type: application/x-ndjson
  2. JSON File
    • Content
      • {“delete”:{“_id”:1000}}
        • Remove record whose ID is 1000

Command File


setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=bible"
set "_estype=biblekjv"
set "_esoperation=_bulk"
set "_espath=/%_esindex%/%_estype%/%_esoperation%"

set "_jsonfile=jsonldfile\deleteBatch.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"
set "_contentType=Content-Type: application/x-ndjson"

%_bin% -i -H "%_contentType%" %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal

Json file ( line delimited)


{"delete":{"_id":1000}}

Output

deleteUsingBatch_20180803_0351PM.png

Add Data

Add Batch Data based on JSON file (line delimited)

Overview

  1. Command File
    • ElasticSearch
      • URL
        • Index :- bible
        • Type :- biblekjv
        • Method :- _bulk
    • cURL
      • Content-Type: application/x-ndjson
  2. JSON File
    • Content
      • {“index”:{“_id”:1001}}
        {“bookID”:66,”book”:”Revelation”,”chapterID”:1,”verseID”:1,”passage”:”Revelation 1:1″,”verse”:”The Revelation of Jesus Christ, which God gave unto him, to shew unto his servants things which must shortly come to pass; and he sent and signified it by his angel unto his servant John:”}

        • Record – 1
          • Index ID
        • Record – 2
          • Column Value pair
          • Comma delimited
        • Record – N
          • Empty line
          • Make sure not ended with LF nor CR/LF
Command File

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=bible"
set "_estype=biblekjv"
set "_esoperation=_bulk"
set "_espath=/%_esindex%/%_estype%/%_esoperation%"

set "_jsonfile=jsonldfile\revelation.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"

set "_contentType=Content-Type: application/x-ndjson"

%_bin% -i -H "%_contentType%"  %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal 

XML File

{"index":{"_id":1001}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":1,"passage":"Revelation 1:1","verse":"The Revelation of Jesus Christ, which God gave unto him, to shew unto his servants things which must shortly come to pass; and he sent and signified it by his angel unto his servant John:"}
{"index":{"_id":1002}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":2,"passage":"Revelation 1:2","verse":"2Who bare record of the word of God, and of the testimony of Jesus Christ, and of all things that he saw."}
{"index":{"_id":1003}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":3,"passage":"Revelation 1:3","verse":"3Blessed is he that readeth, and they that hear the words of this prophecy, and keep those things which are written therein: for the time is at hand."}
{"index":{"_id":1004}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":4,"passage":"Revelation 1:4","verse":"4John to the seven churches which are in Asia: Grace be unto you, and peace, from him which is, and which was, and which is to come; and from the seven Spirits which are before his throne; "}
{"index":{"_id":1005}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":5"passage":"Revelation 1:5","verse":"5And from Jesus Christ, who is the faithful witness, and the first begotten of the dead, and the prince of the kings of the earth. Unto him that loved us, and washed us from our sins in his own blood, "}
{"index":{"_id":1006}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":6,"passage":"Revelation 1:6","verse":"6And hath made us kings and priests unto God and his Father; to him be glory and dominion for ever and ever. Amen"}
{"index":{"_id":1007}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":7,"passage":"Revelation 1:7","verse":"7Behold, he cometh with clouds; and every eye shall see him, and they also which pierced him: and all kindreds of the earth shall wail because of him. Even so, Amen."}
{"index":{"_id":1008}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":8,"passage":"Revelation 1:8","verse":"8I am Alpha and Omega, the beginning and the ending, saith the Lord, which is, and which was, and which is to come, the Almighty."}

XML File – Image

revelation_20180804_0808AM.png

Output

addUsingBatch_20180803_0403PM

Review Data

Review Data based on JSON file

Command File

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=bible"
set "_estype=biblekjv"
set "_esoperation=_search"
set "_espath=/%_esindex%/%_estype%/%_esoperation%"

set "_jsonfile=jsonfile\search_book_Revelation.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"

set "_contentType=Content-Type: application/json"

%_bin% -i -H "%_contentType%"  %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal

XML File

{
  "query":
  {
    "bool":
    {
      "must":
      [
        { "match":
             { "book": "Revelation" }
        }
      ]
    }
  }
  , "sort" :
  {
      "bookID": {"order": "asc"}
    , "chapterID": {"order": "asc"}
    , "verseID": {"order": "asc"}
  }
} 

Output

fetchUsingQuery_20180803_0425PM

Source Code Control

Git

DanielAdeniji/elasticSearchcURLBatch

Link

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s