On MS Windows, Profiling Java Apps Memory Utilization Using VMMap

Background

In our last post, we covered the usage of a few Oracle standard tools to profile Java Applications.

BTW the referenced post is available here.

 

SysInternals

VMMap

In this post, we will briefly cover using SysInternal’s VMMap GUI to track down memory utilization.

 

Artifact

VMMap is available from here.

The current version is v3.21 and it was released a couple of years ago; July 20, 2015 to be exact.

Download & Extract

Download and Extract it; no need for installation.

Usage

ElasticSearch

Select or Launch Process

 

Metrics

 

WebSphere

Select or Launch Process

 

Metric

Logstash – Configuration – Field Names – document_id

Background

In Logstash, got bogged down trying to use the document_id Configuration Option.

Reproduction

logstash

Configuration

Image

configuration_file_20180809_0820AM.PNG

ElasticSearch

Query

Image

query_stackoverflowUser_20180809_0825AM.PNG

Data

Image

stackoverflowUser_output_20180809_0830AM.PNG

 

Troubleshooting

ElasticSearch

Data Review

  1. Hits
    • Number of Hits
      • We added 10 records
        • Data Source Query says “select top 10 *
      • But,  hits/total
      • Reads  1
    • Lone Hit
      • Data
        • _id
          • “_id”: “%{[Id]}”
        • field
          • id
            • “id”: 10

Logstash

Output

Image

output_20180809_0851AM.PNG

Textual


{"id":-1,"lastaccessdate":"2008-08-26T07:16:53.810Z","reputation":1,"accountid":-1,"location":"on the server farm","downvotes":890820,"displayname":"Community","@timestamp":"2018-08-09T13:48:09.088Z","age":null,"emailhash":null,"views":649,"@version":"1","creationdate":"2008-07-31T07:00:00.000Z","aboutme":"Hi, I'm not really a person.

\n\nI'm a background process that helps keep this site clean!

\n\nI do things like

\n\n
<ul>\n
	<li>Randomly poke old unanswered questions every hour so they get some attention</li>
\n
	<li>Own community questions and answers so nobody gets unnecessary reputation from them</li>
\n
	<li>Own downvotes on spam/evil posts that get permanently deleted</li>
\n
	<li>Own suggested edits from anonymous users</li>
\n
	<li><a href="http://meta.stackexchange.com/a/92006">Remove abandoned questions</a></li>
\n</ul>
\n","websiteurl":"http://meta.stackexchange.com/","upvotes":225495}

SQL Server

Diagram

dbo.Users

Image

stackOverflow2010.Users.PNG

Explanation
  1. Column name is Id

Resolution

In Logstash, Field names are lower cased

Logstash

Configuration

Image

configuration_file_20180809_0903AM.PNG

ElasticSearch

Query

Query Output

Image

output_20180809_0908AM.PNG

Explanation
  1. hits = 10

References

  1. Logstash
    • Configuring
      • Logstash Reference [6.3] » Configuring Logstash » Accessing Event Data and Fields in the Configuration
        Link
    • Transforming Data
      • Logstash Reference [6.3] » Transforming Data » Extracting Fields and Wrangling Data
        Link
    • Configuring Logstash
      • Logstash Reference [6.3] » Configuring Logstash » Structure of a Config File
        Link
    • Output
      • Logstash Reference [6.3] » Output plugins » Elasticsearch output plugin
        Link
    • Filter
      • Logstash Reference [6.2] » Filter plugins » Mutate filter plugin
        Link

 

 

 

 

Logstash – Error – ” Unrecognized VM option ‘UseParNewGC’ “

Background

During my initial evaluation of Logstash ran into an easy to address error.

Reproduce

Invoke

The invocation is straight forward


set "_binfolder=C:\Downloads\Elastic\Logstash\v6.3.2\extract\bin"
set "_configuration=stackOverflow2010.User.conf"

call %_binfolder%\logstashImpl.bat -f %_configuration%

Output

Image

UseParNewGC_20180802_0334PM

Textual


Unrecognized VM option 'UseParNewGC'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Troubleshooting

Java

Java Version

Outline

We can request the version Number for the java runtime executable, java.exe.

Script


java -version

Output

Image

java_version_20180808_0332PM

Textual


java version "10.0.2" 2018-07-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.2+13)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.2+13, mixed mode)

Knowledge Base ( KB)

Googled on the the term and got a good hit

  1. [META] Java 10 Support #9345
    • Opened By :- andrewvc
    • Date Opened :- 2018-April-9th
    • Link
      Link
    • Resolution
      • Resolution #1
        • By :- Armin Braun ( original-brownbear )
        • Web Links
        • Workaround :- Currently there is no temporary fix sorry. Switching to Java 8 is the only stable solution for the time being.
      • Resolution #2
        • By :- Josh Daone ( JoshDaone )
        • Web Links
        • Workaround :- @original-brownbear thanks! Switched back to openjdk-8-jre and working properly.

Resolution

Target Specific Java Version

Objective

Rather than rely on Java that is referenced in our path, we will set the environment variable JAVA_HOME to a pre-v10 Version.

Review Installed Java

Launched Windows explorer and access the “C:\Program Files\Java” folder.

Review Java Folders

java_explorer_20180808_0424PM

Code


set "JAVA_HOME=C:\Program Files\Java\jdk1.8.0_181"

Additional Reading

  1. Andy Luis
    • mpvjava
      • JDK 9 Migration : 5 point checklist for Garbage Collection
        Link
      • About
        Link
  2.  OpenJDK
    • JEP 214: Remove GC Combinations Deprecated in JDK 8
      Link
  3. ORACLE
    • JDK 9 Release Notes – Removed APIs, Features, and Options
      Link

 

Dedicated

Dedicated to Armin Braun ( original-brownbear )
Less Talking … More Coding …

References

  1. Elastic
    • elastic/logstash
      [META] Java 10 Support #9345
      Link

ElasticSearch – Logstash – SQL Server

Background

Elastic’s Logstash is an ETL tool that allows us to “Request, Collect, Parse, and Send” Data.

Logstash

Definition

Elastic

Here is Elastic’s own definition

Link

definition_logstash_20180808_0125PM

Artifact

Logstash is available here.

The current version is 6.3.2 and it is a very recent release ( 2018-July-24th).

artifact_6_3_2

Download

For MS Windows, please choose the ZIP Version.

Extract

Please extract the compressed file.

JDBC Driver

Microsoft SQL Server

Download

Please download Microsoft SQL Server JDBC Driver from here.

Extract

Extract the downloaded file.

Usage

Files

Configuration File

Outline

  1. Input
    • jdbc
      • Connection String
        • Syntax :- jdbc:sqlserver://[host]:[port-number]
        • Sample :- jdbc:sqlserver://localhost:1433
      • jdbc_user :- database user
      • jdbc_password :- database user passsword
      • jdbc_driver_library :- full file name of JDBC Driver
      • jdbc_driver_class
        • com.microsoft.sqlserver.jdbc.SQLServerDriver
      • statement
        • select top 10 * from [StackOverflow2010].[dbo].[Users] tblU order by tblU.[Id] asc
  2. Output
    • elasticSearch
      • hosts
        • Syntax :- Elastic Search host and Port
        • Sample :- localhost:9200
      • Index
        • Sample :- stackoveflow2010user
      • Document Type
        • Sample :- _doc
      • document_id
        • Syntax :- %{[column-name]}
        • Sample :- %{[id]}

Configuration

input {
  jdbc {
    jdbc_connection_string => "jdbc:sqlserver://localhost:1433"
    # The user we wish to execute our statement as
    jdbc_user => "stackoverflow"
    jdbc_password => "hIy8jA2lNl"
    # The path to our downloaded jdbc driver
    jdbc_driver_library => "C:\Downloads\Microsoft\Java\jdbc\v6.0.8112.200\extract\sqljdbc_6.0\enu\jre8\sqljdbc42.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    # our query
    statement => "select * from [StackOverflow2010].[dbo].[Users] tblU order by tblU.[Id] asc"
    }
  }
output {
  stdout { codec => json_lines }
  elasticsearch {
  "hosts" => "localhost:9200"
  "index" => "stackoveflow2010user"
  "document_type" => "_doc"
  document_id => "%{[id]}"
  }
}

Command File


setlocal

REM 2018-08-08 11:16 AM Daniel Adeniji ( dadeniji)
REM SET JAVA_HOME to Version 1.8
set "JAVA_HOME=C:\Program Files\Java\jdk1.8.0_181"

set "_binfolder=C:\Downloads\Elastic\Logstash\v6.3.2\extract\bin"
set "_configuration=stackOverflow2010.User.conf"

call %_binfolder%\logstashImpl.bat -f %_configuration%

endlocal

Processing

Script


stackOverflow2010.User.cmd

Output

Output – 01

processing_20180808_0132PM

Output – 02

processing_20180808_0134PM

Output – 03

processing_20180808_0136PM

Output – 04

processing_20180802_011PM

 

Validation

Tools

Postman

Queries

Query – Microsoft

Objective

Find matches for Microsoft

Query


http://localhost:9200/stackoveflow2010user/_doc/_search?q=Microsoft

Design

query_Microsoft_20180808_0117PM

Output

Microsoft_Result_20180808_0118PM

 

References

  1. Elastic
    • Docs
      • Logstash
        • Running Logstash from the Command Line
          Link
    • Blog
      • Suyog Rao
        • Little Logstash Lessons: Handling Duplicates
          Link
  2. StackOverflow
    • Logstash to Keep Two Databases Synced – Cannot Access %{document_id}
      Link
  3. QBox
    • Vineeth Mohan
      • Migrating MySql Data Into Elasticsearch Using Logstash
        Link

 

ElasticSearch – Type – Mapping

Background

It is very easy to create new types, tables in traditional tables, and corresponding columns in ElasticSearch.

ElasticSearch

If we do not create mappings before hand, ElasticSearch will create the type, columns and column types for us.

Schema-Less

The ElasticSearch exercises we have tackled thus far relies on the ability of ElasticSeach to create columns upon usage; the first time a record is added.

biblekjv

Here is the schema created for our biblekjv example.

Image

biblekjv_mapping_20180806_1024PM_01

Textual

  1. Columns
      • book
        • type is text
        • And, type is keyword
    • bookID
      • type is long
    • chapter
      • type is long
    • chapterID
      • type is long
    • passage
      • type is text
      • And, type is keyword

Explanation

  1. Numeric entries
    • Are codified as long
    • The largest available datatype for numbers
  2. Strings
    • Represented as both text and keyword
    • Datatype
      • Text
        • Tokenizable
      • Keyword
        • Exact Matches

Schema

Definition

Definition – ElasticSearch

Link

Mapping is the process of defining how a document, and the fields it contains, are stored and indexed. For instance, use mappings to define:

  • which string fields should be treated as full text fields.
  • which fields contain numbers, dates, or geolocations.
  • whether the values of all fields in the document should be indexed into the catch-all _all field.
  • the format of date values.
  • custom rules to control the mapping for dynamically added fields.

Sample – event

Goal

In this post we will create a schema to underine an event.

Sample

Sample – Periscope

Single Event Tables and Common Analysis Queries
Link

event_periscope_20180807_0238PM

Code

Create Type
Outline
  1. We will have two files, the JSON file and the command file
  2. The JSON file
    • Mapping
      • Top – mappings
      • Type :- _doc
      • Properties
        • Property
          • userID :- long
          • loginFirst :- date
          • loginLast :- date
          • firstSpendAt :- date
          • totalSpend :- double
          • dob :- date
          • gender :- keyword ( m/f/etc)  {Gender Inclusive}
          • platform :- keyword ( ios/android)
json

{
    "mappings":
    {
        "_doc":
        {
            "properties":
            {

                "userID":
                {
                    "type": "long"
                }

                , "loginFirst":
                {
                    "type": "date"
                }               

                , "loginLast":
                {
                    "type": "date"
                }                               

                , "firstSpendAt":
                {
                    "type": "date"
                }                                               

                , "totalSpend":
                {
                    "type": "double"
                }                   

                , "dob":
                {
                    "type": "date"
                }   

                , "gender":
                {
                    "type": "keyword"
                }   

                , "platform":
                {
                    "type": "keyword"
                }
            } // "properties"

        } // mapping name

    } //mappings

}  

Batch file

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=/event"
set "_estype="

set "_espath=%_esindex%%_estype%"

set "_contentType=Content-Type: application/json

Rem Index - Delete

%_bin% -XDELETE -i %_url%%_espath% 

Rem Index / Mapping - Create

set "_jsonfile=jsonfile\event_mapping.json"

set "_jsonfileFQDN=%cd%\%_jsonfile%"

%_bin% -i  -XPUT %_url%%_espath%  -H "%_contentType%" --data-binary @%_jsonfileFQDN%

endlocal

Output
Output – Delete

type_create_delete_20180807_0253PM.png

Output – Create

typeMappingCreate_20180807_0253PM

Display Type
Outline
  1. We will have a lone file, the command file
  2. Command file
  3. Payload
Batch file

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=/event"
set "_estype="
set "_esoperator=/_mapping"

Rem Index / Mapping - List

set "_espath=%_esindex%%_estype%%_esoperator%?pretty"

%_bin% -i  %_url%%_espath% -H "%_contentType%" 

Output
Output – Browse

typeMappingBrowse_20180807_0320PM

Load Data
Outline
    1. There are two files, the JSON file and the command file
    2. The JSON file
      • Two lines for each record
        • The first record will contain the _id
        • The second record sill contain the actual data
          • Property/Value pair
    3. Command file
      • Syntax
        • curl.exe -i -H “Content-Type: application/x-ndjsonhttp://%5Bserver}:{port}/{index}/{type}/_bulk?pretty –data-binary @[filename]
      • Sample
JSON

{"index":{"_id":1}}
{"userID":137314,"loginFirst":"2015-04-01","loginLast":"2015-04-05","firstSpendAt":"2015-04-03","totalSpend":50,"dob":"1994-01-01","gender":"f","platform":"ios"}
{"index":{"_id":2}}
{"userID":137312,"loginFirst":"2015-04-01","loginLast":"2015-04-02","firstSpendAt":"2015-04-02","totalSpend":2500,"dob":"1987-01-01","gender":"m","platform":"android"}
{"index":{"_id":3}}
{"userID":137310,"loginFirst":"2015-04-01","loginLast":"2015-04-01","dob":"2000-01-01","gender":"m","platform":"ios"}
{"index":{"_id":4}}
{"userID":137311,"loginFirst":"2015-04-01","loginLast":"2015-04-02","dob":"1995-01-01","gender":"f","platform":"ios"}
{"index":{"_id":5}}
{"userID":137313,"loginFirst":"2015-04-01","loginLast":"2015-04-04","dob":"1976-01-01","gender":"m","platform":"andriod"}

Command

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=event"
set "_estype=_doc"
set "_esoperation=_bulk"

set "_espath=/%_esindex%/%_estype%/%_esoperation%"

set "_jsonfile=jsonldfile\event.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"

set "_contentType=Content-Type: application/x-ndjson"

%_bin% -i -H "%_contentType%"  %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal
Browse Data
Outline
  1. There are two files the json file and command file
  2. Payload
    • Syntax :-
      • curl.exe -i -H “Content-Type: application/json” http://{server}:{port}/{type}/_search?pretty –data-binary @{filename}
    • Sample :-
JSON

{
   "sort" :
  {
      "_id": {"order": "asc"}

  }
}

Command

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=event"
set "_estype=_doc"
set "_esoperation=_search"
set "_espath=/%_esindex%/%_estype%/%_esoperation%"

set "_jsonfile=jsonfile\event_data_browse.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"

set "_contentType=Content-Type: application/json"

%_bin% -i -H "%_contentType%"  %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal
Output

dataBrowse_20180807_0357PM

Source Control

GitHub

DanielAdeniji/elasticSearchMappingEvent
Link

References

  1. Elastic
    • Docs
      • Mapping
        • Definition
        • Field Datatypes
          • Elasticsearch Reference [6.3] » Mapping » Field datatypes
            Link
          • Elasticsearch Reference [6.3] » Mapping » Field datatypes »Data datatypes
            Link
          • Elasticsearch Reference [6.3] » Mapping » Field datatypes » Numeric datatypes
            Link
          • Elasticsearch Reference [6.3] » Mapping » Field datatypes » Boolean datatype
            Link
        • Indices Mapping
          • Get Mapping
            • Indices Get Mapping
              Link
          • Put Mapping
            • Elasticsearch Reference [5.0] » Indices APIs » Put Mapping
              Link
      • Indices
        • Delete Index
          • Elasticsearch Reference [2.3] » Indices APIs » Delete Index
          • Elasticsearch Reference [6.3] » Indices APIs » Delete Index
      • Reference
      • Elastic for Apache Hadoop
        • Elasticsearch for Apache Hadoop [6.3] » Elasticsearch for Apache Hadoop » Mapping and Types
          « Apache Storm support
          Link
    • Github
      • elastic/elasticsearch
        • INDEX DELETE with wildcard doesn’t delete all matching indexes #7295
          Link
        • v5.3.1 Entire Bulk request is rejected if containing a single invalid index request #24319
          Link
    • Discuss
      • Index a new document/35281
        Link
  2. Periscope
    • Single Event Tables and Common Analysis Queries
      Link
  3. GitHubGist
    • kuznero
      • kuznero/elasticsearch-curl.md
        Link
  4. StackOverflow
    • curl command to close all indices at once in elastic search
      Link
    • Reindex ElasticSearch index returns “Incorrect HTTP method for uri [/_reindex] and method [GET], allowed: [POST]”
      Link
    • Elasticsearch : Root mapping definition has unsupported parameters index : not_analyzed
      Link
  5. logz.io
    • Daniel Berman
      • ElasticSearch Mapping
        Link

 

ElasticSearch – cURL – Batch Operations – Day 1

Background

In earlier posts we exclusively used Postman to perform Data Entry against Elastic Search.

Batch

Preface

In this post we will discuss using cURL, a command line client tool.

What is cURL?

Wikipedia

Link

cURL is a computer software project providing a library and command-line tool for transferring data using various protocols. The cURL project produces two products, libcurl and cURL. It was first released in 1997. The name stands for “Client URL”.

Get cURL

Please get cURL from here.

Please choose the binary specific to your OS, download and extract it.

No need to install, just reference the folder you extracted it into.

 

Use Scenario

Remove Data

Remove Data – Based on Query

Overview

  1. Command File
    • ElasticSearch
      • URL
        • Index :- bible
        • Type :- biblekjv
        • Method :- _delete_by_query
    • cURL
      • Content-Type: application/json
  2. JSON File
    • Query
      • Match on
        • book=Revelation

Command File


REM Retrieving a document
rem https://www.elastic.co/guide/en/elasticsearch/guide/current/get-doc.html

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

REM ElasticSearch - URL
set "_url=http://localhost:9200"

REM ElasticSearch - Index,Type,Operation
set "_esindex=bible"
set "_estype=biblekjv"
set "_esoperation=_delete_by_query"
set "_espath=/%_esindex%/%_estype%/%_esoperation%"

REM ElasticSearch - Jsonfile
set "_jsonfile=jsonfile\deleteByQuery_book_Revelation.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"

REM HTTP - ContentType
set "_contentType=Content-Type: application/json"

%_bin% -i -H "%_contentType%"  %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal 

JSON File

{

  "query":
  {
    "bool":
    {
      "must":
      [
        {
            "match":
            {
                "book":"Revelation"
            }
        }
      ]
    }
  }

}

Output

deleteUsingQuery_20180803_0330PM

Remove Data – Based on Bulk

Overview

  1. Command File
    • ElasticSearch
      • URL
        • Index :- bible
        • Type :- biblekjv
        • Method :- _bulk
    • cURL
      • Content-Type: application/x-ndjson
  2. JSON File
    • Content
      • {“delete”:{“_id”:1000}}
        • Remove record whose ID is 1000

Command File


setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=bible"
set "_estype=biblekjv"
set "_esoperation=_bulk"
set "_espath=/%_esindex%/%_estype%/%_esoperation%"

set "_jsonfile=jsonldfile\deleteBatch.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"
set "_contentType=Content-Type: application/x-ndjson"

%_bin% -i -H "%_contentType%" %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal

Json file ( line delimited)


{"delete":{"_id":1000}}

Output

deleteUsingBatch_20180803_0351PM.png

Add Data

Add Batch Data based on JSON file (line delimited)

Overview

  1. Command File
    • ElasticSearch
      • URL
        • Index :- bible
        • Type :- biblekjv
        • Method :- _bulk
    • cURL
      • Content-Type: application/x-ndjson
  2. JSON File
    • Content
      • {“index”:{“_id”:1001}}
        {“bookID”:66,”book”:”Revelation”,”chapterID”:1,”verseID”:1,”passage”:”Revelation 1:1″,”verse”:”The Revelation of Jesus Christ, which God gave unto him, to shew unto his servants things which must shortly come to pass; and he sent and signified it by his angel unto his servant John:”}

        • Record – 1
          • Index ID
        • Record – 2
          • Column Value pair
          • Comma delimited
        • Record – N
          • Empty line
          • Make sure not ended with LF nor CR/LF
Command File

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=bible"
set "_estype=biblekjv"
set "_esoperation=_bulk"
set "_espath=/%_esindex%/%_estype%/%_esoperation%"

set "_jsonfile=jsonldfile\revelation.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"

set "_contentType=Content-Type: application/x-ndjson"

%_bin% -i -H "%_contentType%"  %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal 

XML File

{"index":{"_id":1001}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":1,"passage":"Revelation 1:1","verse":"The Revelation of Jesus Christ, which God gave unto him, to shew unto his servants things which must shortly come to pass; and he sent and signified it by his angel unto his servant John:"}
{"index":{"_id":1002}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":2,"passage":"Revelation 1:2","verse":"2Who bare record of the word of God, and of the testimony of Jesus Christ, and of all things that he saw."}
{"index":{"_id":1003}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":3,"passage":"Revelation 1:3","verse":"3Blessed is he that readeth, and they that hear the words of this prophecy, and keep those things which are written therein: for the time is at hand."}
{"index":{"_id":1004}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":4,"passage":"Revelation 1:4","verse":"4John to the seven churches which are in Asia: Grace be unto you, and peace, from him which is, and which was, and which is to come; and from the seven Spirits which are before his throne; "}
{"index":{"_id":1005}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":5"passage":"Revelation 1:5","verse":"5And from Jesus Christ, who is the faithful witness, and the first begotten of the dead, and the prince of the kings of the earth. Unto him that loved us, and washed us from our sins in his own blood, "}
{"index":{"_id":1006}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":6,"passage":"Revelation 1:6","verse":"6And hath made us kings and priests unto God and his Father; to him be glory and dominion for ever and ever. Amen"}
{"index":{"_id":1007}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":7,"passage":"Revelation 1:7","verse":"7Behold, he cometh with clouds; and every eye shall see him, and they also which pierced him: and all kindreds of the earth shall wail because of him. Even so, Amen."}
{"index":{"_id":1008}}
{"bookID":66,"book":"Revelation","chapterID":1,"verseID":8,"passage":"Revelation 1:8","verse":"8I am Alpha and Omega, the beginning and the ending, saith the Lord, which is, and which was, and which is to come, the Almighty."}

XML File – Image

revelation_20180804_0808AM.png

Output

addUsingBatch_20180803_0403PM

Review Data

Review Data based on JSON file

Command File

setlocal

set "_binfolder=C:\Downloads\Curl\v7.61.0\Windows\extract\curl-7.61.0-win64-mingw\bin"
set "_binapp=curl.exe"
set "_bin=%_binfolder%\%_binapp%"

set "_url=http://localhost:9200"

set "_esindex=bible"
set "_estype=biblekjv"
set "_esoperation=_search"
set "_espath=/%_esindex%/%_estype%/%_esoperation%"

set "_jsonfile=jsonfile\search_book_Revelation.json"
set "_jsonfileFQDN=%cd%\%_jsonfile%"

set "_contentType=Content-Type: application/json"

%_bin% -i -H "%_contentType%"  %_url%%_espath%?pretty --data-binary @%_jsonfileFQDN%

endlocal

XML File

{
  "query":
  {
    "bool":
    {
      "must":
      [
        { "match":
             { "book": "Revelation" }
        }
      ]
    }
  }
  , "sort" :
  {
      "bookID": {"order": "asc"}
    , "chapterID": {"order": "asc"}
    , "verseID": {"order": "asc"}
  }
} 

Output

fetchUsingQuery_20180803_0425PM

Source Code Control

Git

DanielAdeniji/elasticSearchcURLBatch

Link

Elasticsearch / Jörg Prante – JDBC importer

Background

On the ElasticSearch’s web site, one of the tools listed under the “Data Import & Export Validation page” is Jörg Prante’s – JDBC importer.

In this post, we will do a quick assessment.

 

Artifacts

Jörg Prante’s – JDBC importer is open source and it is available on GitHub.

The project’s URL is https://github.com/jprante/elasticsearch-jdbc.

The releases URL is https://github.com/jprante/elasticsearch-jdbc/releases.

As of today (2018-July-30), here is a screen shot.

The most recent release is 2.3.4.1.

The date is 2016-August-28th; a month short of 2 years ago.

Download

Downloaded the latest version.

 

Build

Built the maven project by invoking the gradlew.bat batch file.

 

Prepare

Environment Variable

JAVA_HOME

Prepared the command shell environment by setting the JAVA_HOME Environment variable.

Here we set JAVA_HOME to our installed Java JDK 1.8 folder.

 

set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_181

 

Command

 

gradlew clean install 

Output

 

Build Deliverable

The build deliverable is placed in the build\distributions folder.

Based on our Source Code Release Number; the exact file name is elasticsearch-jdbc-2.3.4.1-2.3.4.1-dist.zip.

 

Copy to Lib Folder

Please extract the Lib folder from the *dist.zip into the Lib folder under the main folder.

 

Processing

Preparation

Prepare payload

We chose to package our script into two files.  The main initiator and the JSON configuration.

stackoverflow2010.json


{
    "jdbc" :
	{
        "url" : "jdbc:sqlserver://localhost:1433;databaseName=Stackoverflow2010;"
        , "user" : "stackoverflow"
        , "password" : "hIy8jA2lNl"
        , "sql" :  "select top 1 * from Stackoverflow2010.dbo.Users"
	}

    , "elasticsearch" :
		{
             "cluster" : "elasticsearch"
           , "host" : "localhost"
           , "port" : 9200
        }

    , "index" : "stackoveflow2010"
	, "type" : "user"
}

 

invokeStackoverflow2010.cmd


@echo off

setlocal

set DIR=%~dp0

set LIB=%DIR%..\lib\*

set BIN=%DIR%..\bin

set "JAVA_HOME=C:\Program Files\Java\jdk1.8.0_181"

"%JAVA_HOME%\bin\java" -cp %LIB% -Dlog4j.configurationFile="%BIN%\log4j2.xml" "org.xbib.tools.Runner" "org.xbib.tools.JDBCImporter"	stackoverflow2010.json

endlocal

Invoke Command File

Invoked Command File.

Nothing shows up on the console.

But,  bad news on the elasticSearch log file.

Text


java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.6.0]
	at org.elasticsearch.transport.TcpTransport.ensureVersionCompatibility(TcpTransport.java:1462) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1409) ~[elasticsearch-6.3.2.jar:6.3.2]
	at 

Explanation

  1. Received message from unsupported version: [2.0.0] minimal compatible version is: [5.6.0]

 

Other Issues

Compile \ Build

Java JDK

Version

Unfortunately, when we tried using Java JDK 10x we ran into a ditch.

Prepare Environment

To prepare the environment:


set java_home=C:\Program Files\Java\jdk-10.0.2

Invoke gradlew

Command Issued is:


gradlew --help

Output

and the response received is ….

Image

build_java_jdk_v10_20180731_0812PAM

Textual

>set java_home=C:\Program Files\Java\jdk-10.0.2

>gradlew --help

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine java version from '10.0.2'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

>

Summary

Other issues aside, looks like we fell victim to Client’s API compatibility issue.

Our tool is a bit dated as it is stuck at v 2.0.0

Unfortunately, ElasticSearch requires at minimum Version 5.6.0