Logstash – Configuration – Field Names – document_id

Background

In Logstash, got bogged down trying to use the document_id Configuration Option.

Reproduction

logstash

Configuration

Image

configuration_file_20180809_0820AM.PNG

ElasticSearch

Query

Image

query_stackoverflowUser_20180809_0825AM.PNG

Data

Image

stackoverflowUser_output_20180809_0830AM.PNG

 

Troubleshooting

ElasticSearch

Data Review

  1. Hits
    • Number of Hits
      • We added 10 records
        • Data Source Query says “select top 10 *
      • But,  hits/total
      • Reads  1
    • Lone Hit
      • Data
        • _id
          • “_id”: “%{[Id]}”
        • field
          • id
            • “id”: 10

Logstash

Output

Image

output_20180809_0851AM.PNG

Textual


{"id":-1,"lastaccessdate":"2008-08-26T07:16:53.810Z","reputation":1,"accountid":-1,"location":"on the server farm","downvotes":890820,"displayname":"Community","@timestamp":"2018-08-09T13:48:09.088Z","age":null,"emailhash":null,"views":649,"@version":"1","creationdate":"2008-07-31T07:00:00.000Z","aboutme":"Hi, I'm not really a person.

\n\nI'm a background process that helps keep this site clean!

\n\nI do things like

\n\n
<ul>\n
	<li>Randomly poke old unanswered questions every hour so they get some attention</li>
\n
	<li>Own community questions and answers so nobody gets unnecessary reputation from them</li>
\n
	<li>Own downvotes on spam/evil posts that get permanently deleted</li>
\n
	<li>Own suggested edits from anonymous users</li>
\n
	<li><a href="http://meta.stackexchange.com/a/92006">Remove abandoned questions</a></li>
\n</ul>
\n","websiteurl":"http://meta.stackexchange.com/","upvotes":225495}

SQL Server

Diagram

dbo.Users

Image

stackOverflow2010.Users.PNG

Explanation
  1. Column name is Id

Resolution

In Logstash, Field names are lower cased

Logstash

Configuration

Image

configuration_file_20180809_0903AM.PNG

ElasticSearch

Query

Query Output

Image

output_20180809_0908AM.PNG

Explanation
  1. hits = 10

References

  1. Logstash
    • Configuring
      • Logstash Reference [6.3] » Configuring Logstash » Accessing Event Data and Fields in the Configuration
        Link
    • Transforming Data
      • Logstash Reference [6.3] » Transforming Data » Extracting Fields and Wrangling Data
        Link
    • Configuring Logstash
      • Logstash Reference [6.3] » Configuring Logstash » Structure of a Config File
        Link
    • Output
      • Logstash Reference [6.3] » Output plugins » Elasticsearch output plugin
        Link
    • Filter
      • Logstash Reference [6.2] » Filter plugins » Mutate filter plugin
        Link

 

 

 

 

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s