Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to set date format in Ingest node's date processor #22982

Closed
john16 opened this issue Feb 5, 2017 · 12 comments
Closed

Unable to set date format in Ingest node's date processor #22982

john16 opened this issue Feb 5, 2017 · 12 comments
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement

Comments

@john16
Copy link

john16 commented Feb 5, 2017

Elasticsearch version: 5.1.2

Plugins installed: []

JVM version: OpenJDK

OS version: Ubuntu 16.04

Description of the problem including expected versus actual behavior:

curl -XPUT 'localhost:9200/_ingest/pipeline/test-pipeline?pretty' -H 'Content
-Type: application/json' -d'
{ 
  "description" : "Test Log",
  "processors" : [
    { 
      "grok" : {
        "field": "message",
        "patterns": [ "%{TIMESTAMP_ISO8601:date} %{NUMBER:reqtime} %{NUMBER:http
code} %{IP:ip} %{GREEDYDATA:text}" ]
      },
      "date": {
        "field": "date",
        "target_field": "date",
        "formats": [ "basic_date_time_no_millis" ]
      },
      "date_index_name" : {
        "field" : "date",
        "index_name_prefix" : "test-",
        "date_rounding" : "d"
      }
    }
  ]
}
'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "java.lang.IllegalArgumentException: Illegal pattern component: b",
        "header" : {
          "processor_type" : "date"
        }
      }
    ],
    "type" : "exception",
    "reason" : "java.lang.IllegalArgumentException: Illegal pattern component: b",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Illegal pattern component: b"
    },
    "header" : {
      "processor_type" : "date"
    }
  },
  "status" : 500
}

Steps to reproduce:
1.
2.
3.

Provide logs (if relevant):

See https://discuss.elastic.co/t/ingest-node-processor-date-unable-to-parse-date

@clintongormley clintongormley added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP discuss labels Feb 5, 2017
@clintongormley
Copy link
Contributor

This is because ingest is not using the same date format as specified here https://www.elastic.co/guide/en/elasticsearch/reference/5.2/mapping-date-format.html

@talevy is ingest using joda? Could we support the standard date formats available in ES?

@john16
Copy link
Author

john16 commented Feb 5, 2017

"formats": [ "yyyy-MM-dd'T'HH:mm:ss" ] also throws an exception because of 'T' in the middle (isn't it a valid Joda pattern?)

@john16
Copy link
Author

john16 commented Feb 7, 2017

What is your conclusion so far? Is it a bug or am I doing something wrong?

I need to parse dates in "2017-02-07T10:53:22" format in Ingest node processor.

@talevy
Copy link
Contributor

talevy commented Feb 7, 2017

Hi @john16, I just tried this with no problems.

Although we do not have support for the Elasticsearch basic_date_time_no_millis-like shorthands, your pattern should work.

here is an example curl request:

curl -XPOST "http://localhost:9200/_ingest/pipeline/_simulate" -d'
{
  "pipeline": {
    "processors" : [
      {
        "date": {
          "field": "my_date",
          "formats": ["yyyy-MM-dd'\''T'\''HH:mm:ss"]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "my_date": "2017-02-07T10:53:22"
      }
    }
  ]
}'

The error you are seeing is because of the escaping necessary for single quotes '. with an escape character: '\''

For Kibana's console, you could use this:

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors" : [
      {
        "date": {
          "field": "my_date",
          "formats": ["yyyy-MM-dd'T'HH:mm:ss"]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "my_date": "2017-02-07T10:53:22"
      }
    }
  ]
}

@talevy
Copy link
Contributor

talevy commented Feb 7, 2017

... is ingest using joda? Could we support the standard date formats available in ES?

@clintongormley I don't see a reason why we wouldn't be able to support the built-in date format names. I'll look into it

@dadoonet
Copy link
Member

dadoonet commented Feb 7, 2017

Then it's a documentation issue as we say that we support Joda patterns in https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html (see format)

@dadoonet
Copy link
Member

dadoonet commented Feb 7, 2017

Hmmm. I misread https://www.elastic.co/guide/en/elasticsearch/reference/5.2/mapping-date-format.html. Ignore my previous comment.

@talevy
Copy link
Contributor

talevy commented Feb 7, 2017

@dadoonet, I do like the idea of supporting Elasticsearch's date-format built-ins. I've begun a PR to see how it would fit. Since it is used during mapping/indexing, there isn't too much of a reason it shouldn't be supported

@dadoonet
Copy link
Member

dadoonet commented Feb 7, 2017

Yeah! Same here and I think we should then update the ingest doc to link to https://www.elastic.co/guide/en/elasticsearch/reference/5.2/mapping-date-format.html or something like this.

@talevy
Copy link
Contributor

talevy commented Feb 8, 2017

@dadoonet, I've opened up an issue for this. ^^^.

@john16
Copy link
Author

john16 commented Feb 8, 2017

The error you are seeing is because of the escaping necessary for single quotes '. with an escape character: '''

Thanks! (May be mention this in docs? Because the same pattern works without escaping in Logstash and if you move from Logstash you need to add escaping).

@talevy
Copy link
Contributor

talevy commented Mar 15, 2018

Closing this in favor of #23032

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement
Projects
None yet
Development

No branches or pull requests

4 participants