Unable to set date format in Ingest node's date processor

[john16]

Elasticsearch version: 5.1.2

Plugins installed: []

JVM version: OpenJDK

OS version: Ubuntu 16.04

Description of the problem including expected versus actual behavior:

curl -XPUT 'localhost:9200/_ingest/pipeline/test-pipeline?pretty' -H 'Content
-Type: application/json' -d'
{ 
  "description" : "Test Log",
  "processors" : [
    { 
      "grok" : {
        "field": "message",
        "patterns": [ "%{TIMESTAMP_ISO8601:date} %{NUMBER:reqtime} %{NUMBER:http
code} %{IP:ip} %{GREEDYDATA:text}" ]
      },
      "date": {
        "field": "date",
        "target_field": "date",
        "formats": [ "basic_date_time_no_millis" ]
      },
      "date_index_name" : {
        "field" : "date",
        "index_name_prefix" : "test-",
        "date_rounding" : "d"
      }
    }
  ]
}
'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "java.lang.IllegalArgumentException: Illegal pattern component: b",
        "header" : {
          "processor_type" : "date"
        }
      }
    ],
    "type" : "exception",
    "reason" : "java.lang.IllegalArgumentException: Illegal pattern component: b",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Illegal pattern component: b"
    },
    "header" : {
      "processor_type" : "date"
    }
  },
  "status" : 500
}

Steps to reproduce:
1.
2.
3.

Provide logs (if relevant):

See https://discuss.elastic.co/t/ingest-node-processor-date-unable-to-parse-date

[clintongormley]

This is because ingest is not using the same date format as specified here https://www.elastic.co/guide/en/elasticsearch/reference/5.2/mapping-date-format.html

@talevy is ingest using joda? Could we support the standard date formats available in ES?

[john16]

"formats": [ "yyyy-MM-dd'T'HH:mm:ss" ] also throws an exception because of 'T' in the middle (isn't it a valid Joda pattern?)

[john16]

What is your conclusion so far? Is it a bug or am I doing something wrong?

I need to parse dates in "2017-02-07T10:53:22" format in Ingest node processor.

[talevy]

Hi @john16, I just tried this with no problems.

Although we do not have support for the Elasticsearch basic_date_time_no_millis-like shorthands, your pattern should work.

here is an example curl request:

curl -XPOST "http://localhost:9200/_ingest/pipeline/_simulate" -d'
{
  "pipeline": {
    "processors" : [
      {
        "date": {
          "field": "my_date",
          "formats": ["yyyy-MM-dd'\''T'\''HH:mm:ss"]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "my_date": "2017-02-07T10:53:22"
      }
    }
  ]
}'

The error you are seeing is because of the escaping necessary for single quotes '. with an escape character: '\''

For Kibana's console, you could use this:

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors" : [
      {
        "date": {
          "field": "my_date",
          "formats": ["yyyy-MM-dd'T'HH:mm:ss"]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "my_date": "2017-02-07T10:53:22"
      }
    }
  ]
}

[talevy]

... is ingest using joda? Could we support the standard date formats available in ES?

@clintongormley I don't see a reason why we wouldn't be able to support the built-in date format names. I'll look into it

[dadoonet]

Then it's a documentation issue as we say that we support Joda patterns in https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html (see format)

[dadoonet]

Hmmm. I misread https://www.elastic.co/guide/en/elasticsearch/reference/5.2/mapping-date-format.html. Ignore my previous comment.

[talevy]

@dadoonet, I do like the idea of supporting Elasticsearch's date-format built-ins. I've begun a PR to see how it would fit. Since it is used during mapping/indexing, there isn't too much of a reason it shouldn't be supported

[dadoonet]

Yeah! Same here and I think we should then update the ingest doc to link to https://www.elastic.co/guide/en/elasticsearch/reference/5.2/mapping-date-format.html or something like this.

[talevy]

@dadoonet, I've opened up an issue for this. ^^^.

[john16]

The error you are seeing is because of the escaping necessary for single quotes '. with an escape character: '''

Thanks! (May be mention this in docs? Because the same pattern works without escaping in Logstash and if you move from Logstash you need to add escaping).

[talevy]

Closing this in favor of #23032