Incorrect date formatting in License Expiration logs

[tsullivan]

Describe the feature:

Elasticsearch version (bin/elasticsearch --version):
8.0.0-SNAPSHOT

[root@73cce364f2b8 elasticsearch]# bin/elasticsearch --version
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Version: 8.0.0-SNAPSHOT, Build: default/tar/dc02278/2019-02-19T08:51:47.611087Z, JVM: 11.0.2

Plugins installed: []

JVM version (java -version):

OS version (uname -a if on a Unix-like system):
Docker

[root@73cce364f2b8 elasticsearch]# uname -a
Linux 73cce364f2b8 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Log message gives info on when my X-Pack license will expire, and the date string as formatted makes no sense. It reads: License [will expire] on [Sunday, 3 10, 2019]

elasticsearch_1 | {"type": "server", "timestamp": "2019-02-19T16:27:25,723+0000", "level": "WARN", "component": "o.e.l.LicenseService", "cluster.name": "dev-docker-cluster", "node.name": "73cce364f2b8", "cluster.uuid": "kNukTB2zQuW70QURs2g_dg", "node.id": "DjxjFDMjQ7WMbLrDsVxWmQ", "message": "License [will expire] on [Sunday, 3 10, 2019].\n# If you have a new license, please update it. Otherwise, please reach out to\n# your support contact.\n# \n# Commercial plugins operate with reduced functionality on license expiration:\n# - security\n# - Cluster health, cluster stats and indices stats operations are blocked\n# - All data operations (read and write) continue to work\n# - watcher\n# - PUT / GET watch APIs are disabled, DELETE watch API continues to work\n# - Watches execute and write to the history\n# - The actions of the watches don't execute\n# - monitoring\n# - The agent will stop collecting cluster and indices metrics\n# - The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]\n# - graph\n# - Graph explore APIs are disabled\n# - ml\n# - Machine learning APIs are disabled\n# - logstash\n# - Logstash will continue to poll centrally-managed pipelines\n# - beats\n# - Beats will continue to poll centrally-managed configuration\n# - deprecation\n# - Deprecation APIs are disabled\n# - upgrade\n# - Upgrade API is disabled\n# - sql\n# - SQL support is disabled\n# - rollup\n# - Creating and Starting rollup jobs will no longer be allowed.\n# - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function." }

Provide logs (if relevant):

�[36melasticsearch_1        |�[0m {"type": "server", "timestamp": "2019-02-19T16:27:23,860+0000", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "dev-docker-cluster", "node.name": "73cce364f2b8",  "message": "publish_address {172.30.0.2:9200}, bound_addresses {0.0.0.0:9200}"  }
�[36melasticsearch_1        |�[0m {"type": "server", "timestamp": "2019-02-19T16:27:23,866+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "dev-docker-cluster", "node.name": "73cce364f2b8",  "message": "started"  }
�[36melasticsearch_1        |�[0m {"type": "server", "timestamp": "2019-02-19T16:27:25,684+0000", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "dev-docker-cluster", "node.name": "73cce364f2b8",  "message": "license [b458f5e9-f44b-4b36-be22-f9836445ddb8] mode [trial] - valid"  }
�[36melasticsearch_1        |�[0m {"type": "server", "timestamp": "2019-02-19T16:27:25,718+0000", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "dev-docker-cluster", "node.name": "73cce364f2b8", "cluster.uuid": "kNukTB2zQuW70QURs2g_dg", "node.id": "DjxjFDMjQ7WMbLrDsVxWmQ",  "message": "recovered [16] indices into cluster_state"  }
�[36melasticsearch_1        |�[0m {"type": "server", "timestamp": "2019-02-19T16:27:25,723+0000", "level": "WARN", "component": "o.e.l.LicenseService", "cluster.name": "dev-docker-cluster", "node.name": "73cce364f2b8", "cluster.uuid": "kNukTB2zQuW70QURs2g_dg", "node.id": "DjxjFDMjQ7WMbLrDsVxWmQ",  "message": "License [will expire] on [Sunday, 3 10, 2019].\n# If you have a new license, please update it. Otherwise, please reach out to\n# your support contact.\n# \n# Commercial plugins operate with reduced functionality on license expiration:\n# - security\n#  - Cluster health, cluster stats and indices stats operations are blocked\n#  - All data operations (read and write) continue to work\n# - watcher\n#  - PUT / GET watch APIs are disabled, DELETE watch API continues to work\n#  - Watches execute and write to the history\n#  - The actions of the watches don't execute\n# - monitoring\n#  - The agent will stop collecting cluster and indices metrics\n#  - The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]\n# - graph\n#  - Graph explore APIs are disabled\n# - ml\n#  - Machine learning APIs are disabled\n# - logstash\n#  - Logstash will continue to poll centrally-managed pipelines\n# - beats\n#  - Beats will continue to poll centrally-managed configuration\n# - deprecation\n#  - Deprecation APIs are disabled\n# - upgrade\n#  - Upgrade API is disabled\n# - sql\n#  - SQL support is disabled\n# - rollup\n#  - Creating and Starting rollup jobs will no longer be allowed.\n#  - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function."  }
�[36melasticsearch_1        |�[0m {"type": "server", "timestamp": "2019-02-19T16:27:26,794+0000", "level": "INFO", "component": "o.e.x.w.WatcherService", "cluster.name": "dev-docker-cluster", "node.name": "73cce364f2b8", "cluster.uuid": "kNukTB2zQuW70QURs2g_dg", "node.id": "DjxjFDMjQ7WMbLrDsVxWmQ",  "message": "reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks"  }
�[36melasticsearch_1        |�[0m {"type": "server", "timestamp": "2019-02-19T16:27:27,889+0000", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "dev-docker-cluster", "node.name": "73cce364f2b8", "cluster.uuid": "kNukTB2zQuW70QURs2g_dg", "node.id": "DjxjFDMjQ7WMbLrDsVxWmQ",  "message": "Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[babynames][1], [.kibana_1][0]] ...])."  }

[elasticmachine]

Pinging @elastic/es-security

[tvernum]

Relates: #27330

@spinscale @rjernst
There are two joda/java time changes that affect this, 1 of them is easy to solve, the other less so.

1. Easy: In Joda 4+ characters means "full format". In Java-Time 4 characters means "full format", but 5 characters means "narrow format" (a single letter), so MMMMM used to mean "full month name", but now means "first character of the month name". Solution: Use MMMM (or LLLL)
2. Ugly: Java-time doesn't seem to support true "full format" names on the ROOT locale. If we force Locale.ENGLISH, then MMMM will output as February, but in Locale.ROOT (our default behaviour) it will only output Feb, I cannot find a way to get it use full names for months or days of the week on the ROOT locale.

Do either of you know of a good solution for point 2? The options seem to be:

1. Accept that it will use abbreviated names. This is a minor behaviour change from 6.x, and not a big deal, but retaining the full names would make the warning message clearer.
2. Force the formatter to use the ENGLISH locale.

[pgomulka]

@tvernum I think you are right - the jdk since 9 is using CLDR which is not specifying full format for ROOT locale. https://openjdk.java.net/jeps/252
we might be able force the jdk8 behaviour where the 'full form' has fully formated names (as per that openjdk ticket -Djava.locale.providers=COMPAT)

I would rather prefer to break the behaviour in 7 and have a 'short form' names.

But still using MMMM - I am not sure if we should expect at some point CLDR returning 'full form' for root locale?
someone raised the same problem: https://bugs.openjdk.java.net/browse/JDK-8205675

[pgomulka]

@tsullivan I have changed this, but I noticed that by default we do run elasticsearch with COMPAT for jdk9+.
did you run your elasticsearch with a custom jvm.options?
by default we have 9-:-Djava.locale.providers=COMPAT in this file.