Fix: generate correct AWS SIGv4 signature for compressed requests

[dblock]

1. Set x-Amz-Decoded-Content-Length to decompressed content.
2. Do not skip any headers.

If you can confirm let's figure out what this code should really be. For starters, I think decompressing initially uncompressed content is probably not very efficient, need to find a way to get that content length directly.

[pgtgrly]

Hi @dblock, I tried replicating the issue and using the updated Apache interceptor here. I am still getting the same exception:

{"year":"2005","director":"James Mangold","title":"Walk the Line"}
Accept-Encoding: gzip
Authorization: AWS4-HMAC-SHA256 Credential=xxx/20220625/us-west-2/es/aws4_request, SignedHeaders=accept-encoding;content-encoding;content-type;host;transfer-encoding;user-agent;x-amz-date;x-amz-decoded-content-length;x-amz-security-token, Signature=43fa67b1aa4d7eda063b64f18c3f24b085c701cd28598646d53c287fe2e61a07
Connection: Keep-Alive
Content-Encoding: gzip
Content-Type: application/json
Host: search-compression-client-test-wjvs7pttvndq3nhffx4ksqy6cy.us-west-2.es.amazonaws.com
Transfer-Encoding: chunked
User-Agent: Apache-HttpAsyncClient/4.1.4 (Java/11.0.14)
X-Amz-Date: 20220625T073929Z
x-Amz-Decoded-Content-Length: 66
X-Amz-Security-Token: 
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
Exception in thread "main" OpenSearchStatusException[Unable to parse response body]; nested: ResponseException[method [PUT], host [https://search-compression-client-test-wjvs7pttvndq3nhffx4ksqy6cy.us-west-2.es.amazonaws.com], URI [/my-index/_doc/1?timeout=1m], status line [HTTP/1.1 403 Forbidden]
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}];
    at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2059)
    at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1775)
    at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728)
    at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696)
    at org.opensearch.client.RestHighLevelClient.index(RestHighLevelClient.java:961)
    at org.example.AmazonOpenSearchServiceSample.main(AmazonOpenSearchServiceSample.java:41)
    Suppressed: ParsingException[Failed to parse object: expecting field with name [error] but found [message]]
        at org.opensearch.common.xcontent.XContentParserUtils.ensureFieldName(XContentParserUtils.java:63)
        at org.opensearch.OpenSearchException.failureFromXContent(OpenSearchException.java:629)
        at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:199)
        at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2079)
        at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2056)
        ... 5 more
Caused by: org.opensearch.client.ResponseException: method [PUT], host [https://search-compression-client-test-wjvs7pttvndq3nhffx4ksqy6cy.us-west-2.es.amazonaws.com], URI [/my-index/_doc/1?timeout=1m], status line [HTTP/1.1 403 Forbidden]
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}
    at org.opensearch.client.RestClient.convertResponse(RestClient.java:350)
    at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
    at org.opensearch.client.RestClient.performRequest(RestClient.java:295)
    at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762)
    ... 4 more

[kumjiten]

Hi @dblock,

it still fails with the interceptor changes as well,
I pulled your changes in my repo.

{"settings":{"index":{"number_of_shards":"2","number_of_replicas":"1"}},"mappings":{"properties":{"age":{"type":"integer"}}},"aliases":{}}
Accept-Encoding: gzip
Authorization: AWS4-HMAC-SHA256 Credential=AKIAV7BDGZUCRKUTEG7B/20220625/eu-west-1/es/aws4_request, SignedHeaders=accept-encoding;content-encoding;content-type;host;transfer-encoding;user-agent;x-amz-date;x-amz-decoded-content-length, Signature=86f6f583723cc4fb3625b5c49714218a2e02f918343a78e2152b081d820f8ab3
Connection: Keep-Alive
Content-Encoding: gzip
Content-Type: application/json
Host: sxxxxxxx
Transfer-Encoding: chunked
User-Agent: Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_322)
X-Amz-Date: 20220625T091625Z
x-Amz-Decoded-Content-Length: 138
OpenSearchStatusException[Unable to parse response body]; nested: ResponseException[method [PUT], host [https://xxxxxxx], URI [/custom-index?master_timeout=30s&timeout=30s], status line [HTTP/1.1 403 Forbidden]
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.'\n"}];
	at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1653)
	at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1407)
	at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1379)
	at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1346)
	at org.opensearch.client.IndicesClient.create(IndicesClient.java:150)
	at RESTClientTest.main(RESTClientTest.java:55)
	Suppressed: ParsingException[Failed to parse object: expecting field with name [error] but found [message]]
		at org.opensearch.common.xcontent.XContentParserUtils.ensureFieldName(XContentParserUtils.java:62)
		at org.opensearch.OpenSearchException.failureFromXContent(OpenSearchException.java:621)
		at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:192)
		at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1673)
		at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1650)
		... 5 more

[dblock]

Hi @dblock, I tried replicating the issue and using the updated Apache interceptor here. I am still getting the same exception

you still have the skip header logic in your code, that should be removed

[dblock]

Hi @dblock, I tried replicating the issue and using the updated Apache interceptor here. I am still getting the same exception

you still have the skip header logic in your code, that should be removed

Looks like my changes aren’t working for me anymore either. Not sure why it was intermittently passing, I’ll keep looking.

[dblock]

Going to close this, leaving https://github.com/dblock/opensearch-java-client-demo in a state where the problem is reproduced.