Fail span_multi queries that don't use top_terms_* rewriting

[jimczi]

The span multi term query limits the number of expanded terms to search only if the inner multi term query uses top_terms_N rewrite. Otherwise it extracts all terms that match the inner multi term query and use them to build a giant span_or query without any limit.
We should force the inner multi term query to use a top_terms_N rewrite in order to limit the memory. Currently a prefix query like a* would extract all term that starts with a since the default rewrite of the prefix query is constant_score. This mode is optimized for prefix query and should never be used on a span_multi which needs to access all positions for the extracted terms.

[jimczi]

Discussed in fixit Friday. We should limit the expansion to a sane value and improve the documentation around rewriting of multi term queries.

[jpountz]

We should limit the expansion to a sane value

Does it mean enforcing a top-terms rewrite or failing the query if there are more than X terms?

[jimczi]

Does it mean enforcing a top-terms rewrite or failing the query if there are more than X terms?

I'd prefer enforcing a top-terms rewrite with an explicit option to limit the expansion:

"span_multi":{
  "match":{
    "prefix" : { "user" :  { "value" : "ki" } }
   },
  "max_expansions": 100
}

Then we can fail the query if an explicit rewrite is set on the inner multi query or if max_expansions is greater than 1024. I also think that max_expansions should always be set and that we should fail queries without an explicit value. A default value would hide the new behavior which is breaking anyway. WDYT ?

[jpountz]

Sounds good to me. Should it be a 7.0 change then?

[jimczi]

We can also add max_expansions as an override of the rewrite in 6.x. This would not break anything in that version and we can add a deprecation warning if the value is not set ?

[jpountz]

👍

[jimczi]

Closed by #30913

[nirmalc]

Thanks @jimczi , are we planning to patch this on 6.4 /6.5 ? - I can rebase against 6.4 if you prefer ( Thats what we plan to launch in prod ;-) )

[jimczi]

@nirmalc it's merged in 6.x already so 6.4 it is.