[Feature Request]: A filter_include argument for Collection.get and Collection.query

[thorwhalen]

Describe the problem

The semantics of the output of Collection.get and Collection.query are not clear.
This, and other related matters, were already discussed in the issue Improve message on collection.get empty embeddings.

Consider this example. When doing a Collection.get with include=["metadatas", "documents"] we get:

{
  'ids': ['foo', 'apple'],
  'embeddings': None,
  'metadatas': [None, None],
  'documents': ['bar', 'crumble'],
  'uris': None,
  'data': None
}

The None of "uris" means "you didn't ask for it".
The [None, None] of the "metadatas" means "you asked for it, but there were None.

I'd argue that:

{
  'ids': ['foo', 'apple'],
  'metadatas': [None, None],
  'documents': ['bar', 'crumble'],
}

might be less confusing.

Describe the proposed solution

Include a filter_include argument in Collection.get and Collection.query, which (if True) will have the effect of filtering in only those fields that were requested (plus "ids").

I propose to set the default of filter_include to be False for now, controlled via a DFLT_FILTER_INCLUDE variable.
This is so that the current behavior doesn't change, so we have minimal disruption.

Later, if worth it, we can switch to a DFLT_FILTER_INCLUDE=True default.
We can also consider letting users control this via some settings/configs mechanism.

Alternatives considered

Alternatives can be found in issue 300.

Importance

would make my life easier

Additional Information

I intend on doing a PR on this.

[thorwhalen]

Personally, I'm not a fan of the filter_include name for the argument, but haven't thought of better. Do propose if you think of something.

[thorwhalen]

Implemented in PR: #1626