Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add decision record about IDS filtering #1983

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Add filtering to IDS catalog requests

## Decision

Extend existing `CatalogRequest` object with list of criteria used to filter `Assets` within `Catalog` response.

## Rationale

In a large scale environment, with multiple EDCs holding the information about hundreds of thousands of `Assets`, instead of
looping thought all of the `ContractOffers`, there is a need to narrow the results down to even single offer. For example, an
`Asset` might hold the 'special' type of data (like registry or database) and should be searchable by its type.

## Approach

As the logic for querying `AssetIndex` by `QuerySpec` is already in place, all we have to do is to pass filtering criteria via IDS Api:


`filter` property has to be added to the IDS message:

```
// in MultiPartCatalogDescriptionRequestSender.java
message.setProperty(CatalogRequest.FILTER, request.getFilter());
```

Then, on the receiving end, it's simply extracted:

```
// in DescriptionRequestHandler.java
var querySpec = ofNullable(message.getProperties().get(CatalogRequest.FILTER))
.map(map -> objectMapper.convertValue(map, typeRef))
.orElse(/*should never occur!!*/)
```

Existing `ContractOfferQuery` will be used to transport both `Range` and filtering criteria to lower layers.

And now, they can be applied to `AssetIndex` search query when constructing the `Catalog` response:

1) pass Criterions to the `CatalogServiceImpl.getDataCatalog()`
2) attach to the existing `ContractOfferQuery` object
3) in `ContractOfferServiceImpl.queryContractOffers()`, merge with `AssetSelectorExpressions` from contract definitions.

## Considerations and limitations

- there is no standardized query language nor will there be one for the foreseeable future;
- querying is based on the "Canonical format": (https://github.com/eclipse-dataspaceconnector/DataSpaceConnector/blob/main/docs/developer/sql_queries.md),
i.e. the schema of the queried objects, i.e. their Java class. That implies that the client must have knowledge of the schema;
- that schema is subject to change without special notice.
1 change: 1 addition & 0 deletions docs/developer/decision-records/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,4 @@
- [2022-08-09 Project structure review](2022-08-09-project-structure-review/)
- [2022-08-11 Versioning and Artifacts](2022-08-11-versioning_and_artifacts/)
- [2022-08-17 Remove H2 Database Tests](2022-08-17-remove_h2_database_tests/)
- [2022-09-18 IDS catalog request filtering](2022-09-18-ids-catalog-request-filtering/)