Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flag to disable operator rewriting hooks and make pgvector compatible #240

Merged
merged 18 commits into from
Dec 9, 2023

Conversation

var77
Copy link
Collaborator

@var77 var77 commented Dec 6, 2023

  • Added lantern.pgvector_compat variable which when set to TRUE will disable the <-> operator rewriting hooks and enable <=> and <+> operator usages for cosine and hamming distances.
  • Added operators for cosine (<=>) and hamming (<+>) distances with corresponding operator classes. These operators will only work if lantern.pgvector_compat=TRUE
  • Added operators for cosine and hamming distances for vector type.
  • Added update file to create/recreate the operator classes, as this operation will drop the existing indexes I have added reindexing logic in update file
  • In CI added function to run pgvector tests with lantern.pgvector_compat=TRUE, so we will be sure that no operator rewriting is happening

@var77 var77 requested a review from Ngalstyan4 December 6, 2023 11:05
Copy link

github-actions bot commented Dec 6, 2023

Benchmarks

metric old new pct change
recall (after create) 0.740 0.740 -
recall (after insert) 0.750 0.778 +3.73%
select bulk tps 487.436 485.806 -0.33%
select bulk latency (ms) 15.557 15.863 +1.97%
select bulk latency (stddev ms) 1.866 3.717 +99.20%
create latency (ms) 1195.214 1197.182 +0.16%
insert bulk tps 11.502 11.589 +0.75%
insert bulk latency (ms) 86.931 86.282 -0.75%
insert bulk latency (stddev ms) 1.991 3.286 +65.04%
disk usage (bytes) 6348800.000 6348800.000 -

Copy link

codecov bot commented Dec 6, 2023

Codecov Report

Merging #240 (fbff1d0) into main (37a8519) will decrease coverage by 0.18%.
The diff coverage is 81.08%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #240      +/-   ##
==========================================
- Coverage   77.85%   77.68%   -0.18%     
==========================================
  Files          23       23              
  Lines        1653     1685      +32     
  Branches      410      415       +5     
==========================================
+ Hits         1287     1309      +22     
- Misses        185      191       +6     
- Partials      181      185       +4     
Files Coverage Δ
src/hooks/executor_start.c 80.43% <100.00%> (-7.94%) ⬇️
src/hooks/op_rewrite.c 83.21% <ø> (ø)
src/hnsw.c 81.98% <96.15%> (+1.36%) ⬆️
src/hooks/post_parse.c 87.50% <0.00%> (-1.25%) ⬇️
src/hnsw/options.c 82.53% <28.57%> (-2.47%) ⬇️

... and 2 files with indirect coverage changes

@var77 var77 force-pushed the varik/op-disable-flag branch from 5eb3afb to 65db102 Compare December 7, 2023 13:18
@var77 var77 changed the title Add flag to disable operator rewriting hooks Add flag to disable operator rewriting hooks and make pgvector compatible Dec 7, 2023
Copy link
Contributor

@Ngalstyan4 Ngalstyan4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good! This was quick:)

@Ngalstyan4 Ngalstyan4 merged commit 6223b7a into main Dec 9, 2023
@Ngalstyan4 Ngalstyan4 deleted the varik/op-disable-flag branch December 9, 2023 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants