Incremental view maintenance #280
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
yjhjstz
changed the title
yjhjstz
force-pushed
the
incremental_view_maintenance
branch
2 times, most recently
from
4a1c99a to
61bfb63
Compare
yjhjstz
force-pushed
the
incremental_view_maintenance
branch
from
7a06694 to
7c345d8
Compare
yjhjstz
force-pushed
the
incremental_view_maintenance
branch
from
7c345d8 to
c34f3e7
Compare
yjhjstz
changed the title
yjhjstz
force-pushed
the
incremental_view_maintenance
branch
4 times, most recently
from
c601e29 to
e393fbc
Compare
9 tasks
avamingli
reviewed
Nov 22, 2023
avamingli
reviewed
Nov 22, 2023
yjhjstz
force-pushed
the
incremental_view_maintenance
branch
from
e393fbc to
21474b1
Compare
|
Hi, I want to see what we changed besides pg's patch, but it's hard to tell as there are separate commits of our own codes. |
yjhjstz
force-pushed
the
incremental_view_maintenance
branch
from
21474b1 to
baed6b4
Compare
tried to arrange [Add documentations about Incremental View Maintenance]. |
yjhjstz
force-pushed
the
incremental_view_maintenance
branch
from
baed6b4 to
70e19ab
Compare
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
If this boolean column is true, a relations is Incrementally Maintainable Materialized View (IMMV). This is set when IMMV is created.
Originally, tuplestores of AFTER trigger's transition tables were freed for each query depth. For our IVM implementation, we would like to prolong life of the tuplestores because we have to preserve them for a whole query assuming that some base tables might be changed in some trigger functions.
Add tab completion and meta-command output for IVM.
In this implementation, AFTER triggers are used to collect tuplestores containing transition table contents. When multiple tables are changed, multiple AFTER triggers are invoked, then the final AFTER trigger performs actual update of the matview. In addition, BEFORE triggers are also used to handle global information for view maintenance. To calculate view deltas, we need both pre-state and post-state of base tables. Post-update states are available in AFTER trigger, and pre-update states can be calculated by removing inserted tuples and appending deleted tuples. Insterted tuples are filtered using the snapshot taken before table modiication, and deleted tuples are contained in the old transition table. Incrementally Maintainable Materialized Views (IMMV) can contain duplicated tuples. This patch also allows self-join, simultaneous updates of more than one base table, and multiple updates of the same base table.
Details steps: Step1: export snapshot, trigger data suppose is dR. Step2: calculation delta V, dV = dR ⋈ S Step3: apply delta V, V_new = dV U V Step4: dispatch cleanup Ivm feature that allows materialized views to be updated automatically and incrementally just after a underlying table is modified. You can create an incremental maintainable materialized view (IMMV) by using CREATE INCREMENTAL MATERIALIZED VIEW command. The followings are designed to support in view definition queries: - SELECT ... FROM ... WHERE ..., joins (inner joins, self-joins) - some built-in aggregate functions (count, sum, avg) - GROUP BY clause - DISTINCT clause Restriction: The following are not supported in a view definition: - Outer joins - Aggregates other than above, window functions, HAVING - Sub-queries, CTEs - Set operations (UNION, INTERSECT, EXCEPT) - DISTINCT ON, ORDER BY, LIMIT, OFFSET
yjhjstz
force-pushed
the
incremental_view_maintenance
branch
from
70e19ab to
572ab43
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #ISSUE_Number
Change logs
background:
Why are the changes needed?
Base on patch v28 from Postgres, here is design code: https://hashdata.feishu.cn/docx/RdS6dND7xotUhLxQi1UctzCVnih
The challenge lies in how to implement IVM in a distributed environment.
Does this PR introduce any user-facing change?
If yes, please clarify the previous behavior and the change this PR proposes.
How was this patch tested?
Add extra test to regress:
Contributor's Checklist
Here are some reminders and checklists before/when submitting your pull request, please check them:
make installcheckmake -C src/test installcheck-cbdb-parallel