Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

moving LastWrittenLSNCache to Neon Extension #11031

Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
80f0c9b
added first draft code for moving lwlsn cache to neon ext
thesuhas Feb 27, 2025
5bb0dc8
corrected hook installations and added getter hook for lsn cache size
thesuhas Feb 28, 2025
877d6b7
added lwlc_init in neon.h
thesuhas Feb 28, 2025
163c7de
got neon building with modified pg17
thesuhas Feb 28, 2025
4213da6
pushing changed postgres for CI test
thesuhas Feb 28, 2025
c55fc72
fix: updated overwritten commit in revisions.json
thesuhas Mar 3, 2025
ba5f22b
updated revisions.json to reflect updated postgres commit
thesuhas Mar 3, 2025
2d3d9d4
updated revisions.json to point to commit without getter funcs (made …
thesuhas Mar 3, 2025
6f3e9b0
updated revisions.json to point to commit without getter funcs (made …
thesuhas Mar 3, 2025
11af1d0
Merge branch 'thesuhas/add_lwlsn_neon_ext' of github.com:neondatabase…
thesuhas Mar 3, 2025
8f339e9
Fix v15 and v16 postgres submodules
tristan957 Mar 3, 2025
ad99420
Fix the v17 submodule
tristan957 Mar 3, 2025
38e8162
updated revisions.json to point to postgres commit with updated behav…
thesuhas Mar 3, 2025
6afd983
updated pg v17
thesuhas Mar 3, 2025
31426e3
updated references to lwlsn func calls in pg v17 to use neon extension
thesuhas Mar 3, 2025
3b6297e
added pg check for disabling lwlsn cache in walredoproc
thesuhas Mar 3, 2025
64ab566
addressed comments and updated function names, calling init_lwlc() in…
thesuhas Mar 4, 2025
c18f15c
pointing to new pg17
thesuhas Mar 4, 2025
613c9d8
added include guards and pg agnostic data types in neon_lwlc
thesuhas Mar 4, 2025
13576ef
Fix build issues
tristan957 Mar 4, 2025
e220863
fixed build issues and warnings
thesuhas Mar 4, 2025
f2a0632
removed install_hook macro
thesuhas Mar 4, 2025
b382579
updated to point to latest postgres changes
thesuhas Mar 5, 2025
1ebb933
pointing to new pg
thesuhas Mar 5, 2025
488ff0c
Merge branch 'main' into thesuhas/add_lwlsn_neon_ext
thesuhas Mar 5, 2025
410b6a4
fixed issue in pg
thesuhas Mar 5, 2025
2a60e4c
updated pg
thesuhas Mar 5, 2025
d6f418e
added a struct to ensure objects are in shared memory
thesuhas Mar 5, 2025
a3c8192
addressed comments, added check to see if lwlsncachectl was previousl…
thesuhas Mar 5, 2025
bbf07b4
pushing logging code for reference, will remove later
thesuhas Mar 6, 2025
e5493e9
fixed shmem startup hook that was missing
thesuhas Mar 6, 2025
53af8c4
removed log statements
thesuhas Mar 7, 2025
cba3167
fixed issue with utilising uninitalised lwlsn cache size variable of …
thesuhas Mar 10, 2025
31115b6
updated pg17 to resolve ci
thesuhas Mar 10, 2025
25a01d6
pointing to merged pg 17
thesuhas Mar 10, 2025
9531180
Merge branch 'main' into thesuhas/add_lwlsn_neon_ext
thesuhas Mar 10, 2025
3f5e6a6
updated revisions.json
thesuhas Mar 10, 2025
814ed1c
moved lwlsn cache to neon ext for pg15 and pg16
thesuhas Mar 11, 2025
52a03b1
updated pg17
thesuhas Mar 11, 2025
a9df95d
added pg14 compatibility
thesuhas Mar 11, 2025
89d61a5
moved shmemrequest to init() for pgv14
thesuhas Mar 11, 2025
3820268
updated revisions.json, removed unnecessary lwlsn func in pagestore smgr
thesuhas Mar 11, 2025
c3a5370
fixed maxlastwrittenlsn not updating
thesuhas Mar 12, 2025
8807020
Merge branch 'main' into thesuhas/add_lwlsn_neon_ext
thesuhas Mar 12, 2025
6df3fc1
cleaned up code, removed unnecessary hooks
thesuhas Mar 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
[submodule "vendor/postgres-v14"]
path = vendor/postgres-v14
url = https://github.com/neondatabase/postgres.git
branch = REL_14_STABLE_neon
branch = thesuhas/move_lwlsn_neon_ext_v14
[submodule "vendor/postgres-v15"]
path = vendor/postgres-v15
url = https://github.com/neondatabase/postgres.git
branch = REL_15_STABLE_neon
branch = thesuhas/move_lwlsn_neon_ext_v15
[submodule "vendor/postgres-v16"]
path = vendor/postgres-v16
url = https://github.com/neondatabase/postgres.git
branch = REL_16_STABLE_neon
branch = thesuhas/move_lwlsn_neon_ext_v16
[submodule "vendor/postgres-v17"]
path = vendor/postgres-v17
url = https://github.com/neondatabase/postgres.git
branch = REL_17_STABLE_neon
branch = move_lwlsn_neon_ext
1 change: 1 addition & 0 deletions pgxn/neon/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ OBJS = \
libpagestore.o \
logical_replication_monitor.o \
neon.o \
neon_lwlc.o \
neon_pgversioncompat.o \
neon_perf_counters.o \
neon_utils.o \
Expand Down
5 changes: 4 additions & 1 deletion pgxn/neon/file_cache.c
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
#include "hll.h"
#include "bitmap.h"
#include "neon.h"
#include "neon_lwlc.h"
#include "neon_perf_counters.h"

#define CriticalAssert(cond) do if (!(cond)) elog(PANIC, "LFC: assertion %s failed at %s:%d: ", #cond, __FILE__, __LINE__); while (0)
Expand Down Expand Up @@ -999,7 +1000,9 @@ lfc_prefetch(NRelFileInfo rinfo, ForkNumber forknum, BlockNumber blkno,
LWLockRelease(lfc_lock);
return false;
}
lwlsn = GetLastWrittenLSN(rinfo, forknum, blkno);

lwlsn = neon_get_lwlsn(rinfo, forknum, blkno);

if (lwlsn > lsn)
{
elog(DEBUG1, "Skip LFC write for %d because LwLSN=%X/%X is greater than not_nodified_since LSN %X/%X",
Expand Down
3 changes: 3 additions & 0 deletions pgxn/neon/neon.c
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@

#include "extension_server.h"
#include "neon.h"
#include "neon_lwlc.h"
#include "control_plane_connector.h"
#include "logical_replication_monitor.h"
#include "unstable_extensions.h"
Expand Down Expand Up @@ -437,6 +438,8 @@ _PG_init(void)

pg_init_libpagestore();
pg_init_walproposer();
init_lwlc();

pagestore_smgr_init();
Custom_XLogReaderRoutines = NeonOnDemandXLogReaderRoutines;

Expand Down
Loading
Loading