-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Daily spellbook sub project #6027
Merged
Merged
Changes from 15 commits
Commits
Show all changes
43 commits
Select commit
Hold shift + click to select a range
eecd8c8
initiate subproject
jeff-dude d620175
Merge branch 'main' into daily-spellbook
jeff-dude 8c80c0e
move entire chainlink directory
jeff-dude 12ab410
add chainlink schemas in dbt project (for now)
jeff-dude 488030c
Merge branch 'main' into daily-spellbook
jeff-dude 9196a75
Drop variable
aalan3 2cb14d5
Add simple readme
aalan3 4244691
Update CI
aalan3 b8dda15
Increase parallelism
aalan3 b68534d
Fix formatting
aalan3 deeb29c
Add daily_spellbook to commit_manifest
aalan3 ab9850a
Temporarily allow manifest fetching to fail
aalan3 e9b3d7d
Test new CI setup
aalan3 09ac1f3
use correct name
aalan3 4951988
Use correct name
aalan3 ef1960b
Test for general spellbook as well
aalan3 b9fbd48
Add daily spellbook to CI
aalan3 9b288ed
Correct paths-ignore
aalan3 f744a22
Change names
aalan3 a5c2936
Fix if statement
aalan3 64efe0d
Set project_dir in a nicer way
aalan3 00ea092
Name
aalan3 9f20df9
Drop moved chainlink models
aalan3 b256f5d
Comment out seeds for now
aalan3 b7da5e3
Use correct paths
aalan3 0e1ff93
Drop old CI workflow
aalan3 833f36b
Add CI file for shared workflow
aalan3 684937f
Add job for Spellbook
aalan3 72b27d7
Add job for tokens
aalan3 e8969c2
Simplify manifest commit
aalan3 2d8ef12
Remove temp comment
aalan3 23ca6aa
initiate subproject
jeff-dude 311ae6f
move entire chainlink directory
jeff-dude 569c04e
add chainlink schemas in dbt project (for now)
jeff-dude ad996c3
Drop variable
aalan3 a1a7174
Add simple readme
aalan3 4074a73
Add CI for daily_spellbook
aalan3 c4820e9
There are not seeds yet
aalan3 852def9
Drop moved chainlink models from dbt_project.yml
aalan3 8241d60
Merge branch 'main' into daily-spellbook
aalan3 ac92a19
Merge branch 'daily-spellbook' of https://github.com/duneanalytics/sp…
jeff-dude e437a4d
remove extra packages
jeff-dude 6308f3d
Merge branch 'main' into daily-spellbook
aalan3 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
name: DBT Daily Spellbook run | ||
|
||
on: | ||
workflow_dispatch: | ||
pull_request: | ||
paths: | ||
- daily_spellbook/** | ||
- .github/workflows/daily_spellbook.yml | ||
- .github/workflows/dbt_run.yml | ||
|
||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.ref }} | ||
cancel-in-progress: true | ||
|
||
jobs: | ||
dbt-run: | ||
uses: ./.github/workflows/dbt_run.yml | ||
with: | ||
project: 'daily_spellbook' | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
## Daily Spellbook | ||
|
||
This is a DBT sub project for the the main models of Spellbook that runs on a daily candence. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
#Welcome to your Spellbook! | ||
|
||
name: "daily_spellbook" | ||
version: "1.0.0" | ||
config-version: 2 | ||
|
||
quoting: | ||
database: false | ||
schema: false | ||
identifier: false | ||
|
||
# This setting configures which "profile" dbt uses for this project. | ||
# profile: "spellbook-poc-tokens" | ||
profile: "spellbook-local" | ||
|
||
vars: | ||
DBT_ENV_CUSTOM_ENV_S3_BUCKET: "{{ env_var('DBT_ENV_CUSTOM_ENV_S3_BUCKET', 'local') }}" | ||
DBT_ENV_INCREMENTAL_TIME: "{{ env_var('DBT_ENV_INCREMENTAL_TIME', '1') }}" | ||
DBT_ENV_INCREMENTAL_TIME_UNIT: "{{ env_var('DBT_ENV_INCREMENTAL_TIME_UNIT', 'day') }}" | ||
|
||
# These configurations specify where dbt should look for different types of files. | ||
# You don't need to change these! | ||
model-paths: ["models", "../sources"] | ||
analysis-paths: ["analyses"] | ||
test-paths: ["tests"] | ||
seed-paths: ["seeds"] | ||
macro-paths: ["../macros"] | ||
snapshot-paths: ["snapshots"] | ||
|
||
target-path: "target" # directory which will store compiled SQL files | ||
clean-targets: # directories to be removed by `dbt clean` | ||
- "target" | ||
- "dbt_packages" | ||
|
||
# Configuring tests | ||
# Full documentation: https://docs.getdbt.com/reference/test-configs | ||
tests: | ||
daily_spellbook: | ||
+store_failures: true # store failures for all tests | ||
|
||
# Configuring seeds | ||
# Full documentation: https://docs.getdbt.com/reference/seed-configs | ||
# For configuring individual seeds (e.g. overriding column types) we recommend a yml file nested in the same folder as the seed | ||
# example: seeds/cryptopunks/schema.yml | ||
# TODO: uncomment when seeds are added | ||
# seeds: | ||
# daily_spellbook: | ||
# +schema: test_data | ||
|
||
# defining search order of macro invokation | ||
dispatch: | ||
- macro_namespace: dbt_utils | ||
search_order: ['trino_utils', 'dbt_utils'] | ||
|
||
# Configuring models | ||
# Full documentation: https://docs.getdbt.com/docs/configuring-models | ||
models: | ||
daily_spellbook: | ||
+post-hook: | ||
- sql: "{{ set_trino_session_property(is_materialized(model), 'writer_scaling_min_data_processed', model.config.get('writer_min_size', '500MB')) }}" | ||
transaction: true | ||
- sql: "{{ set_trino_session_property(is_materialized(model), 'task_scale_writers_enabled', false) }}" | ||
transaction: true | ||
- sql: "{{ optimize_spell(this, model.config.materialized) }}" | ||
transaction: true | ||
- sql: "{{ mark_as_spell(this, model.config.materialized) }}" | ||
transaction: true | ||
+materialized: view | ||
+schema: no_schema # this should be overriden in model specific configs | ||
+view_security: invoker | ||
|
||
chainlink: | ||
+schema: chainlink | ||
arbitrum: | ||
+schema: chainlink_arbitrum | ||
avalanche_c: | ||
+schema: chainlink_avalanche_c | ||
base: | ||
+schema: chainlink_base | ||
bnb: | ||
+schema: chainlink_bnb | ||
ethereum: | ||
+schema: chainlink_ethereum | ||
fantom: | ||
+schema: chainlink_fantom | ||
gnosis: | ||
+schema: chainlink_gnosis | ||
optimism: | ||
+schema: chainlink_optimism | ||
polygon: | ||
+schema: chainlink_polygon | ||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just noting that we will want to remove these soon and put in models. just a lot of models to do. we can wait until the team refactors the code into macros (already underway)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah agree, was surprised to see these in the project file