Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Daily spellbook sub project #6027

Merged
merged 43 commits into from
May 30, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
eecd8c8
initiate subproject
jeff-dude May 24, 2024
d620175
Merge branch 'main' into daily-spellbook
jeff-dude May 24, 2024
8c80c0e
move entire chainlink directory
jeff-dude May 24, 2024
12ab410
add chainlink schemas in dbt project (for now)
jeff-dude May 28, 2024
488030c
Merge branch 'main' into daily-spellbook
jeff-dude May 28, 2024
9196a75
Drop variable
aalan3 May 29, 2024
2cb14d5
Add simple readme
aalan3 May 29, 2024
4244691
Update CI
aalan3 May 29, 2024
b8dda15
Increase parallelism
aalan3 May 29, 2024
b68534d
Fix formatting
aalan3 May 29, 2024
deeb29c
Add daily_spellbook to commit_manifest
aalan3 May 29, 2024
ab9850a
Temporarily allow manifest fetching to fail
aalan3 May 29, 2024
e9b3d7d
Test new CI setup
aalan3 May 29, 2024
09ac1f3
use correct name
aalan3 May 29, 2024
4951988
Use correct name
aalan3 May 29, 2024
ef1960b
Test for general spellbook as well
aalan3 May 29, 2024
b9fbd48
Add daily spellbook to CI
aalan3 May 29, 2024
9b288ed
Correct paths-ignore
aalan3 May 29, 2024
f744a22
Change names
aalan3 May 29, 2024
a5c2936
Fix if statement
aalan3 May 29, 2024
64efe0d
Set project_dir in a nicer way
aalan3 May 29, 2024
00ea092
Name
aalan3 May 29, 2024
9f20df9
Drop moved chainlink models
aalan3 May 29, 2024
b256f5d
Comment out seeds for now
aalan3 May 29, 2024
b7da5e3
Use correct paths
aalan3 May 29, 2024
0e1ff93
Drop old CI workflow
aalan3 May 29, 2024
833f36b
Add CI file for shared workflow
aalan3 May 29, 2024
684937f
Add job for Spellbook
aalan3 May 29, 2024
72b27d7
Add job for tokens
aalan3 May 29, 2024
e8969c2
Simplify manifest commit
aalan3 May 29, 2024
2d8ef12
Remove temp comment
aalan3 May 29, 2024
23ca6aa
initiate subproject
jeff-dude May 24, 2024
311ae6f
move entire chainlink directory
jeff-dude May 24, 2024
569c04e
add chainlink schemas in dbt project (for now)
jeff-dude May 28, 2024
ad996c3
Drop variable
aalan3 May 29, 2024
a1a7174
Add simple readme
aalan3 May 29, 2024
4074a73
Add CI for daily_spellbook
aalan3 May 29, 2024
c4820e9
There are not seeds yet
aalan3 May 29, 2024
852def9
Drop moved chainlink models from dbt_project.yml
aalan3 May 29, 2024
8241d60
Merge branch 'main' into daily-spellbook
aalan3 May 29, 2024
ac92a19
Merge branch 'daily-spellbook' of https://github.com/duneanalytics/sp…
jeff-dude May 29, 2024
e437a4d
remove extra packages
jeff-dude May 29, 2024
6308f3d
Merge branch 'main' into daily-spellbook
aalan3 May 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/workflows/commit_manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
runs-on: [ self-hosted, linux, spellbook-trino-ci ]
strategy:
matrix:
project: [ 'tokens', 'spellbook' ]
project: [ 'tokens', 'spellbook', 'daily_spellbook' ]
max-parallel: 1

steps:
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/daily_spellbook.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: DBT Daily Spellbook run

on:
workflow_dispatch:
pull_request:
paths:
- daily_spellbook/**
- .github/workflows/daily_spellbook.yml
- .github/workflows/dbt_run.yml

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
dbt-run:
uses: ./.github/workflows/dbt_run.yml
with:
project: 'daily_spellbook'

3 changes: 3 additions & 0 deletions daily_spellbook/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Daily Spellbook

This is a DBT sub project for the the main models of Spellbook that runs on a daily candence.
91 changes: 91 additions & 0 deletions daily_spellbook/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#Welcome to your Spellbook!

name: "daily_spellbook"
version: "1.0.0"
config-version: 2

quoting:
database: false
schema: false
identifier: false

# This setting configures which "profile" dbt uses for this project.
# profile: "spellbook-poc-tokens"
profile: "spellbook-local"

vars:
DBT_ENV_CUSTOM_ENV_S3_BUCKET: "{{ env_var('DBT_ENV_CUSTOM_ENV_S3_BUCKET', 'local') }}"
DBT_ENV_INCREMENTAL_TIME: "{{ env_var('DBT_ENV_INCREMENTAL_TIME', '1') }}"
DBT_ENV_INCREMENTAL_TIME_UNIT: "{{ env_var('DBT_ENV_INCREMENTAL_TIME_UNIT', 'day') }}"

# These configurations specify where dbt should look for different types of files.
# You don't need to change these!
model-paths: ["models", "../sources"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["../macros"]
snapshot-paths: ["snapshots"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_packages"

# Configuring tests
# Full documentation: https://docs.getdbt.com/reference/test-configs
tests:
daily_spellbook:
+store_failures: true # store failures for all tests

# Configuring seeds
# Full documentation: https://docs.getdbt.com/reference/seed-configs
# For configuring individual seeds (e.g. overriding column types) we recommend a yml file nested in the same folder as the seed
# example: seeds/cryptopunks/schema.yml
# TODO: uncomment when seeds are added
# seeds:
# daily_spellbook:
# +schema: test_data

# defining search order of macro invokation
dispatch:
- macro_namespace: dbt_utils
search_order: ['trino_utils', 'dbt_utils']

# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models
models:
daily_spellbook:
+post-hook:
- sql: "{{ set_trino_session_property(is_materialized(model), 'writer_scaling_min_data_processed', model.config.get('writer_min_size', '500MB')) }}"
transaction: true
- sql: "{{ set_trino_session_property(is_materialized(model), 'task_scale_writers_enabled', false) }}"
transaction: true
- sql: "{{ optimize_spell(this, model.config.materialized) }}"
transaction: true
- sql: "{{ mark_as_spell(this, model.config.materialized) }}"
transaction: true
+materialized: view
+schema: no_schema # this should be overriden in model specific configs
+view_security: invoker

chainlink:
+schema: chainlink
arbitrum:
+schema: chainlink_arbitrum
avalanche_c:
+schema: chainlink_avalanche_c
base:
+schema: chainlink_base
bnb:
+schema: chainlink_bnb
ethereum:
+schema: chainlink_ethereum
fantom:
+schema: chainlink_fantom
gnosis:
+schema: chainlink_gnosis
optimism:
+schema: chainlink_optimism
polygon:
+schema: chainlink_polygon
Comment on lines +72 to +91
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just noting that we will want to remove these soon and put in models. just a lot of models to do. we can wait until the team refactors the code into macros (already underway)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah agree, was surprised to see these in the project file

File renamed without changes.
Loading
Loading