Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite asset event registration #47677

Merged
merged 10 commits into from
Mar 13, 2025

Conversation

uranusjr
Copy link
Member

@uranusjr uranusjr commented Mar 12, 2025

There are many issues in the original implementation when you send something that's not a simple asset, I ended up almost rewritten the whole thing...

Now, the task runner collects everything the user writes into outlet_events, and send all of them with the task's outlets as declared by the user verbatim to the API server. The server does all the resolution and filtering instead.

Fix #46958 (in combination with #47659).

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:task-sdk labels Mar 12, 2025
@uranusjr uranusjr changed the title Acess events by ref to correctly emit events Rewrite asset event registration Mar 12, 2025
There are many issues in the original implementation when you send
something that's not a simple asset, I ended up almost rewritten the
whole thing...

Now, the task runner collects everything the user writes into
outlet_events, and send all of them with the task's outlets as declared by the
user verbatim to the API server. The server does all the resolution and
filtering instead.
@uranusjr uranusjr force-pushed the sdk-implement-outlet-events branch from aac0864 to 222bb17 Compare March 12, 2025 17:07
The task runner now emits slightly different data; extra is no longer
eagerly resolved; the server side fills in the default automatically.
Fixtures using asset refs now work correctly.

The API server side tests are vastly improved to cover more kinds of
data, including where events against other aliases are not incorrectly
picked up when processing events originated from an alias.
@uranusjr uranusjr force-pushed the sdk-implement-outlet-events branch from e6d66e3 to c2f7f16 Compare March 12, 2025 18:25
The 'aliases' and 'source_alias_names' basically serve the same purpose,
but we use each for a different section of the code. They can become
one.
Exceptions are now raised if an event is raised against inactive assets.
@Lee-W Lee-W self-requested a review March 13, 2025 06:29
@uranusjr uranusjr marked this pull request as ready for review March 13, 2025 07:02
This does not really matter in practice since inactive assets should
have been caught before the task runs, but in case any are found, we can
still try to trigger as many valid events as possible before failing the
task.
@uranusjr uranusjr requested a review from Lee-W March 13, 2025 08:22
@uranusjr
Copy link
Member Author

wtf is going on with those kub tests

@uranusjr uranusjr merged commit 2fa9230 into apache:main Mar 13, 2025
88 of 89 checks passed
@uranusjr uranusjr deleted the sdk-implement-outlet-events branch March 13, 2025 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:API Airflow's REST/HTTP API area:task-sdk
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Asset event is not created for asset decorator Dag
4 participants