feat(ingestion) slack source v2 - now ingests all user and channels #12795
Conversation
…o platform resources
github-actions
bot
added
the
ingestion
Codecov ReportAttention: Patch coverage is ✅ All tests successful. No failed tests found.
❌ Your patch check has failed because the patch coverage (0.00%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage. 📢 Thoughts on this report? Let us know! 🚀 New features to boost your workflow:
|
datahub-cyborg
bot
added
the
needs-review
| blob=json.dumps(value.to_obj()).encode("utf-8"), | ||
| contentType=SerializedValueContentTypeClass.JSON, | ||
| schemaType=SerializedValueSchemaTypeClass.PEGASUS, | ||
| schemaRef=schema_type, |
There was a problem hiding this comment.
I thought we had helpers for this stuff in PlatformResourceInfo or SerializedResourceValue - is there a reason we're not using that stuff
| yield mcp.as_workunit() | ||
| else: | ||
| logger.error("Failed to fetch team information") | ||
| raise Exception("Failed to fetch team information") |
There was a problem hiding this comment.
use structured error reporting https://acryldata.notion.site/Error-reporting-in-ingestion-5343cc6ea0c84633b38070d1a409c569?pvs=74
| ) and user_obj.real_name: | ||
| mutation_required = True | ||
| corpuser_editable_info.displayName = user_obj.real_name | ||
| if mutation_required: |
There was a problem hiding this comment.
can we make a copy of the original corpuser_editable_info, and then we know a mutation is required if the two objects aren't equal
| platform=DATA_PLATFORM_SLACK_URN, instance=self.id | ||
| ) | ||
|
|
||
| def update_with_team_info(self, team_info: dict) -> None: |
There was a problem hiding this comment.
why wouldn't we pass this stuff in as part of the SlackInstance constructor?
or alternatively, classmethod def from_slack_team_info(team_info: dict) -> SlackInstance
datahub-cyborg
bot
added
pending-submitter-response
Checklist