Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In dependency graph, support dependencies other than notebooks #1343

Closed

Conversation

ericvergnaud
Copy link
Contributor

Changes

Refactor DependencyGraph
Introduce 'DependencyLoader' and utility classes

Linked issues

#1202

Resolves #1342

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs ucx ...
  • added a new workflow
  • modified existing workflow: ...
  • added a new table
  • modified existing table: ...

Tests

  • manually tested
  • added unit tests
  • added integration tests
  • verified on staging environment (screenshot attached)

add sample with RUN cell
fix issue where non-PI comments preceding language PI would prevent language PI detection
* main:
  remove `isort` (databrickslabs#1280)
  Addressed Issue with Disabled Feature in certain regions (databrickslabs#1275)
  Improve documentation (databrickslabs#1162)
  Add roadmap workflows and tasks to Table Migration Workflow document (databrickslabs#1274)
  Fix integration test with new DeployedWorkflows (databrickslabs#1250)
  Document troubleshooting guide (databrickslabs#1226)
  Split `DeployedWorkflows` out of `WorkflowsDeployment` (databrickslabs#1248)
  Inject `_TASKS` via constructor to `WorkflowsDeployment` instead of a global variable (databrickslabs#1247)
  Decouple `InstallState` from `WorkspaceDeployment` constructor
  Add document for table migration workflow (databrickslabs#1229)
  Decouple `InstallState` from `WorkflowsDeployment` constructor (databrickslabs#1246)
* main:
  Build notebook dependency graph for `%run` cells (databrickslabs#1279)

# Conflicts:
#	src/databricks/labs/ucx/source_code/notebook.py
#	tests/unit/source_code/test_notebook.py
ericvergnaud and others added 23 commits April 8, 2024 13:56
* main:
  Adding CSV, JSON and include path in mounts (databrickslabs#1329)
  Add missing step sync-workspace-info (databrickslabs#1330)
  disable annotation-unchecked mypy warning (databrickslabs#1331)
  Use service factory to resolve object dependencies (databrickslabs#1209)
…f-notebook-dependencies

* detect-dbutils.notebook.run:
  address code review comments
  move PythonLinter to python_linter file renamed from astlinter
  change model to avoid clashing ownership patterns
  address code review comments
  Adding CSV, JSON and include path in mounts (databrickslabs#1329)
  Add missing step sync-workspace-info (databrickslabs#1330)
  disable annotation-unchecked mypy warning (databrickslabs#1331)
  Use service factory to resolve object dependencies (databrickslabs#1209)

# Conflicts:
#	src/databricks/labs/ucx/source_code/notebook.py
#	tests/unit/source_code/test_notebook.py
* main:
  Integrate detection of notebook dependencies (databrickslabs#1338)

# Conflicts:
#	src/databricks/labs/ucx/source_code/notebook.py
#	src/databricks/labs/ucx/source_code/notebook_migrator.py
#	tests/unit/source_code/test_notebook.py
#	tests/unit/source_code/test_notebook_migrator.py
@ericvergnaud ericvergnaud requested review from a team and FastLee April 10, 2024 13:37
@ericvergnaud ericvergnaud deleted the detect-import-dependencies branch April 10, 2024 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE]: Support not only Notebooks in dependency graph
1 participant