Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE]: detect uses of Spark logging and configuration #1805

Closed
1 task done
vsevolodstep-db opened this issue May 30, 2024 · 0 comments · Fixed by #1808
Closed
1 task done

[FEATURE]: detect uses of Spark logging and configuration #1805

vsevolodstep-db opened this issue May 30, 2024 · 0 comments · Fixed by #1808
Assignees
@vsevolodstep-db
Labels
migrate/code Abstract Syntax Trees and other dark magic

Comments

@vsevolodstep-db
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Problem statement

Spark JVM access is not supported on UC Shared clusters. Some workloads rely on it to do logging, which is not supported. The correct way is to use python standard loggers and configure Spark loggers using a cluster-level conf.

Proposed Solution

New linter checks

Additional Context

No response
@vsevolodstep-db vsevolodstep-db added enhancement New feature or request needs-triage labels May 30, 2024
@vsevolodstep-db vsevolodstep-db self-assigned this May 30, 2024
@github-project-automation github-project-automation bot moved this to Triage in UCX May 30, 2024
@vsevolodstep-db vsevolodstep-db changed the title [FEATURE]: detect uses of Spark logging [FEATURE]: detect uses of Spark logging and configuration May 30, 2024
@vsevolodstep-db vsevolodstep-db added migrate/code Abstract Syntax Trees and other dark magic and removed enhancement New feature or request needs-triage labels May 30, 2024
This was referenced May 30, 2024
Open
Merged
@nfx nfx closed this as completed in #1808 May 30, 2024
nfx pushed a commit that referenced this issue May 30, 2024
@vsevolodstep-db
Add linters to check for spark logging and configuration access (#1808)
612d3e2 
## Changes
Add new linters to check for uses of spark logging, `sc.conf` access and
`rdd.mapPartitions`

### Linked issues
Resolves [#1805](#1805),
enhances [#1603](#1603),
[#1604](#1604)

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [x] manually tested
- [x] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)
@github-project-automation github-project-automation bot moved this from Triage to Archive in UCX May 30, 2024
nfx pushed a commit that referenced this issue May 30, 2024
@vsevolodstep-db @nfx
Add linters to check for spark logging and configuration access (#1808)
21b8ef8 
## Changes
Add new linters to check for uses of spark logging, `sc.conf` access and
`rdd.mapPartitions`

### Linked issues
Resolves [#1805](#1805),
enhances [#1603](#1603),
[#1604](#1604)

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [x] manually tested
- [x] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vsevolodstep-db vsevolodstep-db

Labels
migrate/code Abstract Syntax Trees and other dark magic
Projects
UCX
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add linters to check for spark logging and configuration access databrickslabs/ucx
1 participant
@vsevolodstep-db