Add ScalarParameters replacing GenKW in parameter config #10095

xjules · 2025-02-18T09:42:59Z

Issue
Resolves #10088

Approach
- transformations become dataclases -> TODO: pydantic
- ScalarParameter represents a single parameter instance with a
datasource (DESIGN_MATRIX, SAMPLED)
- all ScalarParameter instances are bound by
ScalarParameters(ParameterConfig) and are stored for all realizations
in a single pl.Dataframe (parquet) file.
- API to load / save scalar parameters is in local_ensemble load|save_param_scalar
- DESIGN_MATRIX now creates only ScalarParameters instances and
modifies the ScalarParameterc Config.

PR title captures the intent of the changes, and is fitting for release notes.
Added appropriate release note label
Commit history is consistent and clean, in line with the contribution guidelines.
Make sure unit tests pass locally after every commit (git rebase -i main --exec 'just rapid-tests')

When applicable

When there are user facing changes: Updated documentation
New behavior or changes to existing untested code: Ensured that unit tests are added (See Ground Rules).
Large PR: Prepare changes in small commits for more convenient review
Bug fix: Add regression test for the bug
Bug fix: Create Backport PR to latest release

codspeed-hq · 2025-02-18T10:06:00Z

CodSpeed Performance Report

Merging #10095 will not alter performance

_{Comparing xjules:draft_paramcfg (33ef660) with main (cd27051)}

Summary

✅ 25 untouched benchmarks

xjules · 2025-02-24T14:50:49Z

src/ert/config/scalar_parameter.py

+
+
+@dataclass
+class TransUnifSettings:


~~TODO: All distributions should contain LOG10 property~~
This will be handled directly by the type of distribution.

xjules · 2025-02-24T14:52:46Z

src/ert/config/scalar_parameter.py

+            if group_name not in transformed_dict:
+                transformed_dict[group_name] = {}
+            transformed_dict[group_name][param_name] = transformed_value
+        # todo log handling when distribution requires it


TODO: here is also the place that we need to handle template_file / output_file!

First draft of the treatment of template / output file is in place.

xjules · 2025-02-24T14:58:41Z

src/ert/config/scalar_parameter.py

+            ]
+        )
+
+    def save_parameters(


TODO: redesign API for save_parameters?

xjules · 2025-03-07T14:59:00Z

tests/ert/unit_tests/sensitivity_analysis/test_design_matrix.py

-                )
-            )
+# TODO refactor this test
+# @pytest.mark.parametrize(


Get rid of most of it

The best tests are the ones that are removed :)

- transformations become dataclases -> TODO: pydantic - ScalarParameter represents a single parameter instance with a datasource (DESIGN_MATRIX, SAMPLED) - all ScalarParameter instances are bound by ScalarParameters(ParameterConfig) and are stored for all realizations in a single pl.Dataframe (parquet) file. - API to load / save scalar parameters is in local_ensemble load|save_param_scalar - DESIGN_MATRIX now creates only ScalarParameters instances and modifies the ScalarParameterc Config. - Reimplement load_all_gen_kwn_data

codecov-commenter · 2025-03-10T17:21:04Z

❌ 13 Tests Failed:

Tests completed	Failed	Passed	Skipped
440	13	427	24

View the top 3 failed test(s) by shortest run time

tests/ert/ui_tests/gui/test_csv_export.py::test_csv_export[*]

Stack Traces | 0.002s run time

item = <Function test_csv_export[*]>

    @pytest.hookimpl(wrapper=True, tryfirst=True)
    def pytest_runtest_setup(item):
        """
        Hook called after before test setup starts, to start capturing exceptions
        as early as possible.
        """
        capture_enabled = _is_exception_capture_enabled(item)
        if capture_enabled:
            item.qt_exception_capture_manager = _QtExceptionCaptureManager()
            item.qt_exception_capture_manager.start()
>       result = yield

.venv/lib/python3.12.../site-packages/pytestqt/plugin.py:180: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../ui_tests/gui/conftest.py:120: in _esmda_run
    log_paths = list(Path(config.analysis_config.log_path).iterdir())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('.../pytest-0/test-data0/update_log')

    def iterdir(self):
        """Yield path objects of the directory contents.
    
        The children are yielded in arbitrary order, and the
        special entries '.' and '..' are not included.
        """
>       for name in os.listdir(self):
E       FileNotFoundError: [Errno 2] No such file or directory: '.../pytest-0/test-data0/update_log'

.../hostedtoolcache/Python/3.12.9.../x64/lib/python3.12/pathlib.py:1056: FileNotFoundError

tests/ert/ui_tests/gui/test_main_window.py::test_that_load_results_manually_can_be_run_after_esmda

Stack Traces | 0.002s run time

item = <Function test_that_load_results_manually_can_be_run_after_esmda>

    @pytest.hookimpl(wrapper=True, tryfirst=True)
    def pytest_runtest_setup(item):
        """
        Hook called after before test setup starts, to start capturing exceptions
        as early as possible.
        """
        capture_enabled = _is_exception_capture_enabled(item)
        if capture_enabled:
            item.qt_exception_capture_manager = _QtExceptionCaptureManager()
            item.qt_exception_capture_manager.start()
>       result = yield

.../ert/ert/.venv/lib/python3.12.../site-packages/pytestqt/plugin.py:180: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../ui_tests/gui/conftest.py:120: in _esmda_run
    log_paths = list(Path(config.analysis_config.log_path).iterdir())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('.../pytest-0/test-data4/update_log')

    def iterdir(self):
        """Yield path objects of the directory contents.
    
        The children are yielded in arbitrary order, and the
        special entries '.' and '..' are not included.
        """
>       for name in os.listdir(self):
E       FileNotFoundError: [Errno 2] No such file or directory: '.../pytest-0/test-data4/update_log'

.../hostedtoolcache/Python/3.12.9.../x64/lib/python3.12/pathlib.py:1056: FileNotFoundError

tests/ert/ui_tests/gui/test_main_window.py::test_that_the_manage_experiments_tool_can_be_used

Stack Traces | 0.002s run time

item = <Function test_that_the_manage_experiments_tool_can_be_used>

    @pytest.hookimpl(wrapper=True, tryfirst=True)
    def pytest_runtest_setup(item):
        """
        Hook called after before test setup starts, to start capturing exceptions
        as early as possible.
        """
        capture_enabled = _is_exception_capture_enabled(item)
        if capture_enabled:
            item.qt_exception_capture_manager = _QtExceptionCaptureManager()
            item.qt_exception_capture_manager.start()
>       result = yield

.../ert/ert/.venv/lib/python3.12.../site-packages/pytestqt/plugin.py:180: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../ui_tests/gui/conftest.py:120: in _esmda_run
    log_paths = list(Path(config.analysis_config.log_path).iterdir())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('.../pytest-0/test-data4/update_log')

    def iterdir(self):
        """Yield path objects of the directory contents.
    
        The children are yielded in arbitrary order, and the
        special entries '.' and '..' are not included.
        """
>       for name in os.listdir(self):
E       FileNotFoundError: [Errno 2] No such file or directory: '.../pytest-0/test-data4/update_log'

.../hostedtoolcache/Python/3.12.9.../x64/lib/python3.12/pathlib.py:1056: FileNotFoundError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

jonathan-eq

I don't think the tests are all that happy 🌧️

jonathan-eq · 2025-03-11T13:22:07Z

src/ert/analysis/_es_update.py

+    if isinstance(config_node, ScalarParameters):
+        return config_node.load_parameters_to_update(ensemble, iens_active_index)
+    dataset = config_node.load_parameters(ensemble, param_group, iens_active_index)
+    assert isinstance(dataset, np.ndarray), "dataset is not an numpy array"


Are we keeping this assertion?

jonathan-eq · 2025-03-11T13:22:42Z

src/ert/config/design_matrix.py


 if TYPE_CHECKING:
-    from ert.config import ParameterConfig
+    pass


Maybe we can remove this TYPE_CHECKING

jonathan-eq · 2025-03-11T13:26:50Z

src/ert/config/ensemble_config.py

@@ -180,12 +182,12 @@ def hasNodeGenData(self, key: str) -> bool:
        config = self.response_configs["gen_data"]
        return key in config.keys

+    # TODO: This might not be needed but it retrieves the group names for genkw config


Is this still needed?

jonathan-eq · 2025-03-11T13:35:27Z

tests/ert/unit_tests/sensitivity_analysis/test_design_matrix.py

-                )
-            )
+# TODO refactor this test
+# @pytest.mark.parametrize(


The best tests are the ones that are removed :)

xjules self-assigned this Feb 18, 2025

xjules added the release-notes:refactor PR changes code without changing ANY (!) behavior. label Feb 18, 2025

xjules force-pushed the draft_paramcfg branch from 8d7051d to 33ef660 Compare February 19, 2025 15:18

xjules mentioned this pull request Feb 24, 2025

Refactor data source for GenKwConfig #9079

Closed

xjules commented Feb 24, 2025

View reviewed changes

xjules force-pushed the draft_paramcfg branch from 80c5e43 to 1db64b4 Compare February 24, 2025 14:55

xjules commented Feb 24, 2025

View reviewed changes

src/ert/config/scalar_parameter.py

]

)

def save_parameters(

Copy link

Contributor Author

xjules Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: redesign API for save_parameters?

xjules mentioned this pull request Feb 28, 2025

Migrate GenKW template feature into global template option (substitutions) #10180

Open

xjules force-pushed the draft_paramcfg branch 2 times, most recently from 096cbc4 to ebef6f6 Compare March 4, 2025 09:44

xjules changed the title ~~Add draft for parameter config~~ Add ScalarParameters replacing GenKW in parameter config Mar 7, 2025

xjules commented Mar 7, 2025

View reviewed changes

xjules force-pushed the draft_paramcfg branch 2 times, most recently from 80bc01b to ddcf1e0 Compare March 8, 2025 18:30

xjules force-pushed the draft_paramcfg branch from e4124c5 to 68d4b20 Compare March 10, 2025 12:40

xjules added 2 commits March 10, 2025 16:25

Make sure to create scalars only if there were gen_kw configs

3313f2d

Make correct real_nr type for parameter config

e244aa1

xjules and others added 3 commits March 10, 2025 23:59

Fix esmda parameter merging

cbdda0b

test_libres_facade update snapshot

ab8400b

Drop index column

8d14295

jonathan-eq reviewed Mar 11, 2025

View reviewed changes

xjules and others added 3 commits March 11, 2025 15:38

Improve scalar_parameters API

503915f

Add snapshot for polars parameters

53459f7

scalarparameters: provide export to xr.Dataset

c2bc290

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ScalarParameters replacing GenKW in parameter config #10095

Add ScalarParameters replacing GenKW in parameter config #10095

xjules commented Feb 18, 2025 •

edited

Loading

codspeed-hq bot commented Feb 18, 2025 •

edited

Loading

xjules Feb 24, 2025 •

edited

Loading

xjules Feb 24, 2025

xjules Feb 28, 2025

xjules Feb 24, 2025

xjules Mar 7, 2025

jonathan-eq Mar 11, 2025

codecov-commenter commented Mar 10, 2025

jonathan-eq left a comment

jonathan-eq Mar 11, 2025

jonathan-eq Mar 11, 2025

jonathan-eq Mar 11, 2025

jonathan-eq Mar 11, 2025



		@dataclass
		class TransUnifSettings:

Add ScalarParameters replacing GenKW in parameter config #10095

Are you sure you want to change the base?

Add ScalarParameters replacing GenKW in parameter config #10095

Conversation

xjules commented Feb 18, 2025 • edited Loading

When applicable

codspeed-hq bot commented Feb 18, 2025 • edited Loading

Merging #10095 will not alter performance

Summary

xjules Feb 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Mar 10, 2025

❌ 13 Tests Failed:

jonathan-eq left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xjules commented Feb 18, 2025 •

edited

Loading

codspeed-hq bot commented Feb 18, 2025 •

edited

Loading

xjules Feb 24, 2025 •

edited

Loading