needextend shall support dynamic functions

[arwedus]

observed behavior

Hi @danwos , when I try the following code, I get an error:

.. needextend:: SYSFCT_Foo
    :+implements: [[get_matching_need_ids("REQ_SYS_Foo_")]]

With the following dynamic function definition:

def get_matching_need_ids(app, need, needs, id_prefix="") -> List[str]:
    """Get a list of needs matching id_prefix

    Example Usage::

        :derives: [[get_matching_need_ids("REQ_SYS_TSR_")]]

    Args:
        app: app object (sphinx-needs dynamic functions API parameter)
        need: current need (sphinx-needs dynamic functions API parameter)
        needs: dictionary of all needs (sphinx-needs dynamic functions API parameter)
        id_prefix: needs ID filter. Only returns needs which's ID starts with id_prefix.

    Returns:
        List[str]: list of ids for needs matching all filter criteria
    """
    filtered_needs = []
    for need_id, need_entry in needs.items():
        if id_prefix and need_id.startswith(id_prefix):
            filtered_needs.append(need_id)
    return filtered_needs

I get the following error:

Exception occurred while building, starting debugger:
Traceback (most recent call last):
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx/cmd/build.py", line 281, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx/application.py", line 347, in build
    self.builder.build_update()
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 310, in build_update
    self.build(to_build,
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 376, in build
    self.write(docnames, list(updated_docnames), method)
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 568, in write
    self._write_parallel(sorted(docnames),
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx_monkeypatch/__init__.py", line 207, in _write_parallel
    doctree = self.env.get_and_resolve_doctree(docname, self)
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 591, in get_and_resolve_doctree
    self.apply_post_transforms(doctree, docname)
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 642, in apply_post_transforms
    self.events.emit('doctree-resolved', doctree, docname)
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx/events.py", line 94, in emit
    results.append(listener.handler(self.app, *args))
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx_needs/directives/need.py", line 393, in process_need_nodes
    process_needextend(app, doctree, fromdocname)
  File "/home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx_needs/directives/needextend.py", line 142, in process_needextend
    if found_need["id"] not in env.needs_all_needs[ref_need][f"{option_name}_back"]:
KeyError: '[[get_matching_need_ids(filter="id.startswith(\'REQ_SYS_Foo_\')")]]'
> /home/arwedus/workspace/.sphinx-venv/lib/python3.8/site-packages/sphinx_needs/directives/needextend.py(142)process_needextend()
-> if found_need["id"] not in env.needs_all_needs[ref_need][f"{option_name}_back"]:

expected behavior

Dynamic functions wrapped in [[]] shall work in needextend options just like in need directive options.

possible root cause

The code in needextend.py treats everything that is found in a link_name field like an ID (

sphinx-needs/sphinx_needs/directives/needextend.py

Line 175 in 11d6b1b

if option in link_names:

) and doesn't resolve dynamic functions first. It should read in links like the need directive.

[chrisjsewell]

Heya @arwedus @danwos

Given that you can already use a filter to select multiple needs, e.g.

.. needextend:: id.startswith("SYSFCT_")

Rather than dynamic functions, do you think it would be more "consistent" to also allow for filtering link selections, e.g.

.. needextend:: SYSFCT_Foo
    :+implements: id.startswith("REQ_SYS_Foo_")

what do you think?

[arwedus]

sure, would be straightforward. In the end, I would expect needextend-directive options to "behave" as identical as possible to need-directive options, but I'll take whatever you can implement quickly :D

[danwos]

I'm not sure, will be tough to check if the given value for +implements is just a string or shall be taken as a filter.
I would provide a predefined dynamic function so that you get:

.. needextend:: SYSFCT_Foo
   :+implements: [[ filter('id.startswith("REQ_SYS_Foo_"')) ]]

[arwedus]

@danwos, @chrisjsewell: I'll give this a try, but please before creating a new release, update the documentation on the needextend directive, e.g. add something like this as an example:

.. needextend:: SYSFCT_Foo
   :+implements: [[filter('id.startswith("REQ_SYS_Foo_")')]]

[chrisjsewell]

update the documentation

Ah oops thanks for the reminder!

[arwedus]

@chrisjsewell: I am having a hard time getting this feature to work.

First try:

.. needextend:: SYSFCT_RepresentDynObj_6BA49B645C114c1bAEAE1DDB69D3F0D4
  :+implements: [[filter('id.startswith("REQ_SYS_MJS_ENVREP_OBJ_")')]]

did not work, throw an error:

Unknown dynamic sphinx-needs function: filter. Found in need: SYSFCT_RepresentDynObj_6BA49B645C114c1bAEAE1DDB69D3F0D4

2nd try:

.. needextend:: SYSFCT_RepresentDynamicObjects_6BA49B645C114c1bAEAE1DDB69D3F0D4
  :+implements: [[get_matching_need_ids("REQ_SYS_MJS_EnvRep_OBJ_")]],
                REQ_SYS_MJS_EnvRep_TD_0010293

This works, but the extra requirement (REQ_SYS_MJS_EnvRep_TD_0010293) did not show up in the generated output.

With a first try with spaces inside [[ ]], I also got this error once, but it wasn't reproducible:

Parsing function string failed for need SYSFCT_RepresentDynObj_6BA49B645C114c1bAEAE1DDB69D3F0D4:  get_matching_need_ids("REQ_SYS_MJS_EnvRep_OBJ_") . unexpected indent (<unknown>, line 1)

What worked:

.. needextend:: SYSFCT_RepresentDynObj_6BA49B645C114c1bAEAE1DDB69D3F0D4
  :+implements: [[get_matching_need_ids("REQ_SYS_MJS_EnvRep_OBJ_")]]

First of all: Cool that this feature is implemented now, thanks!

For me, it is kind of un-intuitive that the filter function is treated differently than a dynamic sphinx-needs function, as it does the same things. Is there a way to treat filter just as other dynamic functions that return needs?

And second, I'd assume that it would be possible to mix direct need references with dynamic function calls for need references, but it seems not to be the case. Is this intentional? Could you add this feature, too?