Create reproducible needs.json - remove build timestamp information

[arwedus]

The needs builder does not create reproducible builds.
It should, to avoid issues with build caching systems.

Suggested change:

Remove the "created" entry, which contains a build timestamp, from needs.json altogether.

Alternative change:

sphinx HTML build become reproducible when the document does not contain any build time related information (like Page generated: |today| or a footer containing the build time formatted according to html_last_updated_fmt).

You could add the created entry to a needs.json only of html_last_updated_fmt is set to None. It being set to None is a good indicator that the author of the sphinx project cares for a reproducible documentation build.

You could also just add another needs_ option, which would make this more explicit. But it's a niche feature, and you already have a very long configuration page and I doubt that there is a person on this planet (except @danwos maybe 😆 ) who remembers all of them.

[danwos]

That's an interesting use case and thanks for the proposed solution with html_last_updated_fmt.

I'm struggling currently between a new needs config option (I know, another one....) and maybe an extra builder, which removes timestamps.

Using html_last_updated_fmt feels a little bit like black voodoo, as the option is not directly related to Sphinx-Needs and a user may not consider that an HTML-related option change will affect the needs.json-output.

I will discuss the different approaches with some users. But for sure in the end, there will be a solution.

[chrisjsewell]

I will discuss the different approaches with some users. But for sure in the end, there will be a solution.

@danwos any update descsion on this? I have no strong feelings either way 😄

[danwos]

I would add a new Sphinx-Needs option, maybe just named needs_reproducible_json?
Could be needs_no_timestamp as well, but I'm afraid that maybe one day other stuff needs to be removed or an order needs to be kept so that we need a generic option to activate all such kinds of features with one option.

[arwedus]

Thanks for tackling this! I agree needs_reproducible_json is a good name.

Do you also sort needs in the needs.json? -> just saw it, yes, they're sorted by dictionary keys.
Do you sort the IDs of links and backlinks? -> as far as I checked, no.

Or, put differently - is there any chance that the result of a build of the same input could result in different needs.json files, depending on the order of documents read? (thinking about https://reproducible-builds.org/docs/stable-inputs/ )

[chrisjsewell]

Do you also sort needs in the needs.json

Yep you beat me to it: the sort_keys flag is already set to true:

sphinx-needs/sphinx_needs/needsfile.py

Line 103 in f3725a0

json.dump(self.needs_list, f, indent=4, sort_keys=True)

Do you sort the IDs of links and backlinks?

backlinks are not output in the JSON:

sphinx-needs/sphinx_needs/needsfile.py

Line 70 in 40856b2

# also exclude back links for link types dynamically set by the user

links are not sorted as far as I can see

[arwedus]

Okay, but links appear in the order the author has given and that cannot change depending on the order of parallel reads, so I guess it's okay.

[chrisjsewell]

Yep let me know if you think of anything else non-reproducible, but for now I think we are good