feat: language reference links and examples in docstrings

[david-christiansen]

This PR adds a canonical syntax for linking to sections in the language reference along with formatting of examples in docstrings according to the docstring style guide.

Docstrings are now pre-processed as follows:

• Output included as part of examples is shown with leading line comment indicators in hovers

• URLs of the form lean-manual://section/section-id are rewritten to links that point at the corresponding section in the Lean reference manual. The reference manual's base URL is configured when Lean is built and can be overridden with the LEAN_MANUAL_ROOT environment variable. This way, releases can point documentation links to the correct snapshot, and users can use their own, e.g. for offline reading.

Manual URLs in docstrings are validated when the docstring is added. The presence of a URL starting with lean-manual:// that is not a syntactically valid section link causes the docstring to be rejected. This allows for future extensibility to the set of allowed links. There is no validation that the linked-to section actually exists. To provide the best possible error messages in case of validation failures, Lean.addDocString now takes a TSyntax ``docComment instead of a string; clients should adapt by removing the step that extracts the string, or by calling the lower-level addDocStringCore in cases where the docstring in question is obtained from the environment and has thus already had its links validated.

A stage0 update is required to make the documentation site configurable at build time and for releases. A local commit on top of a stage0 update that will be sent in a followup PR includes the configurable reference manual root and updates to the release checklist.

[leanprover-community-bot]

Mathlib CI status (docs):

• 💥 Mathlib branch lean-pr-testing-7240 build failed against this PR. (2025-02-26 13:12:00) View Log
• 💥 Mathlib branch lean-pr-testing-7240 build failed against this PR. (2025-02-26 14:12:25) View Log
• 💥 Mathlib branch lean-pr-testing-7240 build failed against this PR. (2025-02-26 17:53:58) View Log
• ❗ Batteries/Mathlib CI will not be attempted unless your PR branches off the nightly-with-mathlib branch. Try git rebase 96947280df9cb23b2a87e8c125808e30e96ee58d --onto 7bfa8f6296ebfebc497d5b30d34f494bcc9782a2. You can force Mathlib CI using the force-mathlib-ci label. (2025-03-11 07:45:21)
• ❗ Batteries/Mathlib CI will not be attempted unless your PR branches off the nightly-with-mathlib branch. Try git rebase 96947280df9cb23b2a87e8c125808e30e96ee58d --onto 8fc8e8ed19ef218022f5a94cbf5e472e3b777e44. You can force Mathlib CI using the force-mathlib-ci label. (2025-03-11 10:42:54)

[david-christiansen]

Please hold off on merges until after the upcoming RC is out. It's too late in the process to add a step to the release process this time around.

[mhuisi]

!bench

[leanprover-bot]

Here are the benchmark results for commit 567728b.
There were significant changes against commit c3402b8:

  Benchmark                    Metric         Change
  ==============================================================
- Init.Prelude async           branches         1.5%   (489.5 σ)
- Init.Prelude async           instructions     1.5%   (313.4 σ)
- bv_decide_realworld          task-clock       4.1%    (15.5 σ)
+ identifier auto-completion   branches       -92.2% (-3468.3 σ)
+ identifier auto-completion   completion     -99.4%
+ identifier auto-completion   instructions   -92.2% (-3397.8 σ)
+ identifier auto-completion   maxrss         -44.4% (-1774.8 σ)
+ identifier auto-completion   task-clock     -92.3% (-1379.5 σ)
+ identifier auto-completion   wall-clock     -94.5% (-2345.3 σ)
- ilean roundtrip              parse            8.4%    (14.6 σ)
- ilean roundtrip              task-clock       1.8%    (13.6 σ)
- ilean roundtrip              wall-clock       1.8%    (13.3 σ)
- import Lean                  branches         4.6%   (319.0 σ)
- import Lean                  instructions     4.3%   (499.8 σ)
- lake build clean             instructions     8.5%    (67.6 σ)
- lake config elab             instructions     2.9%   (398.6 σ)
- lake config import           instructions     6.9%   (107.2 σ)
- lake config tree             instructions     6.7%   (308.9 σ)
- lake env                     instructions     6.7%   (106.4 σ)
- lake startup                 instructions    27.5%   (414.2 σ)
- language server startup      branches        11.4%    (84.6 σ)
- language server startup      instructions    10.7%    (75.0 σ)
- liasolver                    instructions     1.5%   (184.2 σ)
- parser                       task-clock       4.6%    (19.0 σ)
- parser                       wall-clock       4.6%    (19.1 σ)
- rbmap_library                instructions     1.4%   (197.7 σ)
- simp_arith1                  branches         1.9%   (117.8 σ)
- simp_arith1                  instructions     1.7%   (120.2 σ)
- stdlib                       instructions     1.1%   (571.8 σ)
- stdlib                       task-clock       1.2%    (10.6 σ)

[david-christiansen]

Thanks!

I'll try getting rid of the pre-validation and see if that makes things faster. The check for badly-formatted links can be a separate CI step.

[david-christiansen]

!bench

[leanprover-bot]

Here are the benchmark results for commit dfcccd4.
There were significant changes against commit 9694728:

  Benchmark                    Metric         Change
  ==============================================================
+ binarytrees                  wall-clock      -7.0%   (-11.2 σ)
+ identifier auto-completion   branches       -92.5% (-1040.2 σ)
+ identifier auto-completion   completion     -99.3% (-1788.4 σ)
+ identifier auto-completion   instructions   -92.4% (-1017.3 σ)
+ identifier auto-completion   maxrss         -41.0%  (-476.5 σ)
+ identifier auto-completion   task-clock     -92.9% (-1214.5 σ)
+ identifier auto-completion   wall-clock     -95.1% (-2959.2 σ)
- import Lean                  branches         2.3%    (71.8 σ)
- import Lean                  instructions     2.2%    (86.9 σ)
+ import Lean                  task-clock      -7.0%   (-85.8 σ)
+ import Lean                  wall-clock      -7.0%   (-82.3 σ)
- lake build clean             instructions     4.4%    (33.1 σ)
- lake config elab             instructions     1.5%    (47.0 σ)
- lake config import           instructions     3.7%   (154.0 σ)
- lake config tree             instructions     3.6%   (348.1 σ)
- lake env                     instructions     3.6%    (56.0 σ)
- lake startup                 instructions    14.7%   (227.2 σ)
- language server startup      branches         5.8%   (143.9 σ)
- language server startup      instructions     5.6%   (132.3 σ)

[david-christiansen]

!bench

[leanprover-bot]

Here are the benchmark results for commit 2cc98f3.
There were significant changes against commit 9694728:

  Benchmark                    Metric          Change
  ===============================================================
+ identifier auto-completion   branches        -92.7% (-2303.9 σ)
+ identifier auto-completion   completion      -99.3% (-2489.5 σ)
+ identifier auto-completion   instructions    -92.6% (-2260.6 σ)
+ identifier auto-completion   maxrss          -41.0%  (-921.9 σ)
+ identifier auto-completion   task-clock      -93.0% (-1624.8 σ)
+ identifier auto-completion   wall-clock      -95.2% (-3102.2 σ)
+ stdlib                       type checking    -1.7%   (-16.0 σ)

[david-christiansen]

!bench

[leanprover-bot]

Here are the benchmark results for commit 9e65a99.
There were significant changes against commit 9694728:

  Benchmark         Metric       Change
  ================================================
- ilean roundtrip   parse          4.5%   (12.2 σ)
+ rbmap_1           task-clock    -1.0% (-143.9 σ)
+ rbmap_1           wall-clock    -1.0% (-664.9 σ)

[david-christiansen]

!bench

[leanprover-bot]

Here are the benchmark results for commit afbc678.
There were significant changes against commit 9694728:

  Benchmark     Metric       Change
  ==========================================
- import Lean   task-clock     4.5% (36.6 σ)
- import Lean   wall-clock     4.5% (29.0 σ)