Add reload() command for interactive scene reloading

[Splines]

Motivation

The motivation for this change is described in this discussion. In summary: the checkpoint_paste() mechanism is great but doesn't handle all situations. A simple example is when you use some values that are defined outside of the class that inherits from Scene (see test code down below).

In the current workflow, users have to exit the IPython terminal, i.e. also close the Manim window. Then execute the manimgl command manually again to reload the scene. This is cumbersome to do during iterative scene development where such a workflow might be frequently used.

Tip

Therefore, we introduce a new reload() command to the IPython shell which will quit the active IPython embedded shell and restart the scenes with the same arguments that were passed to manimgl initially. During all of this, the ManimGL window stays open and is reused. As optional argument, reload() takes a line number where we should start at. See the video down below for an example.

Proposed changes

I did my best to add docstrings to the respective classes and methods so the code should already give away the main changes. But for a summary:

• Add a new reload() method to the scene.py. It raises a KillEmbedded exception by means of the exit_raise IPython line magic.
• Introduce a new ReloadManager that will run a while True loop and catches KillEmbedded exceptions to restart the scenes.
• During the reinitialization of the scenes, we will reuse the window. The window state is reset at every scene reload.

manimlib.extract_scene.main(config) will return a list of scenes. I haven't quite understood how the user can invoke running multiple scenes. Therefore this scenario of multiple scenes is not tested by me. As heuristic for the window (see the retrieve_scenes_and_run method): I will reuse the window of the first scene (when iterating the array) that has a window other than None, which should be just the window of the first scene in the list.

Known limitations

• This is my first time contributing to this amazing project. I arrived at the current changes through a lot of experimenting and am unaware of side effects that these changes may cause. In some small tests like the one below, scene rendering seemed to work fine. Maybe reload() can be marked as experimental. If users find strange behavior, they should test if it works when quitting, then running manimgl again. Is there some kind of test suite available to avoid manual testing all the time?
• reload() can be called with any line number. If this line number doesn't correspond to a comment, checkpoint_paste() will be screwed up. I think this can be accepted as known limitation. A future PR could tackle this case and e.g. decrease the line number until a comment inside the construct() method is found.

Proposal for developer well-being

Expand

While this PR was a lot of fun to work with, a major pain point was missing linting in this repo. Whenever I save a file in VSCode, my Python code will be linted automatically, which is a good thing as I think it's quite tedious to discuss stylistic changes in PRs (like: "don't use this ternary operator here", "use snake case here", "use double quotes, not single quotes" etc.). With linting we just accept the style that is prescribed by the repo and can focus on what is more important: the semantics of the code.

I didn't want to turn off the automatic linting in VSCode via Pylint because I still wanted that my new code changes were formatted nicely. Therefore, what I had to do in the end is write a bit of code, then use the Stage selected ranges features to only stage the ranges in the Git diff that I modified (and leave those out that were automatically modified by the linting). As you can imagine this was quite a tedious process to do.

For some inspiration: in this repo, I've set up pylint for the repository here and here specifically for VSCode as well as here, then also added a linter workflow that runs in GitHub Actions to see if every PR conforms 100%. Ideally, one would set up the linter, possibly define some exceptions to the rule (e.g. having to add a docstring to every method might be overkill in this repo), then run the linter for every Python file in the codebase and commit this.

Test

from manimlib import *

# change value to `-2`, then
# - use checkpoint_paste() -> it will use the cached value
# - reload(), then checkpoint_paste() -> it will use the new value
EXTERNAL_VALUE = 2


class ReloadTest(Scene):
    def construct(self):
        ## Circle creation
        circle = Circle()
        self.play(ShowCreation(circle))

        ## Circle movement
        self.play(circle.animate.shift(EXTERNAL_VALUE * UP + 2 * RIGHT))

Call via:

manimgl ./reload_sample.py ReloadTest -se 11

manim-reload-sample.mp4

_{In the video, just ignore the other things that are going on in our manim-notebook extension for VSCode that is currently under development. Clicking on Preview Manim Cell will just copy the content to the clipboard and then execute the checkpoint_paste() command as usual.}

[Splines]

@3b1b Just let me know if there's anything I can help with to ease the review of this PR.

[3b1b]

Thank you for the contribution, I'll take a closer look and give it a try. Also, duly noted regarding the pain points you felt. The lack of a standard test suite is an obvious shortcoming, and I'm definitely open to suggestions on setting up pylint for the repo.

For the use case shown here, for changes outside the scene like the one you're describing, what I've been doing is simply running that code in the Python terminal. For I'd highlight the line EXTERNAL_VALUE = -2, hit my keyboard shortcut, and that updates the value in the session. There are cases where this doesn't work, e.g. if you changed the code for the Circle class, and wanted to run methods on an existing circle instance respecting those updates. In those cases, though, I'd typically copy in the text of the entire file where the updates were made, then do a checkpoint_paste run of the current scene from before the circle (or whatever it is) was instantiated.

I haven't felt too much pain around this yet. Just so I understand the goal of this PR better, can you highlight an example where running the updated code by highlighting it and pasting it into the current iPython terminal fails, and calling this reload will not? I don't doubt that such examples exist, I ask in the spirit of framing my understanding of what this is doing.

Also, regarding multiple scenes, that functionality is meant for the case of rendering multiple scenes to file, e.g. with the -w flag, it's less relevant to interactive scene development. Would this PR interfere with that use case, or is it only relevant to interactive development?

[Splines]

can you highlight an example where running the updated code by highlighting it and pasting it into the current iPython terminal fails, and calling this reload will not?

@3b1b To be honest, I don't think I have an example where highlighting it, pasting it into the IPython terminal and then using checkpoint_paste() doesn't work, but reload does.

However, here is an example that might make the use case more clear. I've coded it up to illustrate the evolute of an arbitrary parameterized function. In addition to numpy, you have to have sympy installed as I don't do numerical approximations of derivatives here, but rather calculate them symbolically via sympy:

pip install sympy

Then see this example (tested with Python 3.12.7)

# Manim 1.7.1, commit 003c4d8626
import sys

sys.path.append(".")
from manimlib import *
from dataclasses import dataclass
from typing import Callable
import numpy as np
from sympy import symbols, sympify, lambdify, Matrix, Expr, Pow, Add, sqrt

COLOR_RED = "#E53054"
COLOR_BLUE_LIGHT = "#51BADE"


@dataclass
class Coordinates2D:
    x: Callable[[float], float]
    y: Callable[[float], float]


@dataclass
class Expr2D:
    x: Expr
    y: Expr


@dataclass
class Time:
    start: float
    end: float

    def at_progress(self, t: float) -> float:
        return self.start + t * (self.end - self.start)


class Function2D:
    def __init__(self, x: str, y: str, independent_variable: str):
        self.t = symbols(independent_variable)
        self.x: Expr = sympify(x)
        self.y: Expr = sympify(y)

    def lambdify(self) -> Coordinates2D:
        return Coordinates2D(
            x=lambdify(self.t, self.x, "numpy"),
            y=lambdify(self.t, self.y, "numpy"),
        )

    def f_prime(self) -> Expr2D:
        return Expr2D(self.x.diff(self.t), self.y.diff(self.t))

    def f_primes(self) -> tuple[Expr2D, Expr2D]:
        f_prime = self.f_prime()
        f_prime2 = Expr2D(f_prime.x.diff(self.t), f_prime.y.diff(self.t))
        return f_prime, f_prime2

    def normal(self) -> Expr2D:
        f_prime = self.f_prime()
        norm = self.norm(f_prime)
        return Expr2D(-f_prime.y / norm, f_prime.x / norm)  # type: ignore

    def norm(self, f_prime: Expr2D) -> Expr:
        return sqrt(Add(f_prime.x**2, f_prime.y**2))

    def curvature(self) -> Expr:
        f_prime, f_prime2 = self.f_primes()
        matrix = Matrix([[f_prime.x, f_prime.y], [f_prime2.x, f_prime2.y]])
        return matrix.det() / self.norm(f_prime) ** 3  # type: ignore

    def radius(self) -> Expr:
        return Pow(self.curvature(), -1)

    def radius_eval(self) -> Callable[[float], float]:
        radius = self.radius()
        return lambdify(self.t, radius, "numpy")

    def evolute(self):
        normal = self.normal()
        radius = self.radius()
        return Expr2D(
            self.x + radius * normal.x,  # type: ignore
            self.y + radius * normal.y,  # type: ignore
        )

    def evolute_eval(self):
        evolute = self.evolute()
        return Coordinates2D(
            x=lambdify(self.t, evolute.x, "numpy"),
            y=lambdify(self.t, evolute.y, "numpy"),
        )


class EvoluteScene(Scene):
    def construct(self):
        time = Time(0, TAU)

        ## Define objects
        func = Function2D("3 * cos(t) + cos(3*t)", "3 * sin(t) + sin(3*t)", "t")
        func_lambda = func.lambdify()
        curve = ParametricCurve(
            lambda t: np.array([func_lambda.x(t), func_lambda.y(t), 0]),
            t_range=(time.start, time.end, 0.01),
            color=COLOR_RED,
            stroke_width=4,
        )
        point = GlowDot(color=COLOR_RED, glow_factor=0.9)
        point.move_to(curve.point_from_proportion(0))

        ## Coordinate system
        axes = Axes(
            x_range=(-5, 5, 1),
            y_range=(-5, 5, 1),
        )
        labels = axes.get_axis_labels()
        self.play(ShowCreation(axes), Write(labels), self.camera.frame.animate.set_width(20))
        self.wait()

        ## Function
        self.play(ShowCreation(curve))
        self.play(MoveAlongPath(point, curve), run_time=5)
        self.wait()

        ## Evolute
        evolute = func.evolute_eval()
        evolute_curve_func = ParametricCurve(
            lambda t: np.array([evolute.x(t), evolute.y(t), 0]),
            t_range=(time.start, time.end, 0.01),
            color=COLOR_BLUE_LIGHT,
            stroke_width=2,
            use_smoothing=False,
        )
        evolute_curve = DashedVMobject(
            evolute_curve_func,
            num_dashes=100,
        )
        self.play(ShowCreation(evolute_curve))

        ## Move a new point along the evolute and draw a circle around it
        radius = func.radius_eval()

        evolute_point = GlowDot(color=COLOR_BLUE_LIGHT)
        evolute_point.move_to(evolute_curve_func.point_from_proportion(0))

        circle = Circle()
        circle.set_stroke(color=COLOR_BLUE_LIGHT, width=2)
        circle.move_to(evolute_point)

        def update_circle(mob, t):
            mob.set_width(2 * radius(time.at_progress(t)))
            mob.move_to(evolute_point)

        self.add(circle, evolute_point)
        update_circle(circle, 0)

        ## Update circle's position and radius dynamically
        self.play(
            MoveAlongPath(point, curve),
            MoveAlongPath(evolute_point, evolute_curve_func),
            UpdateFromAlphaFunc(circle, update_circle),
            run_time=5,
            rate_func=linear,
        )
        self.wait()


def main():
    # Just to try things out
    func = Function2D("2 * cos(t)", "sin(t)", "t")
    ev = func.evolute()
    print(f"({ev.x}, {ev.y})")
    evolute = func.evolute_eval()
    t = 0
    print(evolute.x(t), evolute.y(t))


if __name__ == "__main__":
    main()

The reason I've introduced this command is more a matter of convenience. In the example here, I made some mistakes when I coded the Function2D class, e.g. take a power of 2 instead of 3 in curvature(). Copying the code of the whole class, pasting it to the terminal, then running checkpoint_paste() (the latter via a shortcut but still) feels cumbersome to me when I want to quickly iterate on changes and try things out.

Another reason for this change is that I'm currently working on a new Manim Notebook extension for VSCode with a small team (work in progress before first stable release) that will introduce the concepts of Manim Cells into the editor:

It will automatically spawn new terminal instances in VSCode that run Manim such that new users can profit from a terminal-free workflow (almost), but advanced users can still interfere with the Terminal at any point. E.g. we also detect when the terminal is closed, or when a new Manim shell is opened. We also parse the terminal for niceties like progress bars:

That's all to say: this workflow could greatly profit from such a reload() command as it would allow for an easy solution to avoid caching issues with checkpoint_paste(). E.g. I could imagine a ▶ Preview Manim Cell (reload) CodeLens next to the ▶ Preview Manim Cell CodeLens. So far, we offer a Quit Preview command and a Start scene at cursor command that would do this job. However, this means that the OpenGL window is closed and has to be opened again, which takes quite some time (like 4-5s on my machine compared to reload() which gets the job done in under 1s because it recycles the OpenGL window).

[Splines]

Also, regarding multiple scenes, that functionality is meant for the case of rendering multiple scenes to file, e.g. with the -w flag, it's less relevant to interactive scene development. Would this PR interfere with that use case, or is it only relevant to interactive development?

Thanks for clarifying. In the end, I still do a

for scene in self.scenes:
    scene.run()

inside retrieve_scenes_and_run() in the new reload_manager.py. So, this shouldn't interfere with the -w option.

[Splines]

Ah and by the way (should this get merged): I don't think all my individual commits from this PR are that interesting to keep on the master branch (I was experimenting a lot in the beginning). So feel free to use a Squash commit for my PR if you'd like to.

[3b1b]

Okay, that is helpful context, thanks! I'll find some time to give this a closer look in the next day or two.

[3b1b]

In general, I'm hesitant to add complexity without a clear use case. For most of the examples you described, it still feels easier to press command-A (to highlight everything) then command-R (or whatever keybinding you have tied to checkpoint_paste) to update the iPython terminal state to reflect the updated classes/functions/constants/etc outside the scene. In fact, for code changes outside the file of the scene, from what I can tell the reload call does not incorporate those, whereas the command+A, command+R on that file would.

That said, the notebook example makes clear what you're going for, and I could understand for that project how this would be useful. Also, there are times when for large scenes I find myself wanting to go back to an earlier part of the scene which hasn't been checkpointed, and it does seem nice to get rid of the lag associated with spinning up a new window.

I'll squash and merge, thanks for taking the time to help out here, and thanks for so clearly writing out the explanation and motivation.

[Splines]

Thanks a lot for reviewing and merging despite adding a use-case that you yourself might not find too beneficial.

I've addressed part of your comment in the follow-up PR #2251.