How to avoid repeating all the @overloads (reportIncompatibleMethodOverride)

[ithinuel]

Hello, I am working on a code base that a construct similar to this:

from typing import Callable, Literal, Union, overload


class A:
    @overload
    def foo(self) -> int: ...
    @overload
    def foo(self, now: Literal[True] = True) -> int: ...
    @overload
    def foo(self, now: Literal[False]) -> Callable[[], int]: ...
    @overload
    def foo(self, now: bool) -> Union[int, Callable[[], int]]: ...

    def foo(self, now: bool = True) -> Union[int, Callable[[], int]]:
        """Does foo things."""
        raise NotImplementedError()


class B(A):
    def foo(self, now: Literal[True] = True) -> int:
        assert now is True
        return 42


class C(A):
    def foo(self, now: bool = True) -> int:
        assert now is True
        return 42


class D(A):
    def foo(self) -> int:
        return 42


class E(A):
    @overload
    def foo(self) -> int: ...
    @overload
    def foo(self, now: Literal[True] = True) -> int: ...
    @overload
    def foo(self, now: Literal[False]) -> Callable[[], int]: ...
    @overload
    def foo(self, now: bool) -> Union[int, Callable[[], int]]: ...

    def foo(self, now: bool = True) -> Union[int, Callable[[], int]]:
        assert now is True
        return 42

Pyright reports a reportIncompatibleMethodOverride in B, C and D but not E.
I’m not familiar enough with python to fully grasp what’s happening here.

Can someone explain what am I doing wrong, please?
Is any way to satisfy Pyright without having to repeat all the @overloads in the subclass?

[erictraut]

The type system assumes that any subclass of A is compatible with A. That assumption is referred to as the "Liskov Substitution Principle" or LSP. In your example, the base class A has a method foo that accepts a set of arguments. All subclasses of A must accept the same sets of arguments.

Here's a code sample that demonstrates why this is a problem:

def func(a: A):
    a.foo(now=False)

func(B())

This code snippet does not violate any type assignment rules because B is a subclass of A. However, it will crash at runtime because B.foo violates the LSP. By contrast, E does not violate the LSP.

[ithinuel]

Thank you very much for your answer.

I am still struggling to understand why is it required to have the @overload listed in E.

For example, if I add F to the snippet:

class F(A):
    def foo(self, now: bool = True) -> Union[int, Callable[[], int]]:
		if now:
			return 42
		else:
			return lambda: 42

This runs fine with

def func(a: A):
    print(a.foo())
    print(a.foo(now=True))
    res = a.foo(now=False))
    print(res())

but Pyright reports

  /home/ithinuel/test.py:52:9 - error: Method "foo" overrides class "A" in an incompatible manner
    Return type mismatch: base method returns type "int", override returns type "int | (() -> int)"
      Type "int | (() -> int)" is not assignable to type "int"
        "function" is not assignable to "int" (reportIncompatibleMethodOverride)

If I add only one of the overload, pyright reports

  /home/ithinuel/test.py:53:9 - error: "foo" is marked as overload, but additional overloads are missing (reportInconsistentOverload)
  /home/ithinuel/test.py:55:9 - error: Method "foo" overrides class "A" in an incompatible manner
    Override does not handle all overloads of base method (reportIncompatibleMethodOverride)

The first line is reported for any of the overload used which is fair. After all, there’s no overload if there’s only one signature.
The second is not reported if using def foo(self, now: bool) -> Union[int, Callable[[], int]]: ... which is the most generic signature.

So adding def foo(self, now: bool) -> Union[int, Callable[[], int]]: ... and any of the other overload (does not matter how many, nor which) is enough to satisfy pyright.

As far as I understand, the @overload should only be required on A and should not need to be replicated on the subclasses.
In the likely case I am wrong and @overload are indeed required there as well, I don’t get why any two are enough (provided one of them is the most generic variant).

[erictraut]

As far as I understand, the @overload should only be required on A and should not need to be replicated on the subclasses.

No, that's a wrong assumption. If a subclass defines an overloaded method (whether or not that method is overriding a method in a parent class), you need to define the overload signatures for the method. You may find that it's useful to read this chapter of the Python typing spec. This section may also be useful because it defines assignability rules for callable types. An overridden method must be assignable to the method it is overriding.

[ithinuel]

Thank you, that was very informative.

Also reading this stackoverflow post and this other discussion I understand a bit more what I had wrong.