Fix annotations for builtins.reversed.__new__

[jorenham]

fixes #11645

---

Deferred due to a mypy bug: python/mypy#15182

[srittau]

It seems that mypy has problems with the first overload, but I'm not sure why. Everything seems correct to me and pyright seems to agree. For now, I would suggest to split this PR in two, one PR for the second overload and a separate PR for the first one. This should allow us to get part of the changes into typeshed soon, and will make it easier to debug the mypy problem.

[jorenham]

@srittau I have tried many different approaches, but none of them seem to be accepted by mypy. As you said, pyright has no problem with this. I can't draw any other conclusions than this being a mypy bug, and it's easy to reproduce:

https://mypy-play.net/?mypy=latest&python=3.12&gist=0c754a7826a80f50ced671c26d891950&flags=strict

The expected behavior can be illustrated by pyright, which works as expected and doesn't require the type: ignore[misc]:
https://pyright-play.net/?strict=true&code=GYJw9gtgBALgngBwJYDsDmUkQWEMoAqiApgGoCGIAUFQVALyEkUgAUA5AewJQ1UDGAG3IBnEVAByYGAEEiCYgC4qUVVAAmxYFAD6OlMQDue1kJEAaKJTSLC3KAFoAfIWVr3UEMRgBXECisQNBUPVRDQqi8AN2JyQR14BVYpWXliVgAWACZubiA

I wasn't able to find anything on https://typing.readthedocs.io/ that would restrict the type of the return value from a (non-meta) class' __new__ method.
But for some reason, mypy blindly assumes that this is always a covariant type that is bound to the class at hand (i.e. __new__: (cls, ...) -> Self with Self somehow being covariant). Any annotations that suggest otherwise result in a type error in the misc category. But even if that error is explicitly suppressed, mypy will implicitly ignore it.

[srittau]

I've marked this PR as deferred for now due to the mypy bug.

[jorenham]

@srittau I don't quite follow.

Currently, all typecheckers will reject a reversed call, if its argument isn't iterable or a sequence, even if implements __reversed__. The reason why the typetests did not complain here, is because it simply wasn't tested.

This PR makes it so that (presumably) only mypy will incorrectly reject it, due to a bug. But the new tests that I included only fail because of mypy's bug.

So I don't see how deferring this would be any better than mergin this. I believe it's reasonable to ignore mypy's false negatives, since they're directly caused by a known bug.

[srittau]

The mypy primer output above shows the real-world impact this change has on a selected number of packages. Each green line is a new mypy warning. As you can see, this change unfortunately leads to an excessive amount of new errors. As far as I understand, many reversed(...) calls now return ...[Never], which leads to all the errors above.

[jorenham]

@srittau ah ok I see. I blindly assumed that "green is good" for some reason 🤷🏻

[jorenham]

According to the updated typing spec, it is indeed allowed for __new__ to return some non Self type:
https://typing.readthedocs.io/en/latest/spec/constructors.html#new-method

[bluenote10]

Note that the same was attempted in #13269 so the discussion there may apply here as well.

[srittau]

At the moment, this doesn't look promising. Unless someone has another good idea, I suggest we close this PR – at least for now.

[jorenham]

At the moment, this doesn't look promising. Unless someone has another good idea, I suggest we close this PR – at least for now.

Yea, not much movement on the mypy front unfortunately...

Maybe it could help to avoid duplicate PR's if we keep this open? It's also fine to close it if you want, though; whatever floats for boat 🤷🏻.

[srittau]

I see that it's already marked as deferred. I've also marked it as draft, just so it doesn't stand out as much in the open PRs list.

[jorenham]

I also rebased onto main, pro forma I guess.

[tungol]

Mypy python/mypy#21441 appear to fix the blocker. Once that makes it into a release, this can be revisited.

[jorenham]

Mypy python/mypy#21441 appear to fix the blocker. Once that makes it into a release, this can be revisited.

python/mypy#21441 was fixed in mypy 2.2.0 :)

[jorenham]

those blank lines were inserted by pre-commit for some reason 🤷

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

comtypes (https://github.com/enthought/comtypes)
- comtypes/server/register.py:226: error: Incompatible types in assignment (expression has type "Iterator[tuple[int, str]]", variable has type "list[tuple[int, str]]")  [assignment]
+ comtypes/server/register.py:226: error: Incompatible types in assignment (expression has type "reversed[tuple[int, str]]", variable has type "list[tuple[int, str]]")  [assignment]

discord.py (https://github.com/Rapptz/discord.py)
- discord/abc.py:1815: error: Incompatible types in assignment (expression has type "Iterator[MessagePin]", variable has type "list[MessagePin]")  [assignment]
+ discord/abc.py:1815: error: Incompatible types in assignment (expression has type "reversed[MessagePin]", variable has type "list[MessagePin]")  [assignment]

pytest (https://github.com/pytest-dev/pytest)
+ src/_pytest/monkeypatch.py:417: error: Unused "type: ignore" comment  [unused-ignore]
+ src/_pytest/monkeypatch.py:422: error: Unused "type: ignore" comment  [unused-ignore]

jinja (https://github.com/pallets/jinja)
+ src/jinja2/utils.py:570: error: Returning Any from function declared to return "Iterator[Any]"  [no-any-return]