Skip to content

[WIP] Try native parser#21331

Closed
ilevkivskyi wants to merge 3 commits intopython:masterfrom
ilevkivskyi:try-native-parser
Closed

[WIP] Try native parser#21331
ilevkivskyi wants to merge 3 commits intopython:masterfrom
ilevkivskyi:try-native-parser

Conversation

@ilevkivskyi
Copy link
Copy Markdown
Member

@ilevkivskyi ilevkivskyi commented Apr 25, 2026

No description provided.

@ilevkivskyi ilevkivskyi marked this pull request as draft April 25, 2026 15:15
@github-actions

This comment has been minimized.

Sign in to view
@ilevkivskyi
Copy link
Copy Markdown
Member Author

ilevkivskyi commented Apr 25, 2026

OK, so most notable things from mypy_primer are:

  • There is some line number churn because types in annotations now have more precise line numbers, IMO this is good.
  • Type comments like this don't work anymore: 
    d = [

]  # type: list[int]
    I think we can support specifically line and end_line. IMO looking for a type comment everywhere in between in silly.
  • It looks like something is different about left/right associativity in the new parser. For example: 
    # mypy: enable-error-code="redundant-expr"
x: int
y: int
z: int
if (
    x is None  # Old parser error and new parser error
    or y is None  # Old parser error only
    or z is None
):
    pass
    Not sure if need to do anything about it.
  • New parser has different rules about whether (redundant?) parenthesis are part of the expression, example: 
    foo = (  # Old parser error here
    "no way"  # New parser error here
) + 1
    Again, not sure if need to do anything about this.

In any case, may be worth mentioning in release notes that some line number changes may happen. cc @JukkaL

Also some of the test failures look suspicious, most notably few mypyc test cases. @JukkaL could you please check these test cases? In the meantime I can look at the type comment issue, likely a trivial fix.

@ilevkivskyi ilevkivskyi mentioned this pull request Apr 25, 2026
Merged
ilevkivskyi added a commit to mypyc/ast_serialize that referenced this pull request Apr 27, 2026
@ilevkivskyi
Support type comment on the last line of an assignment (#57)
1f78fcf 
Fix is trivial, see python/mypy#21331 for
context.
@ilevkivskyi
Copy link
Copy Markdown
Member Author

ilevkivskyi commented Apr 27, 2026

OK, there is one more problem, this test fails with the new parser

[case testIgnoreInPlatformSpecificCodeUsed]
# flags: --warn-unused-ignores

import sys
assert sys.platform == "win32"

42 + "no way"  # type: ignore[operator]
[builtins fixtures/isinstancelist.pyi]

while passes with old parser. I think this is important to fix before the release.

@JukkaL
Copy link
Copy Markdown
Collaborator

JukkaL commented Apr 27, 2026

I think this is important to fix before the release.

I can have a look at this.

@ilevkivskyi
Copy link
Copy Markdown
Member Author

ilevkivskyi commented Apr 27, 2026

I can have a look at this.

OK, go ahead, I briefly looked at this, the problem is that we should update ser.skipped_lines here https://github.com/mypyc/ast_serialize/blob/main/src/serialize_ast.rs#L642

@ilevkivskyi
Copy link
Copy Markdown
Member Author

ilevkivskyi commented Apr 27, 2026

@JukkaL did you already start working on this? Because I already have a fix, sorry.

@ilevkivskyi ilevkivskyi mentioned this pull request Apr 27, 2026
Merged
@ilevkivskyi
Copy link
Copy Markdown
Member Author

ilevkivskyi commented Apr 27, 2026

@JukkaL here is the fix mypyc/ast_serialize#60, please feel free to amend/merge/release it as needed.

ilevkivskyi added a commit to mypyc/ast_serialize that referenced this pull request Apr 29, 2026
@ilevkivskyi
Mark statements after top-level false assert as skipped (#60)
53e49f6 
See python/mypy#21331 for context.
@github-actions

This comment has been minimized.

Sign in to view
This was referenced Apr 29, 2026
Merged
Merged
ilevkivskyi added a commit that referenced this pull request Apr 30, 2026
@ilevkivskyi
Make new parser consistent with the old one (#21377)
ddacf23 
This fixes some issues discovered in
#21331

Two things here:
* Old parser keeps `metaclass` in keywords, so it is better to keep it
in the new one as well
* Old parser has right-associativity for BoolOps, so should have the new
one

If we want to have the new behavior for some reason, it is possible to
instead make some fixes "downstream" in semantic
analysis/type-checking/etc. _But_, it is non-trivial, and IMO it is not
a good time to change the behavior while we still have both parsers.

Note I don't add test for the metaclass change because a whole bunch of
mypyc tests fails with new parser already without this PR.
ilevkivskyi added a commit to mypyc/ast_serialize that referenced this pull request Apr 30, 2026
@ilevkivskyi
Support debug f-string elements (#61)
335644b 
For some reason Ruff parser has a "bolted-on" support for debug
f-strings like `f"test {some.expr()=}"`. I essentially replicate the the
post-processing logic in Python parser, except for implicit collapsing
of consecutive literal elements, which is (already) done by the caller.

This fixes another few issues fond in
python/mypy#21331
@ilevkivskyi
Copy link
Copy Markdown
Member Author

ilevkivskyi commented Apr 30, 2026

OK, there is yet another problem with native parser affecting mypyc. Native parser sets import flags (like is_top_level, is_mypy_only) only in tree.imports but not in tree.defs. At the same time there are many places where mypyc needs those flags.

I don't see how to restore those reliably, so I guess we will need to bump cache_version again...

@ilevkivskyi
Copy link
Copy Markdown
Member Author

ilevkivskyi commented Apr 30, 2026

I am going to proceed with ast_serializev0.2.3 release to get the f-string fixes, and will bump the cache version in v0.3.0.

@ilevkivskyi ilevkivskyi mentioned this pull request Apr 30, 2026
Merged
ilevkivskyi added a commit that referenced this pull request Apr 30, 2026
@ilevkivskyi
Run semantic analysis pass one if file doesn't exsit (#21379)
e8e1a63 
Noticed this while working on the other fixes from
#21331. If file doesn't exist, it
will be parsed with the old parser, even if the native parser is
enabled, so we need to run semantic analysis pass one on it.

Not adding a test because this will not be needed soon (and I am lazy,
sorry).
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 30, 2026

Diff from mypy_primer, showing the effect of this PR on open source code:

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/utilities/collections.py:504: error: "None" object is not iterable  [misc]
+ src/prefect/utilities/collections.py:505: error: "None" object is not iterable  [misc]
- src/prefect/_internal/concurrency/api.py:181: error: "Call" expects no type arguments, but 1 given  [type-arg]
- src/prefect/_internal/concurrency/api.py:181: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
+ src/prefect/_internal/concurrency/api.py:182: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
- src/prefect/_internal/concurrency/api.py:181: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ src/prefect/_internal/concurrency/api.py:182: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ src/prefect/_internal/concurrency/api.py:183: error: "Call" expects no type arguments, but 1 given  [type-arg]
+ src/prefect/_internal/concurrency/api.py:183: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
+ src/prefect/_internal/concurrency/api.py:183: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
- src/prefect/server/api/deployments.py:459: error: Invalid "type: ignore" comment  [syntax]
+ src/prefect/server/api/deployments.py:459: error: Unused "type: ignore" comment  [unused-ignore]
- src/prefect/workers/base.py:30: error: Invalid "type: ignore" comment  [syntax]
+ src/prefect/workers/base.py:30: error: Unused "type: ignore" comment  [unused-ignore]

operator (https://github.com/canonical/operator)
- ops/charm.py:137: error: Invalid "type: ignore" comment  [syntax]
- ops/charm.py:137: error: Incompatible types in assignment (expression has type "str | None", variable has type "str")  [assignment]

colour (https://github.com/colour-science/colour)
- colour/characterisation/aces_it.py:975: error: Ellipses cannot accompany other parameter types in function type signature  [syntax]
+ colour/characterisation/aces_it.py:976: error: Unexpected "..."  [misc]
+ colour/characterisation/aces_it.py:977: error: Unexpected "..."  [misc]

Expression (https://github.com/cognitedata/Expression)
- tests/test_fn.py:13: error: Invalid "type: ignore" comment  [syntax]
+ tests/test_fn.py:13: error: Unused "type: ignore" comment  [unused-ignore]
- tests/test_catch.py:100: error: Invalid "type: ignore" comment  [syntax]
- tests/test_catch.py:100: error: Incompatible return value type (got "int", expected "str")  [return-value]
- tests/test_catch.py:100: error: Unsupported operand types for + ("int" and "str")  [operator]

antidote (https://github.com/Finistere/antidote)
- src/antidote/lib/interface_ext/__init__.py:1359: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
- src/antidote/lib/interface_ext/__init__.py:1359: error: Missing type arguments for generic type "staticmethod"  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: Missing type arguments for generic type "staticmethod"  [type-arg]

core (https://github.com/home-assistant/core)
- homeassistant/components/zwave_js/api.py:146: error: Parameter 1 of Literal[...] cannot be of type "Any"  [valid-type]
+ homeassistant/components/zwave_js/api.py:148: error: Parameter 1 of Literal[...] cannot be of type "Any"  [valid-type]

Tanjun (https://github.com/FasterSpeeding/Tanjun)
- tanjun/commands/menu.py:420: error: Type argument "_UserCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:422: error: Type argument "_UserCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_UserCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:422: error: Type argument "_UserCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_MessageCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:424: error: Type argument "_MessageCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_MessageCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:424: error: Type argument "_MessageCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]

ibis (https://github.com/ibis-project/ibis)
- ibis/selectors.py:434: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
+ ibis/selectors.py:437: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
- ibis/selectors.py:434: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ ibis/selectors.py:437: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
- ibis/expr/types/generic.py:152: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:164: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:152: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:164: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/generic.py:170: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:171: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:170: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:171: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/temporal.py:428: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:431: error: "Value" expects no type arguments, but 1 given  [type-arg]
- ibis/expr/types/temporal.py:1161: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:1174: error: "Value" expects no type arguments, but 1 given  [type-arg]

static-frame (https://github.com/static-frame/static-frame)
+ static_frame/core/frame.py:7140: error: Unused "type: ignore" comment  [unused-ignore]
+ static_frame/core/frame.py:7141: error: Need type annotation for "name"  [var-annotated]
+ static_frame/core/frame.py:7383: error: Unused "type: ignore" comment  [unused-ignore]
+ static_frame/core/frame.py:7384: error: Need type annotation for "name"  [var-annotated]

apprise (https://github.com/caronc/apprise)
- apprise/plugins/fluxer.py:512: error: Unsupported operand types for + ("str" and "bool")  [operator]
+ apprise/plugins/fluxer.py:513: error: Unsupported operand types for + ("str" and "bool")  [operator]
- apprise/plugins/fluxer.py:512: note: Right operand is of type "bool | str"
+ apprise/plugins/fluxer.py:513: note: Right operand is of type "bool | str"
- apprise/plugins/fluxer.py:512: error: Incompatible types in assignment (expression has type "Any | str", target has type "bool")  [assignment]
+ apprise/plugins/fluxer.py:513: error: Incompatible types in assignment (expression has type "Any | str", target has type "bool")  [assignment]

JukkaL pushed a commit that referenced this pull request Apr 30, 2026
@ilevkivskyi @JukkaL
Make new parser consistent with the old one (#21377)
ffe9570 
This fixes some issues discovered in
#21331

Two things here:
* Old parser keeps `metaclass` in keywords, so it is better to keep it
in the new one as well
* Old parser has right-associativity for BoolOps, so should have the new
one

If we want to have the new behavior for some reason, it is possible to
instead make some fixes "downstream" in semantic
analysis/type-checking/etc. _But_, it is non-trivial, and IMO it is not
a good time to change the behavior while we still have both parsers.

Note I don't add test for the metaclass change because a whole bunch of
mypyc tests fails with new parser already without this PR.
@ilevkivskyi
Copy link
Copy Markdown
Member Author

ilevkivskyi commented Apr 30, 2026

OK, I am happy with the primer now.

JukkaL pushed a commit that referenced this pull request Apr 30, 2026
@ilevkivskyi @JukkaL
Run semantic analysis pass one if file doesn't exsit (#21379)
2b9844d 
Noticed this while working on the other fixes from
#21331. If file doesn't exist, it
will be parsed with the old parser, even if the native parser is
enabled, so we need to run semantic analysis pass one on it.

Not adding a test because this will not be needed soon (and I am lazy,
sorry).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ilevkivskyi @JukkaL