Skip to content

Conversation

@mtshiba
Copy link
Contributor

@mtshiba mtshiba commented Nov 28, 2025

Summary

Closes astral-sh/ty#957

As explained in astral-sh/ty#957, literal union types for recursively defined values ​​can be widened early to speed up the convergence of fixed-point iterations.
This PR achieves this by embedding a marker in UnionType that distinguishes whether a value is recursively defined.

This also allows us to identify values ​​that are not recursively defined, so I've increased the limit on the number of elements in a literal union type for such values.

Test Plan

call/union.md has been updated

@astral-sh-bot
Copy link

astral-sh-bot bot commented Nov 28, 2025

Diagnostic diff on typing conformance tests

No changes detected when running ty on typing conformance tests ✅

@AlexWaygood AlexWaygood added the ty Multi-file analysis & type inference label Nov 28, 2025
@astral-sh-bot
Copy link

astral-sh-bot bot commented Nov 28, 2025

mypy_primer results

Changes were detected when running on open source projects
pandera (https://github.com/pandera-dev/pandera)
- pandera/api/pandas/model.py:210:13: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | str | ExtensionDtype | ... omitted 3 union elements`, found `dict[Unknown, Unknown | None]`
+ pandera/api/pandas/model.py:210:13: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | Literal["bool", "boolean", "?", "b1", "bool_", ... omitted 220 literals] | ExtensionDtype | ... omitted 3 union elements`, found `dict[Unknown, Unknown | None]`
+ pandera/engines/numpy_engine.py:52:41: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | Literal["bool", "boolean", "?", "b1", "bool_", ... omitted 220 literals] | ExtensionDtype | ... omitted 3 union elements`, found `str`
- pandera/engines/pandas_engine.py:1962:26: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | str | ExtensionDtype | ... omitted 3 union elements`, found `ArrowDtype | None`
+ pandera/engines/pandas_engine.py:1962:26: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | Literal["bool", "boolean", "?", "b1", "bool_", ... omitted 220 literals] | ExtensionDtype | ... omitted 3 union elements`, found `ArrowDtype | None`
- pandera/engines/pandas_engine.py:1988:26: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | str | ExtensionDtype | ... omitted 3 union elements`, found `ArrowDtype | None`
+ pandera/engines/pandas_engine.py:1988:26: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | Literal["bool", "boolean", "?", "b1", "bool_", ... omitted 220 literals] | ExtensionDtype | ... omitted 3 union elements`, found `ArrowDtype | None`
- pandera/engines/pyarrow_engine.py:440:22: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | str | ExtensionDtype | ... omitted 3 union elements`, found `ArrowDtype | None`
+ pandera/engines/pyarrow_engine.py:440:22: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | Literal["bool", "boolean", "?", "b1", "bool_", ... omitted 220 literals] | ExtensionDtype | ... omitted 3 union elements`, found `ArrowDtype | None`
- pandera/engines/pyarrow_engine.py:465:22: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | str | ExtensionDtype | ... omitted 3 union elements`, found `ArrowDtype | None`
+ pandera/engines/pyarrow_engine.py:465:22: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | Literal["bool", "boolean", "?", "b1", "bool_", ... omitted 220 literals] | ExtensionDtype | ... omitted 3 union elements`, found `ArrowDtype | None`
- tests/pandas/test_pydantic.py:308:53: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | str | ExtensionDtype | ... omitted 3 union elements`, found `dict[Unknown, Unknown] | dict[str | Unknown, Any | None | type[Any]]`
+ tests/pandas/test_pydantic.py:308:53: error[invalid-argument-type] Argument to bound method `astype` is incorrect: Expected `type | Literal["bool", "boolean", "?", "b1", "bool_", ... omitted 220 literals] | ExtensionDtype | ... omitted 3 union elements`, found `dict[Unknown, Unknown] | dict[str | Unknown, Any | None | type[Any]]`
- Found 1635 diagnostics
+ Found 1636 diagnostics

freqtrade (https://github.com/freqtrade/freqtrade)
- freqtrade/rpc/rpc.py:1451:88: error[invalid-argument-type] Argument to bound method `replace` is incorrect: Expected `Sequence[str | bytes | date | ... omitted 10 union elements] | NAType | date | ... omitted 13 union elements`, found `dict[Unknown | NaTType, Unknown | None]`
+ freqtrade/rpc/rpc.py:1448:28: error[no-matching-overload] No overload of bound method `select_dtypes` matches arguments

openlibrary (https://github.com/internetarchive/openlibrary)
+ openlibrary/plugins/worksearch/code.py:1207:9: error[invalid-argument-type] Argument to function `run_solr_query` is incorrect: Expected `Literal["UNLABELLED", "BOOK_SEARCH", "BOOK_SEARCH_API", "BOOK_SEARCH_FACETS", "BOOK_CAROUSEL", ... omitted 11 literals]`, found `str`
- Found 1122 diagnostics
+ Found 1123 diagnostics

scikit-build-core (https://github.com/scikit-build/scikit-build-core)
- src/scikit_build_core/_logging.py:153:13: warning[unsupported-base] Unsupported class base with type `<class 'Mapping[str, Style]'> | <class 'Mapping[str, Divergent]'>`
- Found 42 diagnostics
+ Found 41 diagnostics

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ tests/indexes/test_index_float.py:113:13: error[type-assertion-failure] Type `Index[int | float]` does not match asserted type `Index[Any]`
- tests/series/test_series_float.py:52:13: error[type-assertion-failure] Type `Series[int | float]` does not match asserted type `Series[Any]`
+ tests/series/test_series_float.py:52:13: error[type-assertion-failure] Type `Series[int | float]` does not match asserted type `Series[list[Unknown | float]]`
+ tests/series/test_series_float.py:108:13: error[type-assertion-failure] Type `Series[int | float]` does not match asserted type `Unknown`
+ tests/series/test_series_float.py:108:25: error[no-matching-overload] No overload of bound method `astype` matches arguments
+ tests/series/test_series_float.py:110:15: error[no-matching-overload] No overload of bound method `astype` matches arguments
- Found 5859 diagnostics
+ Found 5863 diagnostics

pydantic (https://github.com/pydantic/pydantic)
- pydantic/fields.py:943:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:943:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:983:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:983:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1026:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1026:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1066:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1066:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1109:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1109:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1148:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1148:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1188:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1188:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1567:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, Divergent], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`
+ pydantic/fields.py:1567:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`

No memory usage changes detected ✅

@codspeed-hq
Copy link

codspeed-hq bot commented Nov 28, 2025

CodSpeed Performance Report

Merging #21683 will degrade performances by 9.91%

Comparing mtshiba:faster-literal-type-convergence (a515490) with main (ecab623)

Summary

❌ 1 regression
✅ 21 untouched
⏩ 30 skipped1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Mode Benchmark BASE HEAD Change
Simulation ty_micro[many_string_assignments] 76.4 ms 84.8 ms -9.91%

Footnotes

  1. 30 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@mtshiba
Copy link
Contributor Author

mtshiba commented Nov 29, 2025

Hmm, for example, when inspecting the following code, I see a clear performance improvement locally, but the change on codspeed seems less clear:

class Counter:
    def __init__(self: "Counter"):
        self.count = 0

    def increment(self: "Counter"):
        self.count = self.count + 1

reveal_type(Counter().count)  # revealed: Unknown | int

main:

# release
$ hyperfine -i "ty check widen.py"
Benchmark 1: ty check widen.py
  Time (mean ± σ):     103.3 ms ±   8.6 ms    [User: 56.9 ms, System: 45.0 ms]
  Range (min … max):    96.4 ms … 136.6 ms    22 runs

# debug
$ hyperfine -i "ty check widen.py"
Benchmark 1: ty check widen.py
  Time (mean ± σ):     899.9 ms ±   6.2 ms    [User: 849.7 ms, System: 51.9 ms]
  Range (min … max):   889.4 ms … 910.3 ms    10 runs

#21683:

# release
$ hyperfine -i "ty check widen.py"
Benchmark 1: ty check widen.py
  Time (mean ± σ):      59.6 ms ±   9.2 ms    [User: 26.5 ms, System: 37.8 ms]
  Range (min … max):    52.5 ms …  95.7 ms    31 runs

# debug
$ hyperfine -i "ty check widen.py"
Benchmark 1: ty check widen.py
  Time (mean ± σ):     267.2 ms ±   4.1 ms    [User: 222.2 ms, System: 55.6 ms]
  Range (min … max):   261.6 ms … 276.5 ms    10 runs

@mtshiba
Copy link
Contributor Author

mtshiba commented Nov 29, 2025

Conversely, the apparent performance degradation is due to increasing MAX_NON_RECURSIVE_UNION_LITERALS.
Even setting it to 512 did not result in a panic, which shows that the divergent cases are being correctly avoided.

The performance impact of changing MAX_NON_RECURSIVE_UNION_LITERALS is as follows:

190: https://codspeed.io/astral-sh/ruff/runs/compare/6929edd91e804897641888e6..6929fa9ed77b771deb1a9244
256: https://codspeed.io/astral-sh/ruff/runs/compare/6929edd91e804897641888e6..6929f4ded77b771deb1a9211
512: https://codspeed.io/astral-sh/ruff/runs/compare/6929edd91e804897641888e6..6929ef181e804897641888ec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

set a lower limit for size of literal unions we will do precise type inference of operations for

2 participants