From 0c2c78d02a9ea0d53c4ebbd7ebc67891bffe8417 Mon Sep 17 00:00:00 2001 From: Till Varoquaux Date: Sun, 16 Aug 2026 11:16:53 -0400 Subject: [PATCH] Update PEP 835 with refined rationale and ecosystem statistics - Move Format.TYPE to Rejected Ideas - Update formatting to '@ annot' throughout - Refine __rmatmul__ language to encourage it as a robust fallback - Integrate Annotated usage and migration stats --- peps/pep-0835.rst | 189 ++++++++++++++++++++++------------------------ 1 file changed, 92 insertions(+), 97 deletions(-) diff --git a/peps/pep-0835.rst b/peps/pep-0835.rst index edd6ce5f013..d5750894f82 100644 --- a/peps/pep-0835.rst +++ b/peps/pep-0835.rst @@ -15,7 +15,7 @@ Abstract ======== This PEP proposes overloading the ``@`` operator on types to allow writing -``Annotated[T, M]`` as ``T @M``. +``Annotated[T, M]`` as ``T @ M``. Motivation ========== @@ -36,6 +36,10 @@ However, its verbosity remains a barrier to adoption: warned that relying on ``Annotated`` for everyday documentation would make code unreadable, contributing to the PEP's withdrawal [5]_. +Of the 10,000 most downloaded PyPI projects, **43.4%** transitively depend on +``Annotated``. Migrating the FastAPI codebase to the new shorthand removed +2,524 LOCs. + A container-like syntax opposes Python's evolution. Python is actively replacing generic wrappers like ``typing.Union`` with operators like ``|`` (and the proposed ``&`` for intersections), evolving type annotations into an arithmetic @@ -60,25 +64,25 @@ This resolves the conflict between concise syntax and strict typing:: id: Annotated[int, Field(gt=0)] # Proposed: Concise while preserving PEP 593 semantics - id: int @Field(gt=0) + id: int @ Field(gt=0) As an operator, ``@`` composes natively:: # Inside generics list[Annotated[str, Field(max_length=50)]] - list[str @Field(max_length=50)] + list[str @ Field(max_length=50)] # Within a union Annotated[int, Ge(0)] | Annotated[str, Len(5)] - int @Ge(0) | str @Len(5) + int @ Ge(0) | str @ Len(5) # Across an entire union Annotated[str | None, Field(description="Optional string")] - (str | None) @Field(description="Optional string") + (str | None) @ Field(description="Optional string") # Within callables Callable[[Annotated[int, Ge(0)]], str] - Callable[[int @Ge(0)], str] + Callable[[int @ Ge(0)], str] Historical Context and Prior Art -------------------------------- @@ -110,7 +114,7 @@ apply: that evaluates to a valid type, as specified in the `Typing Specification `__ (:pep:`484`). -- **Type metadata**: e.g., ``int @``. Data attached to a type expression +- **Type metadata**: e.g., ``int @ ``. Data attached to a type expression via ``Annotated`` (:pep:`593`, :pep:`746`). - **Non-typing annotation**: e.g., `@no_type_check `__ @@ -145,7 +149,7 @@ type:: x: Annotated[int, Range(0, 10)] # Proposed shorthand - x: int @Range(0, 10) + x: int @ Range(0, 10) Operator Precedence ------------------- @@ -155,22 +159,22 @@ constraints can be attached directly to specific types within a union without parentheses. For example, a US zip code might be a 5-digit integer *or* a 5-character string:: - zip_code: int @Ge(10000) @Le(99999) | str @Len(5) + zip_code: int @ Ge(10000) @ Le(99999) | str @ Len(5) To attach metadata to the entire union, use parentheses:: # Attaches only to 'str' - int | str @Metadata # equivalent to: int | Annotated[str, Metadata] + int | str @ Metadata # equivalent to: int | Annotated[str, Metadata] # Attaches to the entire union - (int | str) @Metadata # equivalent to: Annotated[int | str, Metadata] + (int | str) @ Metadata # equivalent to: Annotated[int | str, Metadata] Flattening Multiple Metadata ---------------------------- -Chained metadata flattens the resulting ``Annotated`` object. ``T @m1 @m2`` +Chained metadata flattens the resulting ``Annotated`` object. ``T @ m1 @ m2`` evaluates to ``Annotated[T, m1, m2]``, never ``Annotated[Annotated[T, m1], m2]``. This mirrors ``typing.Annotated``'s existing runtime behavior. @@ -178,18 +182,18 @@ existing runtime behavior. This same logic applies when the left-hand operand is an existing ``Annotated`` type:: - Annotated[int, m1] @m2 # AnnotatedType(int, m1, m2) (flattened) + Annotated[int, m1] @ m2 # AnnotatedType(int, m1, m2) (flattened) Runtime Behavior ---------------- ``@`` produces a ``types.AnnotatedType`` (a new built-in C type). The existing ``typing.Annotated`` unifies with this type. ``typing.Annotated[X, Y]`` and ``X -@Y`` return the exact same object: +@ Y`` return the exact same object: .. code-block:: pycon - >>> type(int @Field()) is type(Annotated[int, Field()]) + >>> type(int @ Field()) is type(Annotated[int, Field()]) True >>> typing.Annotated is types.AnnotatedType True @@ -206,8 +210,8 @@ The ``repr()`` of an ``AnnotatedType`` uses the shorthand syntax: .. code-block:: pycon - >>> int @Field(gt=0) - int @Field(gt=0) + >>> int @ Field(gt=0) + int @ Field(gt=0) Handling of ``None`` -------------------- @@ -227,10 +231,13 @@ multiplication: If ``NoneType.__matmul__`` existed, this would silently return an ``AnnotatedType`` instead of raising a ``TypeError``. -``annotationlib.Format.TYPE`` sidesteps this limitation in typing -expressions. It evaluates structurally, correctly parsing ``None @Metadata`` -into an ``AnnotatedType``. Outside of type expressions, use ``Annotated[None, -Metadata]``. +Because ``NoneType`` lacks ``__matmul__``, expressions like ``None @ Metadata`` +will raise a ``TypeError`` at runtime. + +To support this safely at runtime, library authors are encouraged to implement +``__rmatmul__`` on their metadata objects. This provides a robust fallback for +types that do not invoke ``__matmul__`` on the left operand (such as +``NoneType`` or ``ForwardRef``). Supported Left-Hand Operands ----------------------------- @@ -244,49 +251,15 @@ constructs that support the ``|`` union operator (``types.GenericAlias``, Applying ``@`` to a class evaluates as type metadata; applying it to an instance performs arithmetic. -For example, ``int @Field()`` produces an ``AnnotatedType``, while ``42 @ +For example, ``int @ Field()`` produces an ``AnnotatedType``, while ``42 @ something`` raises a ``TypeError`` (or delegates to ``__rmatmul__``). -Likewise, ``ndarray @Field()`` produces an ``AnnotatedType``, even though +Likewise, ``ndarray @ Field()`` produces an ``AnnotatedType``, even though ``ndarray`` instances define ``__matmul__`` for matrix multiplication. Custom metaclasses can overload ``__matmul__`` provided ``@`` is avoided in type expressions. -Forward References and Deferred Evaluation -------------------------------------------- - -Under :pep:`749`'s lazy evaluation, existing ``annotationlib`` formats do not -preserve the structure of ``@`` expressions when names are unresolvable. -``Format.FORWARDREF`` stringifies unresolvable names:: - - class Model: - ref: NotYetDefined @Field(gt=0) - -This produces an opaque ``ForwardRef('"NotYetDefined" @Field(gt=0)')``. The -metadata is enclosed within the string, preventing libraries from easily -inspecting it. - -``Format.TYPE`` assumes typing semantics and evaluates type expressions -structurally: - -* **Unresolvable Names:** Unresolvable names are wrapped in a ``ForwardRef`` - independently, leaving operators intact. -* **Union Types:** The ``|`` operator evaluates to ``types.UnionType``. -* **Annotated Types:** The ``@`` operator evaluates to ``types.AnnotatedType``. -* **Metadata Boundary:** Structural evaluation applies only to the type space. - Expressions within the metadata (the right-hand side of ``@`` or the - subsequent arguments of ``Annotated``) are evaluated as values. Unresolvable - expressions there produce a single opaque ``ForwardRef`` (identical to - ``Format.FORWARDREF``). - -For example, the ``NotYetDefined @Field(gt=0)`` annotation evaluates into an -``AnnotatedType`` where the metadata remains immediately accessible:: - - AnnotatedType(ForwardRef('NotYetDefined'), Field(gt=0)) - -This also resolves the pre-existing limitation where ``"Foo" | int`` produced -``ForwardRef('Foo | int')`` under ``Format.FORWARDREF``. Rationale ========= @@ -295,7 +268,7 @@ Rationale operator provides identical semantics without nesting. Reusing the existing ``@`` operator leaves the Python parser unchanged. The -operator's new meaning is isolated to type evaluation, where ``T @M`` lowers to +operator's new meaning is isolated to type evaluation, where ``T @ M`` lowers to ``Annotated[T, M]``. Type checkers (Mypy, Pyright) also require no parser changes, handling the syntax directly during semantic analysis. We prototyped a Ruff conversion rule for automated migrations, and CPython prototype testing @@ -319,7 +292,7 @@ Selecting the correct operator for metadata involves balancing three considerations: 1. **Precedence:** Binding tighter than ``|`` (Union) and ``&`` (proposed - Intersection) ensures expressions like ``int @Field() | str`` parse correctly + Intersection) ensures expressions like ``int @ Field() | str`` parse correctly unparenthesized. This excludes operators like ``|``, ``^``, and ``&``. 2. **Ecosystem Compatibility:** New keywords require ecosystem-wide parser migrations. Because Python's type system is evolving into an arithmetic of @@ -345,6 +318,12 @@ operators like ``+`` or ``/``. Backwards Compatibility ======================= +Analyzing the top 10,000 PyPI projects revealed 0 metaclass overloads for +``__matmul__`` or ``__rmatmul__``. (For comparison, the analysis caught rare +``__or__`` and ``__and__`` overloads in libraries like ``pgpy`` (#1,690), +``dataclass-wizard`` (#2,205), and ``multimethod`` (#2,377)). + + The pure-Python ``typing._AnnotatedAlias`` class is replaced with a native C implementation (``types.AnnotatedType``). ``typing.Annotated`` becomes a reference to this C type rather than a special form with a custom metaclass. @@ -377,7 +356,7 @@ How to Teach This In Python, the ``@`` symbol already has an established association with metadata through decorators. The annotation shorthand extends this intuition to the type -system: ``int @Field(gt=0)`` reads as "``int``, decorated with ``Field(gt=0)``." +system: ``int @ Field(gt=0)`` reads as "``int``, decorated with ``Field(gt=0)``." For beginners, the key rule is: **in a type annotation, ``@`` means "with this metadata."** For experienced developers, the mental model maps directly to @@ -388,13 +367,6 @@ primary syntax for applying metadata. The verbose ``typing.Annotated`` form should be treated as an advanced detail, primarily relevant to library authors or when dynamically generating types. -**Visual Style:** Format the shorthand as ``type @annot`` (e.g., ``int -@Metadata(...)``), with a space before the ``@`` and no space after it. This -distinguishes it from standard matrix multiplication (``A @ B``) and aligns -visually with function decorators (``@decorator``) and Java annotations. Code -formatters (like Ruff and Black) should enforce this spacing within typing -contexts. - Usage Examples ============== @@ -405,9 +377,9 @@ scenarios:: from annotated_types import Len class Project(BaseModel): - name: str @Field(title="Project Name") @Len(1) - url: HttpUrl @Field(description="The project homepage") - stars: int @Field(ge=0) = 0 + name: str @ Field(title="Project Name") @ Len(1) + url: HttpUrl @ Field(description="The project homepage") + stars: int @ Field(ge=0) = 0 **FastAPI Dependency Injection:** In FastAPI, the shorthand simplifies complex parameter definitions:: @@ -417,7 +389,7 @@ parameter definitions:: app = FastAPI() @app.get("/secure") - async def secure_endpoint(token: str @Header(description="Auth token")): + async def secure_endpoint(token: str @ Header(description="Auth token")): return {"status": "authorized"} **SQLModel and Database Definitions:** SQLModel relies on ``Annotated`` to @@ -426,10 +398,10 @@ define column properties. The shorthand syntax cleans up these definitions:: from sqlmodel import SQLModel, Field class Hero(SQLModel, table=True): - id: (int | None) @Field(primary_key=True) = None - name: str @Field(index=True) + id: (int | None) @ Field(primary_key=True) = None + name: str @ Field(index=True) secret_name: str - age: (int | None) @Field(index=True) = None + age: (int | None) @ Field(index=True) = None **Testing and Formal Verification:** Libraries like Hypothesis and CrossHair use ``annotated-types`` to constrain test generation. The shorthand provides a clean @@ -442,14 +414,38 @@ syntax for specifying test boundaries:: @dataclass class InventoryItem: # A non-negative quantity - quantity: int @Ge(0) + quantity: int @ Ge(0) # A price bounded between 1 and 100 - price: float @Interval(gt=0, le=100) + price: float @ Interval(gt=0, le=100) @given(...) def test_inventory(item: InventoryItem): assert item.price * item.quantity >= 0 +Ecosystem Migration +=================== + +We validated this shorthand by migrating several major type-directed frameworks. +To ensure tests continued to pass, we used a two-phase approach: + +1. **Add Support**: We first updated the internal machinery of the libraries to + accept the ``@`` syntax. +2. **Move the Project**: We then migrated every internal usage of ``Annotated`` + to the ``@`` operator. + +The required modifications to add support were: + +* **FastAPI:** 0 LOCs (inherited from Pydantic) +* **Pydantic:** 42 LOCs +* **SQLAlchemy:** 280 LOCs +* **cattrs:** 153 LOCs +* **Hypothesis:** 13 LOCs +* **Beartype:** 10 LOCs + +Most of these changes were isolated to test suites and string representation +logic (like ``__repr__``) to maintain backward compatibility. The actual +evaluation logic required to support the ``@`` operator was negligible. + Reference Implementation ======================== @@ -457,9 +453,6 @@ Prototype implementations are available for the following tools: - **CPython:** `CPython at-type-annot `_ -- **CPython (with annotation-lib structural forward references):** `CPython - forward-stringifier - `_ - **Mypy:** `Mypy at-type-annot `_ - **Mypyc/ast_serialize:** `ast_serialize at-type-annot @@ -488,11 +481,11 @@ Debates around reusing ``@`` yielded several alternatives: These lacked consensus. The ``@`` symbol also extends cleanly to symbol decorators if the language pursues that route later. -Reliance on ``__rmatmul__`` ---------------------------- +Sole Reliance on ``__rmatmul__`` +-------------------------------- -We explicitly reject relying on metadata objects implementing ``__rmatmul__`` -(e.g., via a base class) to return an ``Annotated`` type:: +We explicitly reject relying *solely* on metadata objects implementing +``__rmatmul__`` (e.g., via a base class) to return an ``Annotated`` type:: class Metadata[T = object]: def __rmatmul__(self, typ: TypeForm[T], /) -> TypeForm[T]: @@ -525,18 +518,24 @@ with matrix multiplication. Within a type expression, Python already reuses standard operators (like ``|`` for unions and ``[]`` for generics) with typing-specific semantics (see `Why the @ Operator?`_). -Evaluating Type Expressions as Runtime Code -------------------------------------------- +Structural Evaluation Format (``Format.TYPE``) +---------------------------------------------- + +Early drafts proposed a new ``Format.TYPE`` in ``annotationlib`` (:pep:`749`) to +structurally evaluate the ``@`` operator and preserve metadata on unresolvable +``ForwardRef`` instances. -The design explicitly couples ``Format.TYPE`` to type expression semantics, -reinforcing the boundary between static typing and runtime evaluation: +This was rejected. Prototype integrations (``beartype``, ``pydantic``, +``fastapi``, ``sqlalchemy``, and ``hypothesis``) showed the existing ecosystem +handles the ``@`` operator using ``annotationlib``'s current tools. -* **Non-Typing Annotations:** Frameworks using custom ``@`` operators in - annotations remain supported via ``Format.STRING`` and ``Format.VALUE``. They - are incompatible with ``Format.TYPE``, which enforces typing semantics. -* **Structural Evaluation:** Because ``Format.TYPE`` handles the ``@`` operator - structurally, it can evaluate constructs like ``None @Metadata`` even though - the global ``NoneType`` does not implement ``__matmul__``. +No-Space Formatting (``@annot``) +-------------------------------- + +We initially considered formatting the shorthand without a space (e.g., +``int @Field``) to visually mirror function decorators. This was rejected +because it forces formatters (like Black and Ruff) to maintain complex, +context-dependent rules for the ``@`` operator. Future Work =========== @@ -600,10 +599,6 @@ Open Issues bypass the standard 5-year deprecation policy (:pep:`387`). Should we fast-track its removal? -**annotationlib.Format.TYPE Extraction:** ``Format.TYPE`` improves -type expression evaluation (e.g., properly resolving ``|`` unions). Does this -warrant a standalone PEP? - Acknowledgements ================