Skip to content

Implement CPython 3.14 subinterpreter APIs - #8605

Draft
youknowone wants to merge 12 commits into
RustPython:mainfrom
youknowone:flatten-eval-loop
Draft

Implement CPython 3.14 subinterpreter APIs#8605
youknowone wants to merge 12 commits into
RustPython:mainfrom
youknowone:flatten-eval-loop

Conversation

@youknowone

Copy link
Copy Markdown
Member

Summary

  • add _interpreters, _interpchannels, and _interpqueues
  • implement cross-interpreter data conversion, exception snapshots, channel and queue lifecycle handling
  • add the CPython 3.14 concurrent.interpreters high-level API, including queues

CPython compatibility review

The implementation was compared against the CPython 3.14 sources, including argument parsing, error shapes, shareability checks, interpreter ID handling, channel defaults, queue semantics, and interpreter finalization order.

In particular, channel and queue cleanup now happens after module and atexit finalization. This preserves values written by finalizers and exposes them as UNBOUND, matching CPython behavior after the owning interpreter exits. The high-level concurrent.interpreters files are synchronized with CPython 3.14.7.

Validation

  • prek run --from-ref 9edd97ec4 --to-ref HEAD
  • cargo clippy
  • cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --exclude rustpython-capi
  • (cd crates/capi && cargo test) — 102 passed
  • targeted subinterpreter defect suite — 47 passed
  • targeted CPython/RustPython comparisons for queue finalization, public queue API, cross-interpreter transfer, default UNBOUND handling, and interpreter destruction
  • clean virtual merge with current upstream/main

AI assistance

Claude (claude-opus-5) assisted with implementation. OpenAI Codex (GPT-5) assisted with CPython source comparison, lifecycle review, validation, and PR preparation. The corresponding implementation commits include the required Assisted-by trailers.

@coderabbitai

coderabbitai Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[ ] lib: cpython/Lib/concurrent
[ ] test: cpython/Lib/test/test_concurrent_futures (TODO: 4)
[ ] test: cpython/Lib/test/test_interpreters
[ ] test: cpython/Lib/test/test__interpreters.py
[ ] test: cpython/Lib/test/test__interpchannels.py
[ ] test: cpython/Lib/test/test_crossinterp.py

dependencies:

  • concurrent (native: _crossinterp, _interpqueues, _interpreters, _queues, concurrent.futures, concurrent.futures._base, interpreter, itertools, multiprocessing.connection, multiprocessing.queues, multiprocessing.synchronize, process, sys, thread, time)
    • logging (native: atexit, collections.abc, email.message, email.utils, errno, http.client, logging.handlers, multiprocessing.queues, select, sys, time, urllib.parse, win32evtlog, win32evtlogutil)
    • multiprocessing (native: _multiprocessing, _posixshmem, _posixsubprocess, _winapi, array, atexit, collections.abc, connection, context, dummy, errno, forkserver, heap, itertools, managers, mmap, msvcrt, multiprocessing.connection, pool, popen_fork, popen_forkserver, popen_spawn_posix, popen_spawn_win32, queues, resource_sharer, resource_tracker, sharedctypes, spawn, synchronize, sys, time, util, xmlrpc.client)
    • pickle (native: _pickle, itertools, sys)
    • collections, functools, os, queue, threading, traceback, types, weakref

dependent tests: (17 tests)

  • concurrent: test_asyncio test_compileall test_concurrent_futures test_context test_genericalias test_inspect test_struct test_sys test_threading test_types test_wmi
    • asyncio: test_asyncio test_external_inspection test_logging test_os test_pdb test_unittest

[ ] test: cpython/Lib/test/test_import (TODO: 4)

dependencies:

dependent tests: (no tests depend on import)

[ ] test: cpython/Lib/test/test_class.py (TODO: 12)
[x] test: cpython/Lib/test/test_genericclass.py
[x] test: cpython/Lib/test/test_subclassinit.py

dependencies:

dependent tests: (no tests depend on class)

[x] test: cpython/Lib/test/test_descr.py (TODO: 31)
[ ] test: cpython/Lib/test/test_descrtut.py (TODO: 2)

dependencies:

dependent tests: (no tests depend on descr)

[x] test: cpython/Lib/test/test_cmd_line_script.py (TODO: 13)

dependencies:

dependent tests: (no tests depend on cmd_line_script)

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

@ShaharNaveh ShaharNaveh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

Apply the patch baseline for the low-level subinterpreter modules:

- crates/vm/src/stdlib/_interpreters.rs: create/destroy/list_all/get_main/
  get_current/is_running/exec/call/run_string/run_func/set___main___attrs/
  is_shareable/whence and the Interpreter*Error exceptions
- crates/vm/src/stdlib/_interpchannels.rs: channel create/destroy/send/recv/
  list_all/list_interpreters/release/close and the Channel*Error exceptions
- crates/vm/src/vm/crossinterp.rs: cross-interpreter data protocol
- crates/vm/src/vm/{interpreter,mod,runtime,vm_new}.rs: interpreter registry
  and lifecycle plumbing the modules need
- Lib/concurrent/interpreters/: high-level PEP 734 package

Assisted-by: Claude:claude-opus-5
Argument matching:
- Add `function::ArgSpec`, a `PyArg_ParseTupleAndKeywords` equivalent that
  applies the `|`, `$` and `:` markers of a format string to a `kwlist`, and
  route every module function of both modules through it. This enforces
  keyword-only parameters, rejects unexpected keywords, accepts keyword forms
  that were previously positional-only (`whence(id=)`, `get_config(id=)`,
  `set___main___attrs(id=, updates=)`, `capture_exception(exc=)`,
  `_register_end_types(send=, recv=)`), and reproduces the arity messages.
- `O!` slots (`shared`, `updates`, `call`'s `args`/`kwargs`) now reject None,
  and `_PyArg_BadArgument` renders None as "None" rather than "NoneType".
- `new_config` takes at most one positional `str`; `list_all` of
  `_interpchannels` is argument-less.
- Convert arguments in `kwlist` order, so an argument's own error is raised
  ahead of the checks that follow it.

_interpreters:
- Add the `CrossInterpreterBufferView` type and build the received memoryview
  on it, keeping the sending interpreter's exporter out of the destination.
- Build `excinfo.errdisplay` from `traceback.TracebackException`.

Feature flags:
- `os.fork` checks finalization before `allow_fork`, matching `os_fork_impl`.
- `_thread.start_new_thread` and `start_joinable_thread` reject an interpreter
  without `allow_threads`.

Lib/concurrent/interpreters/_crossinterp.py: restore the stripped docstrings
and comments.

Assisted-by: Claude:claude-opus-5
…t CPython 3.14

verify_stateless_function follows _PyFunction_VerifyStateless: it rejects
non-dict builtins, a non-empty __defaults__, __kwdefaults__ or __closure__,
and code whose LOAD_GLOBAL names are held by the function's globals or
absent from its builtins.

code_returns_only_none follows _PyCode_ReturnsOnlyNone: generator,
coroutine and async-generator code is rejected up front, the instruction
walk skips inline caches and maps specialized and instrumented opcodes
back, and the LOAD_CONST preceding each RETURN_VALUE is compared against
the index of None in co_consts.

ExcInfo::capture keeps an empty exception message instead of dropping it,
and builds `formatted` as `module.qualname: msg`, leaving out the builtins
and __main__ modules.

_interpreters.call packs the callable through _PyFunction_GetXIData before
pickle and re-raises the stateless-check failure when both fail. A func,
args or kwargs that cannot be rebuilt in the target, and a result without
cross-interpreter data, now raise NotShareableError with the snapshot as
the cause rather than being returned as excinfo.

_interpreters.capture_exception() without an argument returns None.

pickle_loads and _PyFunction_GetXIData attach the underlying failure as
__cause__.

channel_send converts the object to cross-interpreter data after the
channel's `closing` check.

ChannelID rich comparison returns the result of comparing its id with the
other number instead of coercing that result to bool.

PyMemoryView_FromObjectAndFlags raises "memoryview: a bytes-like object is
required, not 'X'" for an object that is not a buffer.

Assisted-by: Claude:claude-opus-5
Bound shareable ints by isize, raising OverflowError("try sending as
bytes") outside that range and keeping it as the NotShareableError cause.

Report an unshareable object by its repr, and attach a failing conversion
as __cause__ of the NotShareableError raised for the object it was called
for, so each level of a nested tuple appears in the chain.

Guard each tuple item conversion with with_recursion("while sharing a
tuple").

Gate the memoryview getdata function on `_interpreters` having been
imported, and route channel_send_buffer through a memoryview so it uses
that function and the resolved fallback.

Create the cross-interpreter exception types from the module_exec of
either module that raises them, instead of only `_interpreters`.

parse_cid raises OverflowError("int too big to convert"); int_arg
reproduces the `i` converter's three overflow messages and is also used
while parsing channel_send arguments.

Add BASETYPE to _queue.Empty, and gate nt.execv/nt.execve on allow_exec.

Assisted-by: Claude:claude-opus-5
_PyInterpreterState_ObjectToID accepts any object with __index__, raises
OverflowError("int too big to convert") outside the int64 range, and
reports a negative ID with the repr of the original object.

channelsmod_send reads the channel's default unboundop and fallback only
when one of the two arguments is negative, so an explicit pair is
validated before the channel is looked up.

Assisted-by: Claude:claude-opus-5
_interpqueues holds a process-wide queue table and exposes create,
destroy, list_all, put, get, bind, release, get_maxsize,
get_queue_defaults, is_full, get_count and _register_heap_types, raising
QueueError and QueueNotFoundError.  QueueEmpty and QueueFull, which
subclass queue.Empty and queue.Full, are registered per interpreter by
concurrent.interpreters._queues.

Cross-interpreter data carries a queue as a refcounted queue ID that
binds when the data is captured and releases when it is dropped, and
is_shareable reports a registered Queue as shareable.

Queue and channel items owned by an interpreter are now cleared from
Interpreter::finalize, after module finalization, rather than from
destroy_owned_interpreter, so values sent by an atexit callback also
become unbound.

is_shareable gates memoryview on the same registration flag as the
getdata lookup.

Assisted-by: Claude:claude-opus-5
Interpreter, ExecutionFailed and the queue aliases, copied verbatim from
CPython 3.14.

Assisted-by: Claude:claude-opus-5
Interpreter::finalize now runs finalize_subinterpreters between the atexit
handlers and the finalizing flag: when the main interpreter still owns
subinterpreters it emits the RuntimeWarning "remaining subinterpreters;
close them with Interpreter.close()" and then finalizes each of them.

runtime::owned_interpreter_ids lists the runtime-owned interpreters.

Assisted-by: Claude:claude-opus-5
_Py_abspath and _PyPathConfig_ComputeSysPath0 arrive as abs_path and
script_sys_path0. run_file hands the absolutized path to get_importer
and to the runner, so __file__ and co_filename no longer stay relative,
and inserts the symlink-resolved script directory as sys.path[0]
instead of the argument's parent.

test_cmd_line_script.test_script_abspath no longer expects a failure.

Assisted-by: Claude:claude-opus-5
The object, number, sequence and mapping protocols, the
unsupported-operand helpers, and the type-specific messages fed from
them now read PyType::slot_name() where they read PyType::name(),
so a type declared with a module keeps it.

Message texts corrected along the way:
- PySequence_Size, PyMapping_Size, PySequence_GetItem and
  PySequence_SetItem/DelItem split their merged wording in two, and
  PyObject_SetItem/DelItem say "object".
- PyObject_GenericSetAttr reports a read-only attribute and the
  missing __dict__.
- check_class drops the type name it appended.
- __int__ and __index__ drop the class prefix; __float__ keeps it.
- float(), complex(), round(), next(), str subscripting and list
  subscripting take their own wording.
- str.join and bytes.join number the item they turn down and name what
  they wanted, and report a non-iterable as "can only join an
  iterable".

test_class.testObjectAttributeAccessErrorMessages,
string_tests.test_subscript and the test_descrtut doctest no longer
expect a failure.

Assisted-by: Claude:claude-opus-5
crossinterp::script_code called vm.compile unconditionally, which only
exists with the rustpython-compiler feature. Builds without it, such as
example_projects/barebone, now raise a TypeError instead of failing to
compile.

Assisted-by: Claude:claude-opus-5
The test imports _testsinglephase in its body without a decorator
guarding it, and now that _interpreters exists requires_subinterpreters
no longer skips it.

Assisted-by: Claude:claude-opus-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants