Skip to content

Allow encode_into arbitrary buffers. - #1160

Open
zdxerr wants to merge 5 commits into
msgspec:mainfrom
zdxerr:encode_into_memoryview
Open

Allow encode_into arbitrary buffers.#1160
zdxerr wants to merge 5 commits into
msgspec:mainfrom
zdxerr:encode_into_memoryview

Conversation

@zdxerr

@zdxerr zdxerr commented Aug 17, 2026

Copy link
Copy Markdown

Lets Encoder.encode_into() (both msgspec.msgpack and msgspec.json) accept any writable, contiguous buffer-protocol object (e.g. a memoryview onto multiprocessing.shared_memory.SharedMemory), not just bytearray.

Currently the encode_into only works on bytearrays, this makes it impossible to use Python's SharedMemory with a zero-copy serialization.

from multiprocessing import shared_memory
import msgspec

enc = msgspec.msgpack.Encoder()
shm = shared_memory.SharedMemory(
    name="shared_data",
    create=True,
    size=1024,  
)

enc.encode_into(Message(datetime.now().astimezone(), Severity.ERROR, "TEST"), shm.buf, 0)

fails with

TypeError: buffer must be a `bytearray`

This changes allows for arbitrary buffers that do not support resizing.

encode_into also return the number of newly written bytes. So offset + newly_written_bytes can by used as the offset for the next element when streaming data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant