pycapnp 2.2.2
A cython wrapping of the C++ Cap'n Proto library
License: BSD-2-Clause
Requires Python: >=3.8
Keywords: capnp, capnproto, Cap'n Proto, pycapnp

# pycapnp

[![Packaging Status](https://github.com/capnproto/pycapnp/workflows/Packaging%20Test/badge.svg)](https://github.com/capnproto/pycapnp/actions)
[![manylinux2014 Status](https://github.com/capnproto/pycapnp/workflows/manylinux2014/badge.svg)](https://github.com/capnproto/pycapnp/actions)
[![PyPI version](https://badge.fury.io/py/pycapnp.svg)](https://badge.fury.io/py/pycapnp)

[Cap'n'proto Mailing List](https://github.com/capnproto/capnproto/discussions) [Documentation](https://capnproto.github.io/pycapnp)


## Requirements

* C++14 supported compiler
  - gcc 6.1+ (5+ may work)
  - clang 6 (3.4+ may work)
  - Visual Studio 2017+
* cmake (needed for bundled capnproto)
  - ninja (macOS + Linux)
  - Visual Studio 2017+
* capnproto-1.0 (>=0.8.0 will also work if linking to system libraries)
  - Not necessary if using bundled capnproto
* Python development headers (i.e. Python.h)
  - Distributables from python.org include these, however they are usually in a separate package on Linux distributions

32-bit Linux requires that capnproto be compiled with `-fPIC`. This is usually set correctly unless you are compiling canproto yourself. This is also called `-DCMAKE_POSITION_INDEPENDENT_CODE=1` for cmake.

pycapnp has additional development dependencies, including cython and pytest. See requirements.txt for them all.


## Building and installation

Install with `pip install pycapnp`. You can set the CC environment variable to control which compiler is used, ie `CC=gcc-8.2 pip install pycapnp`.

Or you can clone the repo like so:

```bash
git clone https://github.com/capnproto/pycapnp.git
cd pycapnp
pip install .
```

By default, the setup script will automatically use the locally installed Cap'n Proto.
If Cap'n Proto is not installed, it will bundle and build the matching Cap'n Proto library.

To enforce bundling, the Cap'n Proto library:

```bash
pip install . -C force-bundled-libcapnp=True
```

If you wish to install using the latest upstream C++ Cap'n Proto:

```bash
pip install . \
    -C force-bundled-libcapnp=True \
    -C libcapnp-url="https://github.com/capnproto/capnproto/archive/master.tar.gz"
```

To enforce using the installed Cap'n Proto from the system:

```bash
pip install . -C force-system-libcapnp=True
```

The bundling system isn't that smart so it might be necessary to clean up the bundled build when changing versions:

```bash
python setup.py clean
```


## Stub-file generation

While not directly supported by pycapnp, a tool has been created to help generate pycapnp stubfile to assist with development (this is very helpful if you're new to pypcapnp!). See [#289](https://github.com/capnproto/pycapnp/pull/289#event-9078216721) for more details.

[Python Capnp Stub Generator](https://gitlab.com/mic_public/tools/python-helpers/capnp-stub-generator)


## Python Versions

Python 3.8+ is supported.


## Development

Git flow has been abandoned, use master.

To test, use a pipenv (or install requirements.txt and run pytest manually).
```bash
pip install pipenv
pipenv install
pipenv run pytest
```


### Binary Packages

Building a Python wheel distributiion

```bash
pip wheel .
```

## Documentation/Example

There is some basic documentation [here](http://capnproto.github.io/pycapnp/).

Make sure to look at the [examples](examples). The examples are generally kept up to date with the recommended usage of the library.

The examples directory has one example that shows off pycapnp quite nicely. Here it is, reproduced:

```python
import os
import capnp

import addressbook_capnp

def writeAddressBook(file):
    addresses = addressbook_capnp.AddressBook.new_message()
    people = addresses.init('people', 2)

    alice = people[0]
    alice.id = 123
    alice.name = 'Alice'
    alice.email = 'alice@example.com'
    alicePhones = alice.init('phones', 1)
    alicePhones[0].number = "555-1212"
    alicePhones[0].type = 'mobile'
    alice.employment.school = "MIT"

    bob = people[1]
    bob.id = 456
    bob.name = 'Bob'
    bob.email = 'bob@example.com'
    bobPhones = bob.init('phones', 2)
    bobPhones[0].number = "555-4567"
    bobPhones[0].type = 'home'
    bobPhones[1].number = "555-7654"
    bobPhones[1].type = 'work'
    bob.employment.unemployed = None

    addresses.write(file)


def printAddressBook(file):
    addresses = addressbook_capnp.AddressBook.read(file)

    for person in addresses.people:
        print(person.name, ':', person.email)
        for phone in person.phones:
            print(phone.type, ':', phone.number)

        which = person.employment.which()
        print(which)

        if which == 'unemployed':
            print('unemployed')
        elif which == 'employer':
            print('employer:', person.employment.employer)
        elif which == 'school':
            print('student at:', person.employment.school)
        elif which == 'selfEmployed':
            print('self employed')
        print()


if __name__ == '__main__':
    f = open('example', 'w')
    writeAddressBook(f)

    f = open('example', 'r')
    printAddressBook(f)
```

Also, pycapnp has gained RPC features that include pipelining and a promise style API. Refer to the calculator example in the examples directory for a much better demonstration:

```python
import asyncio
import capnp
import socket

import test_capability_capnp


class Server(test_capability_capnp.TestInterface.Server):

    def __init__(self, val=1):
        self.val = val

    async def foo(self, i, j, **kwargs):
        return str(i * 5 + self.val)


async def client(read_end):
    client = capnp.TwoPartyClient(read_end)

    cap = client.bootstrap()
    cap = cap.cast_as(test_capability_capnp.TestInterface)

    remote = cap.foo(i=5)
    response = await remote

    assert response.x == '125'

async def main():
    client_end, server_end = socket.socketpair(socket.AF_UNIX)
    # This is a toy example using socketpair.
    # In real situations, you can use any socket.

    client_end = await capnp.AsyncIoStream.create_connection(sock=client_end)
    server_end = await capnp.AsyncIoStream.create_connection(sock=server_end)

    _ = capnp.TwoPartyServer(server_end, bootstrap=Server(100))
    await client(client_end)


if __name__ == '__main__':
    asyncio.run(capnp.run(main()))
```

Changelog
=============
## v2.2.2 (2026-01-16)
- Revert Data fields to bytes and add get_data_as_view for zero-copy access (#390)
- Fix use-after-free in async write causing corruption with large payloads (#392)
- Update macOS builds from 13 to 15 (#393)

## v2.2.1 (2025-10-21)
- Make `message.to_dict()` return bytes for DATA type field (#386)

## v2.2.0 (2025-09-12)
- Add binary support in dictionaries via base64 encoding (#351)
- Add structure-free read_multiple_bytes_packed (#378)
- Support python custom message builder and make Data field's type return MemoryView (#380)

## v2.1.0 (2025-09-04)
- Add Python 3.13 support
  * Python 3.8 is still enabled but will be disabled if changes are needed that prevent compatibility with newer versions of Python (e.g. 3.14)
  * Disabling experimental Python 3.14 build as it currently causes build issues
- Upgrade to Cython version 3
- Include \_custom_build in sdist
- Remove usage of deprecated kj::mvCapture functions
- Make license information SPDX-compatible
- Generate a new certificate that is compatible with strict x509 checking
- Avoid storm of 'warning: moving a temporary object prevents copy elision'
- cibuildwheel updates
- Fix deprecation warning when importing a schema

## v2.0.0 (2024-01-19)
- Updated link for mailing list in README

## v2.0.0b2 (2023-11-25)
- Fix broken test in test_load (#329)
- Update README example to async code (#331)
- Fix 'AttributeError: '\_UnixSelectorEventLoop' object has no attribute 'call_soon'
- Delete and update some Python 3.7-specific todo notes
- Make a server fail early when the KJ loop is not running
- Update documentation to async code (#331) (#332)
- Fix retransmit bug for large messages causing message corruption
- Unlock the GIL for all capnp functions that do IO
- Handle exceptions from server callbacks
- Disable the use of ninja for windows builds
- DynamicCapabilityClient fix
- Make `reraise_kj_exception` available to downstream
- Support `_DynamicListReader` in `_setDynamicField`
- Fix re-raising of KjException
- Allow cancellation of all capability contexts
- Corner case for cancelled server methods that raise exceptions
- Some fixes to the magic import system

## v2.0.0b1 (2023-10-03)
- Update to bundled capnproto-1.0.1
- Remove support for Python 3.7
- Use custom build backend to support build args (#328)
- Update Cython version and Python to 3.12 (#320)
- Wrap all capnp code in a context-manager to avoid segfaults (#317)
- Schema loading from the wire (#307)
- Make pycapnp more GIL friendly (#308)
- Use cibuildwheel in ci (#309)
- Integrate the KJ event loop into Python's asyncio event loop (#310)
- Allow capability implementation methods to be `async` (#312)
- Allow reading and writing messages from sockets in `async` mode (#313)
- Remove the synchronous RPC mode (#315)

## v1.3.0 (2023-01-26)
- Update to bundled capnproto-0.10.3
- Add Python 3.11 to Github Actions builds (#306)
- Prevent race condition in example code (#305)

## v1.2.2 (2022-12-01)
- Update bundled bundled capnp to 0.8.1 due to CVE-2022-46149
- Bundle lib/capnp_api.h and helpers/capabilityHelper.cpp (#301)
- Avoid reading random values for reader options from dangling reference (#300)

## v1.2.1 (2022-09-11)
- Fix packaging for Apple Silicon

## v1.2.0 (2022-08-29)
- Added support for Apple Silicon

## v1.1.1 (2022-05-23)
- Added Python 3.10 support
- aarch64 wheel support
- Fix doc string for `_DynamicResizableListBuilder`
- fix for unreleased buffers under mmap (issue 280)

## v1.1.0 (2021-06-09)
- Validated compatibility with Python 3.10.0b2
- Remove all bare except
- Improve `_StructModuleWhich` to inherit from `enum.Enum`
- Add Union on top level union messages
- Fixed memory leak in `_SegmentArrayMessageReader`
- Removed many pycodestyle warnings
- Avoid crash if `__file__` is not set by importer
- Fixed module.pyx `_set_<field>` for boolean fields
- Fixed setup.py.tmpl support for `*.c++` files
- Fixed `_gen.py` for python3 as `dict_keys` object are not indexable.
- Add test data to sdist
- Add `pyproject.yaml`
- Add missing inheritance to `_Schema` for `_StructSchema`

## v1.0.0 (2020-11-20)
- Validated Python 3.9 (3.7 and 3.8 are also supported)
- Updated package to include LICENSE file
- Updated examples to avoid run_forever() as ctrl+c will not work
- Adding xfail to pytest cases which fail sometimes due to network port oddities (please use asyncio, as Python handles things more gracefully)

## v1.0.0b2 (2020-06-14)
- Minimum capnproto version is now 0.8.0
- Added asyncio ssl calculator test
- Added poll_once to TwoPartyServer API
- More cleanup
- Fix absolute and circular imports
- Fix Promise aliasing issue (Promise to \_Promise)
- Documentation update
- Updated installation instructions
- Added RPC documentation for asyncio

## v1.0.0b1 (2019-12-26)
- Python 3.7+ required (asyncio support)
- TLS/SSL support using asyncio
- Windows support
- General cleanup
- May be incompatible with code written for pycapnp 0.6.4 and lower
- Removing pypandoc/pandoc packaging requirement
- Minimum capnproto version is now 0.7.0

## v0.6.4 (2019-01-31)
- Fix bugs in `read_multiple_bytes` (thanks to @tsh56)
- Remove end-of-life Python versions 2.6, 3.2, and 3.3. Add CI tests for 3.6
- Expose SchemaParser in Cython header

## v0.6.3 (2018-01-14)
- Bump bundled capnp version to v0.6.1 (thanks to @E8Yuval)
- Fix a memleak in RemotePromise (thanks to @E8Yuval)

## v0.6.2 (2017-11-30)
- Add support for buffers/memoryviews in `from_bytes` (thanks to @aldanor)

## v0.6.1 (2017-07-27)
- Fixed upload to PyPi (forgot to cythonize)

## v0.6.0 (2017-07-27)
- Update bundled capnp version to v0.6.0 and fix related problems (thanks to @benmoran)
- Fix memleak with KjException (thanks to @tsh56)

## v0.5.12 (2017-04-18)
- Bump bundled capnp version to v0.5.3.1

## v0.5.11 (2017-04-10)
- Make enums hashable (thanks to @madeleine-empirical)
- Rework logic on when to build bundled libcapnp. Fixes cross-compilation (thanks to @benizl)
- Add traversal_limit_in_words and nesting_limit to RPC classes (thanks to @asilversempirical)
- Include class attributes in __dir__. This allows for code completion of class methods (thanks to @chaoflow )
- Allow setting lists with python tuples (thanks to @chaoflow)
- Fix traversal_limit_in_words and nesting_limit being ignored by `from_bytes` (thanks to @plesner)

## v0.5.10 (2016-11-28)
- Fix bug that prevented event loop from actually being lazy initialized
- Fix possible recursive loop in KjException
- Add `clear_write_flag` method to builder classes

## v0.5.9 (2016-07-07)
- Make the event loop be lazy initialized
- Add support for segment (de)serialization (thanks to @gcv). See to_segments/from_segments methods.
- Fix response objects not referencing parents correctly
- Add test for large reads

## v0.5.8 (2016-05-27)
- Fix build problem with Cython v0.24
- Include the changelog in the manifest (should fix install problems if pandoc is present)
- Include the traceback in exceptions
- Make sure to encode to utf-8, not the default encoding (thanks to @novas0x2a)
- Add --libcapnp-url option in installer to allow installing arbitrary libcapnp versions
- Support mmap objects for reading with from_bytes (thanks to @bpiwowar)
- Change read_multiple and read_multiple_packed to copy by default
- Fix mistakenly discarding the file parameter on reads
- Add reraise_kj_exception to the prettyPrint functions. (thanks to @kdienes)
- Fix KjException init (missing wrapper). (thanks to @E8-Storage)
- Add `result_type` to InterfaceMethodSchema


## v0.5.7 (2015-06-16)
- Update bundled libcapnp to v0.5.2
- Add warnings for using old restorer methods. You should use `bootstrap` instead
- Fix warning from PyEventPort
- Handle AnyPointers better as arguments to RPC functions
- Add support for using keyword arguments with a named struct in an RPC
- Add bootstrap method to TwoPartyServer
- Add `init` method to lists
- Add support for unix sockets in RPC

## v0.5.6 (2015-04-13)
- Fix a serious bug in TwoPartyServer that was preventing it from working when passed a string address.
- Fix bugs that were exposed by defining KJDEBUG (thanks @davidcarne for finding this)


## v0.5.5 (2015-03-06)
- Update bundled C++ libcapnp to v0.5.1.2 security release


## v0.5.4 (2015-03-02)
- Update bundled C++ libcapnp to v0.5.1.1 security release
- Add bootstrap RPC methods
- Fix possible segfault when importing multiple schemas


## v0.5.3 (2015-02-23)
- Fix possible crash due to bad destructor ordering in MessageReader (by @JohnEmhoff)
- Default to no longer using cython


## v0.5.2 (2015-02-20)
- Add read\_multiple\_bytes/read\_multiple\_bytes\_packed methods
- Added Python 3.4 to the travis build matrix
- Bump version for bundled C++ libcapnp to v0.5.1


## v0.5.1 (2014-12-27)
- Remove installation dependency on cython. We now have no dependencies since libcapnp will automatically build as well.


## v0.5.0 (2014-12-15)
- Timer class `capnp.getTimer()`
- pycapnp is now thread-safe and allows an event loop to be run in each thread
  - You must destroy and re-create the event loop to get this functionality (see `test_threads.py`)
- Inheritance now works correctly for interfaces (previously inherited methods were inaccessible from pycapnp)
- Add ability to import modules with dashes or spaces. Use underscores in place of them
- `from_bytes` with builder=True is no longer zero copy. It never worked correctly, and is much safer now
- Add `num_first_segment_words` argument wherever message creation can occur
- Allow restoring a null objectId by passing None to restore
- Support ordered dictionary in `to_dict`
- Add ListSchema class and schemas for native types under `capnp.types` which completes all the Schemas needed to be wrapped. See `test_schema.py` for examples using it
- Add automatic build of C++ libcapnp if it's not detected on the system. Also add flags --force-bundled-libcapnp and --force-system-libcapnp respectively


## v0.4.6 (2014-9-10)
- Fix build for new 0.21 release of Cython. 0.21 is now the minimum supported version of Cython.


## v0.4.5 (2014-6-26)
- Fix `to_dict` not converting enums to strings


## v0.4.4 (2014-04-25)
- Fix compilation problem with gcc 4.8


## v0.4.3 (2014-02-18)
- Fix problem with uninitialized unions in \_from\_dict
- Add accesible version numbers for C++ libcapnp


## v0.4.2 (2014-02-13)
- Remove onDrained since it was removed upstream
- Replace usage of strings as enum type with custom `_DynamicEnum` class.
- Also change `Struct.which()` method to be a property `Struct.which` and return an enum type (`_DynamicEnumField`, which behaves much like `_DynamicEnum`).
- TwoPartyServer.run_forever() now will handle more than 1 simulataneous connection.
- Change exception wrapper to detect and raise AttributeError for field lookup exceptions (Fixes problem in Python3.x `__dir__`)
- Allow setting of fields with python dicts.


## 0.4.1 (2013-12-18)
- Remove python 3.2 from travis tests. Python 3.2 still should work fine, but it's more trouble than it's worth to write unicode tests that work in both it and Python2.
- Fix problems with null characters in Text/Data fields. Fixes #19


## 0.4.0 (2013-12-12)
- Initial working version of RPC
- Add get_root_as_any to _MessageReader
- Add capnp.pxd for public declarations of cython classes
- Fix problems compiling with gcc4.7


## v0.3.18 (2013-11-05)
- Change naming of ReaderOption parameters to be pep8 compliant


## v0.3.17 (2013-11-05)
- Add ReaderOptions to read/read_packed/from_bytes


## v0.3.16 (2013-10-28)
- Add defaults flag to capnp-json. Also remove 'which' field
- Add capnp-json serializer script. Also fix bugs in from_dict
- Fix build for clang/python3. Also remove -fpermissive
- Add `as_builder` method to Struct Reader
- Add warning when writing the same message more than once
- First working version of capability interfaces
- Wrap InterfaceSchema
- Fix setting string fields to support all types of strings
- Fix changed API for DynamicObject/ObjectPointer


## v0.3.15 (2013-09-19)
- Add not having installed the C++ libcapnp library to 'Common Problems'
- Add _short_str function for use in capnp_test_pycapnp.py
- Add test script for testing with https://github.com/kaos/capnp_test
- Add handling of DynamicObject
- Fix lists of lists or dicts for from_dict


## v0.3.14 (2013-09-04)
- Fix problem with to_dict


## v0.3.13 (2013-09-04)
- Add _DynamicStructBuilder.to_bytes() and <struct module>.from_bytes()
- Change == on StructSchema to return cbool
- Add Builder and Reader ABCs for each struct type

## v0.3.12 (2013-09-03)
- Fix handling of empty path '' in load_module
- Add from_dict
- Fix bug in exception handling for which(). Also standardize exceptions.
- Change import hook to require modules to end in '_capnp'
- Add import monkey patch function.
- Change naming for functions to conform to PEP 8. Also deprecate old read/write API
- Update preferred method for reading/writing messages from files

## v0.3.11 (2013-09-01)
- Forgot to change project name in setup.py

## v0.3.10 (2013-09-01)
- Change all references to old project name (change from capnpc-python-cpp to pycapnp)
- Change DynamicValue.Reader lists to be returned as _DynamicListReader
- Unify setters for DynamicList and DynamicStruct
- Add shortcuts for reading from / writing to files.  In Python, it doesn't make much sense to force people to muck around with MessageReaders and MessageBuilders since everything is landing on the heap anyway.  Instead, let's make it easy:  MyType.read[Packed]From(file) reads a file and returns a MyType reader.  MyType.newMessage() returns a MyType builder representing the root of a new message.  You can call this builder's write[Packed]To(file) method to write it to a file.
- Store Builders by value rather than allocate them separately on the heap (matches treatment of Readers).  v0.3 fixes the bug that made this not work.
- Wrap MessageBuilder::setRoot().
- Add tests based on TestAllTypes from the C++ test.capnp.  Fix problems uncovered in capnp.pyx.
- Implement __str__ and __repr__ for struct and list builders.  __str__ uses prettyPrint while __repr__ shows the type name and the low-whitespace stringification.  Also implement __repr__ for StructSchema, just because why not?


## v0.3.9 (2013-08-30)
- Change load to use a global SchemaParser. Make structs settable as field
- Add docstrings for new functions and _DynamicResizableListBuilder


## v0.3.8 (2013-08-29)
- Add initial tests
- Add _capnp for original Cython module. Meant for testing.
- Lowercase schema so it conforms to member naming conventions
- Expose _StructSchema's raw node
- Add some useful _StructSchema, reader, and builder methods
- Add full orphan functionality. Also, allow special orphan lists
- Finish up adding docstrings to all public classes/methods


## v0.3.7 (2013-08-26)
- Add a ton of docstrings and add to official docs
- Add DynamicOrphan


## v0.3.6 (2013-08-26)
- Add intersphinx for linking to python docs
- Add C++ library version check


## v0.3.5 (2013-08-25)
- Add handling of constants in schemas
- Fix new error with DynamicValue.Builder no longer being copyable


## v0.3.4 (2013-08-22)
- Fix Void namespace change
- Updated capnp schema to conform with new union rules


## v0.3.3 (2013-08-22)
- Fix for the removal of DynamicUnion from the C++ API


## v0.3.2 (2013-08-21)
- Add MANIFEST.in to include README


## v0.3.1 (2013-08-21)
- Update docs with lines about upgrading setuptools


## 0.3.0 (2013-08-21)
- Initial commit of docs
- Add querying unnamed enums to structs


## 0.2.1 (2013-08-13)
- Fix enum interface change for benchmark
- Random formatting cleanup
- Allow import paths in the schema loader
- Add travis CI


## 0.2.0 (2013-08-12)
- Initial working version