PyStructTypes

Leverage Python Types to Define C-Struct Interfaces

Reasoning

I made this project for 2 reasons:

1. I wanted to see if I could leverage the typing system to effectively automatically decode and encode c-type structs in python.
2. Build a tool to do this for a separate project I am working on.

I am aware of other very similar c-type struct to python class libraries available, but I wanted to try something new so here we are.

This may or may not end up being super useful, as there are quite a few bits of hacky metaprogramming to get the type system to play nicely for what I want, but perhaps over time it can be cleaned up and made more useful.

StructDataclass

The StructDataclass class is based off of the Dataclass class, and thus is used in a similar fashion.

Basic Structs

Basic structs can mostly be copied over 1:1

struct MyStruct {
    int16_t myNum;
    char myLetter;
};

@struct_dataclass
class MyStruct(StructDataclass):
    myNum: int16_t
    myLetter: char_t

s = MyStruct()
s.decode([4, 2, 65])
# MyStruct(myNum=1026, myLetter=b"A")
s.decode([4, 2, 65], little_endian=True)
# MyStruct(myNum=516, myLetter=b"A")

# We can modify the class values and encode the data to send back
s.myNum = 2562
s.encode()
# [10, 2, 65]

For arrays of basic elements, you need to Annotate them with the TypeMeta object, and set their type to list[_type_].

struct MyStruct {
    uint8_t myInts[4];
    uint16_t myBiggerInts[2];
};

@struct_dataclass
class MyStruct(StructDataclass):
    myInts: Annotated[list[uint8_t], TypeMeta(size=4)]
    myBiggerInts: Annotated[list[uint16_t], TypeMeta(size=2)]

s = MyStruct()
s.decode([0, 64, 128, 255, 16, 0, 255, 255])
# MyStruct(myInts=[0, 64, 128, 255], myBiggerInts=[4096, 65535])

You can also set defaults for both basic types and lists.

All values will default to 0 or the initialized value for the chosen class if no specific value is set.

List defaults will set all items in the list to the same value. Currently setting a complete default list for all values is not implemented.

struct MyStruct {
    uint8_t myInt = 5;
    uint8_t myInts[2];
};

@struct_dataclass
class MyStruct(StructDataclass):
    myInt: uint8_t = 5
    myInts: Annnotated[list[uint8_t], TypeMeta(size=2, default=1)]

s = MyStruct()
# MyStruct(myInt=5, myInts=[1, 1])
s.decode([10, 5, 6])
# MyStruct(myInt=10, myInts=[5, 6])

String / char[] Type

Defining c-string types is a little different. Instead of using size in the TypeMeta, we need to instead use chunk_size.

This is because the way the struct format is defined for c-strings needs to know how big the string data is expected to be so that it can put the whole string in a single variable.

The chunk_size is also introduced to allow for char[][] for converting a list of strings.

struct MyStruct {
    char myStr[3];
    char myStrList[2][3];
};

@struct_dataclass
class MyStruct(StructDataclass):
    myStr: Annotated[string_t, TypeMeta[str](chunk_size=3)]
    myStrList: Annotated[list[string_t], TypeMeta[str](size=2, chunk_size=3)]


s = MyStruct()
s.decode([65, 66, 67, 68, 69, 70, 71, 72, 73])
# MyStruct(myStr=b"ABC", myStrList=[b"DEF", b"GHI"])

If you instead try to define this as a list of char_t types, you would only be able to end up with MyStruct(myStr=[b"A", b"B", b"C"], myStrList=[b"D", b"E", b"F", b"G", b"H", b"I"])

The Bits Abstraction

This library includes a bits abstraction to map bits to variables for easier access.

One example of this is converting a C enum like so:

enum ConfigFlags {
    lights_flag = 1 << 0,
    platform_flag = 1 << 1,
};
#pragma pack(push, 1)

@bits(uint8_t, {"lights_flag": 0, "platform_flag": 1})
class FlagsType(BitsType): ...

f = FlagsType()
f.decode([3])
# FlagsType(lights_flag=True, platform_flag=True)
f.decode([2])
# FlagsType(lights_flag=False, platform_flag=True)
f.decode([1])
# FlagsType(lights_flag=True, platform_flag=False)

Custom StructDataclass Processing and Extensions

There may be times when you want to make the python class do cool fun python class type of stuff with the data structure. We can extend the class functions _decode and _encode to handle this processing.

In this example, lets say you want to be able to read/write the class object as a list, using __getitem__ and __setitem__ as well as keeping the data in a different data structure than what the c struct defines.

struct MyStruct {
    uint8_t enabledSensors[5];
};

@struct_dataclass
class EnabledSensors(StructDataclass):
    # We can define the actual data we are ingesting here
    # This mirrors the `uint8_t enabledSensors[5]` data
    _raw: Annotated[list[uint8_t], TypeMeta(size=5)]

    # We use this to store the data in the way we actually want
    _data: list[list[bool]] = field(default_factory=list)

    def _decode(self, data: list[int]) -> None:
        # First call the super function. This will store the raw values into `_raw`
        super()._decode(data)

        # Erase everything in self._data to remove any old data
        self._data = []

        # 2 Panels are packed into a single uint8_t, the left most 4 bits for the first
        # and the right most 4 bits for the second
        for bitlist in (list(map(bool, map(int, format(_byte, "#010b")[2:]))) for _byte in self._raw):
            self._data.append(bitlist[0:4])
            self._data.append(bitlist[4:])

        # Remove the last item in self._data as there are only 9 panels
        del self._data[-1]

    def _encode(self) -> list[int]:
        # Modify self._raw with updated values from self._data
        for idx, items in enumerate(list_chunks(self._data, 2)):
            # Last chunk
            if len(items) == 1:
                items.append([False, False, False, False])
            self._raw[idx] = sum(v << i for i, v in enumerate(list(itertools.chain.from_iterable(items))[::-1]))
            
        # Run the super function to return the encoded data from self._raw()
        return super()._encode()

    def __getitem__(self, index: int) -> list[bool]:
        # This lets us access the data with square brackets
        # ex. `config.enabled_sensors[Panel.UP][Sensor.RIGHT]`
        return self._data[index]

    def __setitem__(self, index: int, value: list[bool]) -> None:
        # Only use this to set a complete set for a panel
        # ex. `config.enabled_sensors[Panel.UP] = [True, True, False, True]`
        if len(value) != 4 or not all(isinstance(x, bool) for x in value):
            raise Exception("must set all 4 items at once")

s = EnabledSensors()
s.decode([15, 15, 15, 15, 0])

# The `self._data` here would look like:
# [
#   [False, False, False, False],
#   [True, True, True, True],
#   [False, False, False, False],
#   [True, True, True, True],
#   [False, False, False, False],
#   [True, True, True, True],
#   [False, False, False, False],
#   [True, True, True, True],
#   [False, False, False, False],
# ]

# With the get/set functioned defined, we can access the data
# with square accessors.
# s[1][2] == True 

StructDataclass is Composable

You can use StructDataclasses in other StructDataclasses to create more complex structs.

struct RGB {
    uint8_t r;
    uint8_t g;
    uint8_t b;
};

struct LEDS {
    RGB lights[3];
};

@struct_dataclass
class RGB(StructDataclass):
    r: uint8_t
    g: uint8_t
    b: uint8_t

@struct_dataclass
class LEDS(StructDataclass):
    lights: Annotated[list[RGB], TypeMeta(size=3])]

l = LEDS()
l.decode([1, 2, 3, 4, 5, 6, 7, 8, 9])
# LEDS(lights=[RGB(r=1, g=2, b=3), RGB(r=4, g=5, b=6), RGB(r=7, g=8, b=9)])

Future Updates

• Bitfield: Similar to the Bits abstraction. An easy way to define bitfields
• Potentially more ways to define bits (dicts/lists/etc).
• Potentially allowing list defaults to be entire pre-defined lists.
• ???

Examples

You can see a more fully fledged example in the test/examples.py file.