cstruct2 1.0.2b0
A library for handling C-style packed binary structures easily.
Requires Python: >=3.7

# *cstruct2*

Version: 1.0.2 (Beta)

*cstruct2* is a Python library that hopes to simplify and ease the experience of working with C-style packed binary structures. That is, if you were to directly write the memory of a packed (not padded, there's a difference!) structure in C to a file, this library would be able to hasslelessly read them. There exists a myriad of other means to do this in Python, such as reading each byte by hand or using Python's native *struct* library, but these are clunky and hard to read solutions. *cstruct2* provides a metaprogramming interface, wherein you define a class decorated by *cstruct2* that defines each field to be read, in an intelligent way.

The reading of fields of a certain type and explicit width is a given, but *cstruct2* allows you to base a field's length off of the value of another previously read field, making code easier to write and read. Not only that, you can tell a *cstruct2* decorated object to read a *different* field based off of the value of a previously read field, which is highly convenient if you're trying to parse a dusty network protocol (SOCKS5 comes to mind). Of course, it wouldn't be complete without allowing you to recursively read nested *cstruct2* objects within one another, which *cstruct2* allows you to do! There's more than that, of course, but those are the main highlights and superiorities of *cstruct2*.

## Basic Usage

Each field is a member of a decorated class, where the type of the data item is given as a type annotation and the value of that data item is its size, in most cases. For example, if we wanted to define an object *MyStructure* which has an integer 4 bytes in length--*cstruct2* assumes Little Endian ordered integers--and an ASCII string 32 bytes in length:

    from cstruct2.cstruct2 import cstruct2
        
    @cstruct2
    class MyStructure:
	    my_number: int = 4
	    my_string: str = 32
   	  

Every *cstruct2* derived structure will then immediately be instantiated upon invoking its name, where a *.from_stream()* member will be exposed. One will call this method with a Python *file*-like object, most importantly an object with the *read()* method for reading bytes in a stream. This can be from a socket, from an actual file, from a BytesIO or StringIO object, or anything really. Upon invoking this member on such an object, a dictionary will be returned with the values read. In the example above, the following dictionary is within the realm of possibility upon the *.from_stream()* member being called:

    with open("file.bin", "rb") as fp:
	    print(MyStructure.from_stream(fp))

	-->

    {
		"my_number": 256,
		"my_string": "This is a string."
	}


## Primitive Types and Variable Length

Because we are dabbling in the wonders of Python metaprogramming, we've kind of created our own mini programming language. *cstruct2* has a few primitive types, which are just simple standard data types that have a length and can be read in. *cstruct2* currently supports the following primitive types, which are denoted on each field as a Python type annotation:

 - *str* - Any kind of string, could be encoded in a myriad of different ways.
 - *int* - Any kind of integer, which can be encoded in either Little or Big Endian, with differing widths.
 - *float* - Any kind of floating point number, with widths 4 or 8 bytes and differing endianness.
 - *bytes* - An arbitrary number of bytes, where a Python *bytes* objects will be returned to store the bytes.
 - (Future) (For writing only) *anyfield*: allows any type to be written to a stream with the structure, with the field being provided at runtime. When declaring this in the structure, it must be initialized to *None*. *cstruct2* will throw an exception if this field is in a structure that is reading from a stream. When writing the structure to a stream, this field's value in the data dictionary must be a tuple of *(field tuple, value)*. 

Already, if you've been paying attention, we've encountered a problem. With all of the variations described above, how can we tell *cstruct2* about all of them, with one value in the object field declaration? If we want to get more specific about our field, we will have to make its value a tuple, a tuple of more specific information in addition to the field's width. For example, if we want to read in an integer worth 2 bytes but encoded as Big Endian, we'd have to declare:

    @cstruct2
    class MyStructure:
	    number: int = ("big", 2)

Now the question is, what are the available options for settings in tuples, and what are their orders, for each primitive data type? Below, we will list the different settings, exceptions, and orders for each data type, when using them in tuple-form:

 - *str*: (*length*, *encoding*, *wrapper*)
	 - The *encoding* field may be any string encoding value that Python uses when decoding strings (e.g., "utf-8" or "ascii"). 
	 - If *length* is "null"--and this is really important--then a null-terminated string will be read in. That is, *cstruct2* will keep reading bytes until it gets a '\0' byte.
	 - If *length* is "pascal", then the string's length will be determined by the first byte that precedes it, as in a Pascal-style string.
	 - (Future): If *length* is "pascal16" or "pascal32", then the same as regular "pascal" above, but with 16 bit and 32 bit lengths.
 - *int*: (*endianness*, *length*, *wrapper*)
	 - *endianness* can be explicitly declared by using "big" or "little", but "host" and "network" are both options, which are relative endianness options.
 - *float*: (*endianness*, *length*, *wrapper*)
	 - *endianness* has the same string enumerations as above.
 - *bytes*: (*length*, *wrapper*)
 - (Future) (For writing only) *anyfield*: *Structure value required to be None*

You may have noticed a *wrapper* field at the end of every tuple, along with how it was left unexplained in each bullet-point. The *wrapper* field allows you to pass in an object or a function that will receive the read in field and will convert it to another object or datatype afterwards, when it finally puts it into the resulting dictionary, if in structure reading mode. For example, if we want to read in a *str* but then want to cast it to our custom *User* class:

    class User:
	    def __init__(self, usr: str):
		    self.usr = usr
    
    @cstruct2
    class MyStructure:
	    username: str = ("null", User)


Pretty cool. But what if your C structure first sends a *uint16_t* describing the number of bytes that the next string it sends will be? When parsing that manually, you can very easily become confused. However, *cstruct2* allows you to base the length of a later field on the currently read value of a prior field, by simply referencing that field as a string when you pass in a *str* as a length argument. For example:

    @cstruct2
    class MyStructure:
	    len: int = ("little", 2)
	    string: str = "len"

This is the power of variable/derived lengths in *cstruct2*--and it's simply amazing. Anywhere a field is supposed to have a length or a width, you can simply pass in a string that references a previously read field, PROVIDED that the field is an integer type within the *cstruct2* decorated structure. It doesn't matter if you cast the field later on to an *int* by cleverly using wrappers, it **must** be an *int* field within the structure.

## Arrays

Of course, no structure parsing library is complete without the ability to read repeated elements, based off of another field's value. Very commonly are we asked to read in *n* items when dealing with such structures. Simply, arrays in *cstruct2* are declared by setting the field's value to that of a list, where the length of the array is the first element, then the settings of the underlying element(s) in the second and onward elements. For example:

    

	class User:
	    def __init__(self, usr: str):
		    self.usr = usr
		    
	@cstruct2
    class MyStructure:
	    no_names: int = 4
	    names: str = [
		    "no_names",
			16,
			User
	    ]

This structure will read in *no_names* number of 16 byte strings, which will then all be casted to *User* objects, since that is what the wrapper was set to in the *str* field's options. 

The resulting dictionary may look something like this:

    {
	    "no_names": 3,
	    "names": [
		    <User obj blah blah 1>,
		    <User obj blah blah 2>,
		    <User obj blah blah 3>
	    ]
    }

If one wants to wrap the resulting array into something, then the last argument of the list describing the array should be a callable type which then returns the wrapper, in the same way any other field can be wrapped.

## Switch Fields

It is also very common for the field we read to be different based off of the value of another previously read field. If the SOCKS5 protocol tells us the address type can be IPv4, IPv6, or a string domain, we need to have some variability in our code. This is exactly what a switch field allows us to do.

A switch field is made by declaring a field to be that of a *switch* type, then the value being a switch object which has two instantiation arguments:

 - *dependent_value*: str, the previously read field we base our decisions on
 - *decisions*: dict, a map of what each value as read from *dependent_value* should be as a another *cstruct2* field.

Due to the nature of the declared switch statement, the field type cannot be given as a type annotation. Instead, it must be given as the first argument of a tuple describing each possible field in a decision mapping. Of course, the last argument of the tuple for each decision field would describe a wrapper.

Of course, with things like these, it's best to explain additionally with just an example:

    from cstruct2.cstruct2 import cstruct2, switch
    
    @cstruct2
    class MyStructure:
	    usr_type: int = 1
	    user: switch = switch(
		    "usr_type", {
			    0: (int, "little", 4),
				1: (str, "null"),
				2: (int, [3, "big", 8])
			}
		)

The structure is saying: if *usr_type* is equal to 0, then read in a Little Endian 4 byte integer and place that into *user* when we're done; if *usr_type* is equal to 1, then read in a null terminated string; finally, if *usr_type* is equal to 2, then read in an array of ints 2 elements in length, where each integer is a Big Endian 8 byte integer. Wow, would that be a nightmare to manually write!

## Nested *cstruct2* structures

As with C structures, we can have nested *cstruct2* structures, where the result will be placed in a subobject in the main dictionary that parsing would yield. The field's type annotation for a nested *cstruct2* structure must be that of *structure*, with the value equal to the *cstruct2* object representing that nested structure--and if you use a tuple, the last argument of that tuple would be some wrapper which would be fed the corresponding read dictionary of the nested structure. An example only will be provided, because it's crystal clear how it works:

    from cstruct2.cstruct2 import cstruct2, structure
    
    @cstruct2
    class UserStructure:
	    id: int = 4
	    username_len: int = ("big", 2)
	    username: str = "username_len"
    
    @cstruct2
    class MainStructure:
	    code: int = 1
	    users_len: int = 2
	    users: structure = [
		    "users_len",
		    UserStructure
	    ]

The resulting dictionary may look like something like this:

    {
    	"code": 27272727,
    	"users_len": 2,
    	"users": [
    		{
				"id": 1,
				"username_len": 5,
				"username": "hello"
			},
			{
				"id": 2,
				"username_len": 6,
				"username": "hello2"
			}
        ]
    }

It's breathtaking, if you've ever toiled away parsing packed binary structures manually. The amount of bugs, the amount of headaches, and the amount of silly mistakes...  Let's write some C code to compare!

    struct __attribute__((__packed__)) UserStructure {
    	uint64_t id;
    	uint16_t length;
    };
    
    struct __attribute__((__packed__)) MainStructure {
    	uint8_t code;
    	uint16_t users_len;
    };
    
    ...
    
    int main(void) {
    	FILE *fp = fopen("file.bin", "rb");
    	
    	struct MainStructure mstruct;
		fread(&mstruct, 1, sizeof(mstruct), fp);
		for (uint16_t i = 0; i < mstruct.users_len; i++) {
			struct UserStructure ustruct;
			fread(&ustruct, 1, sizeof(ustruct), fp);
			uint16_t little_len = ntohs(ustruct.username_len);
			char *username = (char*) malloc(ustruct.little_len + 1);
			fread(username, 1, ustruct.little_len, fp);
			username[ustruct.little_len] = '\0';
			
			printf("[%d]: (%d) %s", i, ustruct.id, username);
			free(username);
		}

		fclose(fp);
    	return 0;
    }

The manual Python version, if you're not using *struct*, is not much better (and in fact, can be longer!). 

## Writing Structures

It's natural to also want to write these structures back to a stream or to some other output, so naturally *cstruct2* supports it. Writing structures isn't scattered throughout the previous sections because it does not differ much from regular *cstruct2* structure reading mode. There are some minute differences, but they are minute enough to be all under one inclusive section.

A *cstruct2* parsed structure will have the *to_stream()* and *to_bytes()* methods that will write the structure to either a stream or will write it to bytes which will be returned. In both of those methods, a dictionary containing the values for each defined field must be passed as the first argument. Obviously, *to_stream()* then needs the stream as its second argument. Let's give an example:

    @cstruct2
    class MyStructure:
    	age: int = ("little", 4)
    	name: str = "null"
    	things: int = [3, "little", 2]
    
    with open("out.bin", "wb") as fp:
    	MyStructure.to_stream({
			"age": 32,
			"name": "Test Person",
			"things": [2, 3, 5]
        }, fp)

This will write the values to the corresponding fields to the stream provided, which points to a file. Because of how the structure was defined, *age* will be written as a 4 byte integer in Little Endian order to the stream. Similarly, *name* will be written as a null terminated string to the stream... you get the picture now.

## Auxiliary Components

*cstruct2* provides some wrapper classes and helper functions in its library, most notably *cstruct2.cstruct2.SocketWrapper*. If you want to read or write a structure from/to a socket, you will need to construct a *SocketWrapper* on that socket first, before utilizing it with *cstruct2*. For example, if I open a socket as a client, and I wish to utilize it with *cstruct2*:

    from cstruct2.cstruct2 import cstruct2
    from cstruct2.cstruct2_utils import SocketWrapper
    
    @cstruct2
    class Handshake:
    	id: int = ("little", 4)
    	username: str = "pascal"
    	password: str = "pascal"
    
    @cstruct2
    class Response:
    	last_login: int = ("little", 8)
    	messages_len: int = ("little", 2)
    	messages: str = ["messages_len", "null"]
    
    sock = socket.socket()
    sock.connect(("192.168.0.3", 8080))
    sock_wrapper = SocketWrapper(sock)
    
    Handshake.to_stream({
    	"id": 321,
    	"username": "admin",
    	"password": "123"
    }, sock_wrapper)
    
    response: dict = Response.from_stream(sock_wrapper)
    for msg in response["messages"]:
    	print(msg)
    
    sock.close()

A variety of examples are provided in this repository, such as an example SOCKS5 proxy server created with *cstruct2* and a C program that writes packed binary structures to a file, which *cstruct2* will then read and parse.

## Miscellaneous

A list of changes to this library can be seen through the CHANGELOG	file in this repository. A list of things that need to get done can be seen through the TODO file that is also in this repository. The source code to this library is quite messy and inefficient at the moment as well, as a heads up. If you have any questions, complaints, or suggestions, feel free to make issues on this repository or email me at arner@usa.com.