root pointers and how to make them more flexible?

[braiins]

Hi,

while integrating/wrapping FreeRTOS driver code, I have come across a very common pattern regarding memory allocation and would like to know general opinion of the core developers on this topic.
Essentially - our driver framework is very symetric in terms of API - each driver usually provides methods to create new instance (e.g. uart__new()), initialize an already preallocated instance (e.g. uart__init()), and some actually workload methods like uart__read()/uart__write()
This API is very easy to map/wrap to have such driver available in python.

The question is: is there any standard way of locking the instance in memory except for using root pointers? I find the root pointers a bit inflexible. Without the root pointers and relying only on the user python code to keep the reference to the driver is a bit questionable. Since the finaliser would have to trigger freeing of the related RTOS memory resources. This if feasible but not always desired. Generally, we run the RTOS with a minimalistic memory allocation scheme that doesn't allow freeing memory (to prevent fragmentation and guarantee hard-realtime features).

I was actually thinking of having some hashing mechanism that would be able to uniquely map a device ID onto a root pointer or anything similar like that.

Thank you for any comments.

Best regards,

Jan

[dhylands]

The way the gc works is that anything that is "referenced" (i.e. has a pointer to it) is considered to be in use.

So the gc looks through the stack (since registers and local variables on the stack can contain pointers to nwely allocated objects) and as a performance optimization, rather than looking through all of the data segment, it only looks through the root pointers.

Each port is allowed to decide how it wants to manage the root pointers. This is reflected in the fact that each port needs to provide the gc_collect function, and this function is the function that scans the memory containing the root pointers.

For stmhal, the root pointers are defined here:
https://github.com/micropython/micropyt ... #L160-L189 and these are stored inside the mp_state_ctx variable (MICROPY_PORT_ROOT_POINTERS is contained in https://github.com/micropython/micropyt ... ate.h#L137

Now gc_collect_start already scans the mp_state_ctx variable:
https://github.com/micropython/micropyt ... #L281-L282

and a port can add arbitrary additional "pools" or root pointers in the port definition of gc_collect (here's the stmhal version):
https://github.com/micropython/micropyt ... lect.c#L51

So, if you wanted to make a root-pointer dict that was included in the mp_state_ctx and have your drivers just add entries to that dict, that would work fine. stmhal chose to list the pointers explicitly rather than use a dict, but each port can choose to implement this however they like.

Hopefully, that adds some clarity rather than confusion

[braiins]

The way the gc works is that anything that is "referenced" (i.e. has a pointer to it) is considered to be in use.

So the gc looks through the stack (since registers and local variables on the stack can contain pointers to nwely allocated objects) and as a performance optimization, rather than looking through all of the data segment, it only looks through the root pointers.

Each port is allowed to decide how it wants to manage the root pointers. This is reflected in the fact that each port needs to provide the gc_collect function, and this function is the function that scans the memory containing the root pointers.

For stmhal, the root pointers are defined here:
https://github.com/micropython/micropyt ... #L160-L189 and these are stored inside the mp_state_ctx variable (MICROPY_PORT_ROOT_POINTERS is contained in https://github.com/micropython/micropyt ... ate.h#L137

Now gc_collect_start already scans the mp_state_ctx variable:
https://github.com/micropython/micropyt ... #L281-L282

and a port can add arbitrary additional "pools" or root pointers in the port definition of gc_collect (here's the stmhal version):
https://github.com/micropython/micropyt ... lect.c#L51

So, if you wanted to make a root-pointer dict that was included in the mp_state_ctx and have your drivers just add entries to that dict, that would work fine. stmhal chose to list the pointers explicitly rather than use a dict, but each port can choose to implement this however they like.

Hopefully, that adds some clarity rather than confusion

No confusion, the description matches of what I have seen and reverse-engineered from the sources.

Using a dict perfectly fits my intention with hashing. Even though a linear scan for an empty slot among root pointers would also do the job and the penalty would be rather small considering the projected number of runtime driver instances.

The reason why I posted about this topic was not to miss any best practice recommended by the development guidelines.

Thank you Dave for great references to the sources, too!

Cheers,

Jan

[dhylands]

The other thing to keep in mind is that you really only need root pointers for objects that are only referenced by C code.

If you're returning a uart object and it contains pointers to stuff, that portion will be taken care of automatically (since the python code/data will have a pointer to the uart object), and there really isn't a need for a root pointer to the uart object (unless you want it to live longer than the users pythoncode uses it).