[SOLVED]garbage collector + stack of startup task

[BramPeeters]

If you call gc_collect(), the correct stack is correctly taken into account so your entire first example is incorrect.

Ok some progress, now we are on the same page talking about the entire stack and the objects currently referenced from it and not some object that at some point in the past was on that stack.

Now next step : explain to me how a [1] stack that is not in the GC heap can possibly [2] be taken into account ?

There are 2 parts in that sentence , which one(s) are you contesting so we can further elaborate ...
[1] That it is not in the GC heap ? -> In that case do you understand how eg linker files work ? Where the stack comes from that 'main' is executed in ? It exists before gc_init is called so there is no way it can be in the gc heap.
[2] That is not taken into account ? It is not because of the VERIFY_PTR macro which as you already said yourself will not further look at anything that is not in the heap.

I keep repeating the same things over and over, please take some time to understand what i am saying here. I have a problem that I can see when stepping in the debugger, that I understand from the code, and that I know how to fix in the code. At this point I am 99.9% sure that I am *right*.

Remark:
And I don't know what you mean by 'correct' stack. To elaborate:
- There is no correct stack, all the stacks need to be taken into account (which is done in mp_thread_gc_others )
- There is however a correct part of the stack, which would be the part between the current stack pointer and the top of the stack. As I already explained, this is NOT taken into account correctly. And attempt is made to do it for the stack calling gc_collect, but it is then 'redone' when the mp_thread_gc_others function is called where the entire stack will be checked anyway via gc_collect_root((void**)&th, 1);

[jickster]

If you call gc_collect(), the correct stack is correctly taken into account so your entire first example is incorrect.

Ok some progress, now we are on the same page talking about the entire stack and the objects currently referenced from it and not some object that at some point in the past was on that stack.

Now next step : explain to me how a [1] stack that is not in the GC heap can possibly [2] be taken into account ?

There are 2 parts in that sentence , which one(s) are you contesting so we can further elaborate ...
[1] That it is not in the GC heap ? -> In that case do you understand how eg linker files work ? Where the stack comes from that 'main' is executed in ? It exists before gc_init is called so there is no way it can be in the gc heap.
[2] That is not taken into account ? It is not because of the VERIFY_PTR macro which as you already said yourself will not further look at anything that is not in the heap.

I keep repeating the same things over and over, please take some time to understand what i am saying here. I have a problem that I can see when stepping in the debugger, that I understand from the code, and that I know how to fix in the code. At this point I am 99.9% sure that I am *right*.

Remark:
And I don't know what you mean by 'correct' stack. To elaborate:
- There is no correct stack, all the stacks need to be taken into account (which is done in mp_thread_gc_others )
- There is however a correct part of the stack, which would be the part between the current stack pointer and the top of the stack. As I already explained, this is NOT taken into account correctly. And attempt is made to do it for the stack calling gc_collect, but it is then 'redone' when the mp_thread_gc_others function is called where the entire stack will be checked anyway via gc_collect_root((void**)&th, 1);

Let's go to a chat interface. Then I'll paste our conversation in this thread so everyone can benefit.

https://chat.stackoverflow.com/rooms/17 ... thon-stack

[BramPeeters]

No can do, after checking and finding some old stackexchange/stackoverflow accounts, it seems they do not have the required 20 reputation to even be able to say something .
Besides, what is the point of spreading this information over 2 places and I think this forum is where ppl will start looking in the first place because most mp knowledge is concentrated here ...?

EDIT: i see you changed the conversation to public everyone can talk, but it does not help with the 20 reputation limit...

[jickster]

If you call gc_collect(), the correct stack is correctly taken into account so your entire first example is incorrect.

Which platform are you using? ESP32?

[BramPeeters]

Proprietary hardware, the code i am using is a combination of the stm32 (because the cpu is an stm32) and cc3200 (because i use it on top of freertos) ports.

So my 'initial task' was originally a freertos task as in cc3200 (static stack so not on heap).
But if i am not mistaken in the stm32 port the task that runs main will have the same problem because I expect it is in the stack memory as defined by the linker file, so not on the GC heap.

Other remark: i see that there is a 1.9.4, i am still on 1.9.3, if there have been changes in the meantime wrt this i am talking about outdated code

[jickster]

PLEASE pose on question at a time for me or at least number them and separate them so I can reference them easier.

If you call gc_collect(), the correct stack is correctly taken into account so your entire first example is incorrect.

Now next step : explain to me how a [1] stack that is not in the GC heap can possibly [2] be taken into account ?

For each CPU, there is a CPU-specific function of the garbage collection that retrieves the registers and places them on the stack so that they are treated as a source of root-pointers.

PITFALL: If you use hardware floating point . . . I'm not sure if you need to include those registers as source of root-pointers.

Since you're using STM32 CPU, I'll be referencing that port

https://github.com/micropython/micropyt ... lect.c#L60

In gc_collect(), there is an assembly function that collects the contents of the registers and places them on the stack:

Code: Select all

// get the registers and the sp
    uintptr_t regs[10];
    uintptr_t sp = gc_helper_get_regs_and_sp(regs);

The next lines are the ones that do the "magic" of treating the stack as a source of root pointers

Since you've placed the registers onto the stack, via the lines before, the registers ARE taken into account as a source of root-pointers due to the code below:

Code: Select all

#if MICROPY_PY_THREAD
    gc_collect_root((void**)sp, ((uint32_t)MP_STATE_THREAD(stack_top) - sp) / sizeof(uint32_t));
    #else
    gc_collect_root((void**)sp, ((uint32_t)&_ram_end - sp) / sizeof(uint32_t));
    #endif

You have threading enabled so it would call the first one.
Two scenarios:
* Call from thread: the thread's own stack is taken care of by the above code block because "stack_top" is set during the thread switching (well it should be).

* Call from C-code before you call: this was your "gotcha" for me but it's easy.
In your code, as part of initializing micropython, you have to call

Code: Select all

void mp_stack_ctrl_init(void) {
    volatile int stack_dummy;
    MP_STATE_THREAD(stack_top) = (char*)&stack_dummy;
}

FYI, I don't agree with the #else in above code.
I do not have threads but I still call mp_stack_ctrl_init() and my gc_collect() always calls

Code: Select all

gc_collect_root_sp((void**)sp, ((uint32_t)MP_STATE_THREAD(stack_top) - sp) / sizeof(uint32_t));

[BramPeeters]

I know and deeply understand all the stuff you just said, I had to port the gc_helper_get_sp and gc_helper_get_regs_and_sp functions myself because my IAR compiler does not support the syntax micropython is using. I have external non python created threads calling callbacks into python land which require that mp_thread_set_state() is configured for these threads to set the stack top otherwise micropython thinks there is a stack overflow (or something, i don't remember).

But it is besides the point. Everything you described happens BEFORE you reach the numbered the questions which are the crux of the problem, so start looking at what happens after that instead of considering that the endpoint.
Yes, gc_collect_root((void**)sp, ((uint32_t)MP_STATE_THREAD(stack_top) - sp) / sizeof(uint32_t)); is called.

But if sp is not in the heap memory area it WILL be ignored so it does not matter that it is passed via the gc_collect_root function( either in this line of codel or in the mp_thread_gc_others function).

Maybe if I use some fictional numbers it will make it more clearer to you (i use small unrealistic numbers as an example here but i dont want to type 32 addresses all the time)?

Suppose you define in the linker file that the stack for the startup task executing main is between address 20 and 40 ( i am talking about a port without freertos in that case). Suppose the GC heap memory area is configured to go from addres 100 to 200.

So sp will always be located between 20 and 40 (not taking into account stack overflows), eg lets say it is 32 at the moment you call collect so you pass gc_collect_root (32, 3 ). Now you claim everything is fine because we successfully passed to stack to gc_collect_root.
Well it is not, gc_collect_root will then compare that 32 to the heap area which is between 100 and 200, and not even take a peep at the active stack area 32 to 40 because it is not situated in that 100-200 area, the GC heap.
Same happens if the stack of the main task is passed in mp_thread_gc_others.
So nothing referenced from that stack is marked and it will be 'collected'.

So i will ask you the same 2 numbered questions again in the context of the above example, ignore all the rest i have said, it is just context. Please start by answering these with yes or no in your response so i know what needs to be further explained.

[1] Do you understand/agree sp is in the 20 to 40 area which is outside of the heap 100-200.
[2] Do you understand/agree that therefore sp (or the stack) will be ignored if passed to the gc_collect_root function via whatever mechanism?

[jickster]

[1] Do you understand/agree sp is in the 20 to 40 area which is outside of the heap 100-200.
[2] Do you understand/agree that therefore sp (or the stack) will be ignored if passed to the gc_collect_root function via whatever mechanism?

[1] Yes
[2] No I do not agree.

Whatever region of memory you pass in to gc_collect_root() is treated as a region that contains root pointers. That's how that function is written.

Code: Select all

#if MICROPY_PY_THREAD
    gc_collect_root((void**)sp, ((uint32_t)MP_STATE_THREAD(stack_top) - sp) / sizeof(uint32_t));
    #else
    gc_collect_root((void**)sp, ((uint32_t)&_ram_end - sp) / sizeof(uint32_t));
    #endif

It doesn't make sense to say "if you pass in addresses 50-60 to gc_collect_root() then address 50 will be ignored because it's not in the heap"
If you pass in addresses 50-60 to gc_collect_root() you are DEFINING that region to contain root pointers. It doesn't have to be in the heap.

Assume I have a global variable.

Code: Select all

static int * a;
...
a = malloc(4);

You are mistaking the value of a with the value of &a.
It doesn't matter that &a is not in the heap; if you tell the gc a is a root pointer by calling
gc_collect_root((voiid**)&a, 1) then by definition it is and the memory allocated to it will not be collected.

Code: Select all

gc_collect_root((void**)sp, ((uint32_t)MP_STATE_THREAD(stack_top) - sp) / sizeof(uint32_t));

This is NOT passing the ADDRESS of the stack pointer but the VALUE of the stack pointer.

The stack pointer is accessible via a memory-mapped address like 20 to 40 but the assembly function gc_helper_get_regs_and_sp() returns the value of the stack pointer.

[BramPeeters]

Ok, you are absolutely right, thanks, I apologize !

The reason I thought it would be rejected is because i was (falsely) under the impression that in gc_collect_root the addresses 32, 36, 40 would be passed to VERIFY_MARK_AND_PUSH() and hence rejected.

But that is not what happens, the addresses are dereferenced first to get their contents (*32), (*36) and (*40) via the

Code: Select all

*ptr = ptrs[i] 

line before they are passed. Arghh, shame on me!

Code: Select all

void gc_collect_root(void **ptrs, size_t len) {
    for (size_t i = 0; i < len; i++) {
        void *ptr = ptrs[i];
        VERIFY_MARK_AND_PUSH(ptr);
        gc_drain_stack();
    }
}

So i must running into a different problem when my stack is not located in heap memory, i will have to look into that tomorrow cos it is getting pretty late here and my brain is getting fuzzy and obviously it is producing gibbirish even when it is awake.

Anway if you want PM me your paypal account email address and the average price of a beer in your country so I can buy you some to compensate for putting up with me (or if you are in the EU, your account number) !

[jickster]

No need to pay me. I enjoy this.

If you want, donate to Micropython project (once they get such a button).


Sent from my iPhone using Tapatalk Pro