[MIPS] Garbage collection crash during Unix port running on MIPS

Discussion and questions about boards that can run MicroPython but don't have a dedicated forum.
Target audience: Everyone interested in running MicroPython on other hardware.
agonnen
Posts: 26
Joined: Sat Oct 13, 2018 7:52 pm

Re: [MIPS] Garbage collection crash during Unix port running on MIPS

Post by agonnen » Mon Jun 07, 2021 9:15 pm

I'm seeing some problem with FFI on x64 Ubuntu.
I'm trying to implement "machine.Timer" based on timer.py from micropython-lib here: https://github.com/lvgl/lv_binding_micr ... v_timer.py

It works up to some point and then crashes:

Code: Select all

>>> from lv_timer import Timer
>>> def f(t):
...     pass
...     
...     
... 
>>> t = Timer(1)
>>> t.init(period=1, callback=f)
>>> 
>>> 
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff79b7ae1 in ffi_closure_unix64_inner () from /usr/lib/x86_64-linux-gnu/libffi.so.6
(gdb) bt
#0  0x00007ffff79b7ae1 in ffi_closure_unix64_inner () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#1  0x00007ffff79b7fa8 in ffi_closure_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#2  <signal handler called>
#3  0x00007ffff7bca4fe in __read_nocancel () at ../sysdeps/unix/syscall-template.S:84
#4  0x000000000064823a in mp_hal_stdin_rx_chr () at unix_mphal.c:170
#5  0x00000000007672d1 in readline (line=0x7fffffffd5d0, prompt=0x82324d ">>> ") at ../../lib/mp-readline/readline.c:512
#6  0x0000000000646d4e in do_repl () at main.c:194
#7  0x0000000000647eb9 in main_ (argc=1, argv=0x7fffffffd868) at main.c:673
#8  0x00000000006473cf in main (argc=1, argv=0x7fffffffd868) at main.c:436

stijn
Posts: 642
Joined: Thu Apr 24, 2014 9:13 am

Re: [MIPS] Garbage collection crash during Unix port running on MIPS

Post by stijn » Tue Jun 08, 2021 8:07 am

If this is the same problem, not unlikely, you could post the simplest code to reproduce it in https://github.com/micropython/micropython/issues/7273. And try the workaround.

agonnen
Posts: 26
Joined: Sat Oct 13, 2018 7:52 pm

Re: [MIPS] Garbage collection crash during Unix port running on MIPS

Post by agonnen » Tue Jun 08, 2021 8:18 am

stijn wrote:
Tue Jun 08, 2021 8:07 am
If this is the same problem, not unlikely, you could post the simplest code to reproduce it in https://github.com/micropython/micropython/issues/7273. And try the workaround.
The code above is a simple reproduction.
The workaround did not help.

stijn
Posts: 642
Joined: Thu Apr 24, 2014 9:13 am

Re: [MIPS] Garbage collection crash during Unix port running on MIPS

Post by stijn » Tue Jun 08, 2021 9:15 am

In that case it might be an unrelated problem. Actually I didn't look at the code carefully before, but it indeed looks like a different problem because there's no bound method here, and it would be strange that the GC would to sweep the function f. Does the version from micropython-lib have the same problem? Does this segfault happen on the first call of the callback already (if so then probably something in your code is not ok because callbacks in general do work)?

agonnen
Posts: 26
Joined: Sat Oct 13, 2018 7:52 pm

Re: [MIPS] Garbage collection crash during Unix port running on MIPS

Post by agonnen » Tue Jun 08, 2021 8:08 pm

stijn wrote:
Tue Jun 08, 2021 9:15 am
In that case it might be an unrelated problem. Actually I didn't look at the code carefully before, but it indeed looks like a different problem because there's no bound method here, and it would be strange that the GC would to sweep the function f. Does the version from micropython-lib have the same problem? Does this segfault happen on the first call of the callback already (if so then probably something in your code is not ok because callbacks in general do work)?
Well, the problem doesn't happen on the first call of the callback and not in many other calls that follow.
It works for a while and then crashes.
Furthermore, it doesn't crash if I disable gc.

In fact, if I disable gc, run the example and then call "gc.collect()", Micropython hangs in a lock and I can clearly see that it tries to allocate gc memory (calls "m_new_obj_var_maybe" inside "mp_obj_t fun_bc_call").
In this case it looks like the callback was called in the middle of "gc_sweep", so that might explain the deadlock.

Here is the stack trace:

Code: Select all

__lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135	../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff7bc3dbd in __GI___pthread_mutex_lock (mutex=0xabbe10 <mp_state_ctx+2544>) at ../nptl/pthread_mutex_lock.c:80
#2  0x0000000000648a64 in mp_thread_mutex_lock (mutex=0xabbe10 <mp_state_ctx+2544>, wait=1) at mpthreadport.c:292
#3  0x000000000040544a in gc_alloc (n_bytes=<optimized out>, alloc_flags=0) at ../../py/gc.c:448
#4  0x0000000000404b3b in m_malloc_maybe (num_bytes=168) at ../../py/malloc.c:100
#5  0x0000000000505423 in fun_bc_call (self_in=0x7ffff13a5560, n_args=2, n_kw=0, args=0x7fffffffc870) at ../../py/objfun.c:268
#6  0x00000000004a51ed in mp_call_function_n_kw (fun_in=0x7ffff13a5560, n_args=2, n_kw=0, args=0x7fffffffc870) at ../../py/runtime.c:652
#7  0x00000000004f1ea1 in mp_call_method_self_n_kw (meth=0x7ffff13a5560, self=0x7ffff13a60e0, n_args=1, n_kw=0, args=0x7fffffffc990) at ../../py/objboundmeth.c:70
#8  0x00000000004f1f19 in bound_meth_call (self_in=0x7ffff13a6b60, n_args=1, n_kw=0, args=0x7fffffffc990) at ../../py/objboundmeth.c:83
#9  0x00000000004a51ed in mp_call_function_n_kw (fun_in=0x7ffff13a6b60, n_args=1, n_kw=0, args=0x7fffffffc990) at ../../py/runtime.c:652
#10 0x0000000000743978 in call_py_func (cif=0x7ffff13a6fe8, ret=0x7fffffffcba0, args=0x7fffffffca10, func=0x7ffff13a6b60) at modffi.c:252
#11 0x00007ffff79b7c2f in ffi_closure_unix64_inner () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#12 0x00007ffff79b7fa8 in ffi_closure_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#13 <signal handler called>
#14 0x00000000004051ee in gc_sweep () at ../../py/gc.c:306
#15 gc_collect_end () at ../../py/gc.c:361
#16 0x0000000000647fc4 in gc_collect () at gccollect.c:47
#17 0x00000000005b90ef in py_gc_collect () at ../../py/modgc.c:35
#18 0x0000000000504d43 in fun_builtin_0_call (self_in=0x7e2a70 <gc_collect_obj>, n_args=0, n_kw=0, args=0x7fffffffd3d0) at ../../py/objfun.c:56
#19 0x00000000004a51ed in mp_call_function_n_kw (fun_in=0x7e2a70 <gc_collect_obj>, n_args=0, n_kw=0, args=0x7fffffffd3d0) at ../../py/runtime.c:652
#20 0x00000000004a6f56 in mp_call_method_n_kw (n_args=0, n_kw=0, args=0x7fffffffd3c0) at ../../py/runtime.c:668
#21 0x00000000005cb623 in mp_execute_bytecode (code_state=0x7fffffffd380, inject_exc=<optimized out>) at ../../py/vm.c:1090
#22 0x00000000005054e9 in fun_bc_call (self_in=0x7ffff13a70a0, n_args=0, n_kw=0, args=0x0) at ../../py/objfun.c:288
#23 0x00000000004a51ed in mp_call_function_n_kw (fun_in=0x7ffff13a70a0, n_args=0, n_kw=0, args=0x0) at ../../py/runtime.c:652
#24 0x00000000004a5106 in mp_call_function_0 (fun=0x7ffff13a70a0) at ../../py/runtime.c:626
#25 0x0000000000646ca4 in execute_from_lexer (source_kind=2, source=0x7fffffffd5d0, input_kind=MP_PARSE_SINGLE_INPUT, is_repl=true) at main.c:145
#26 0x0000000000646ec9 in do_repl () at main.c:255
#27 0x0000000000647eb9 in main_ (argc=1, argv=0x7fffffffd868) at main.c:673
#28 0x00000000006473cf in main (argc=1, argv=0x7fffffffd868) at main.c:436

agonnen
Posts: 26
Joined: Sat Oct 13, 2018 7:52 pm

Re: [MIPS] Garbage collection crash during Unix port running on MIPS

Post by agonnen » Tue Jun 08, 2021 9:54 pm

Update: the problem I was seeing on Linux was really unrelated to the problem discussed on this thread.
This also explains why the workaround suggested here didn't have any effect.

My problem was root caused to memory allocations during signal handler (or interrupt handler) that are triggered spontaneously by Micropython.

Related issues:
- https://github.com/micropython/micropython/issues/7372
- https://github.com/micropython/micropython/issues/7373

Sorry for the distraction.

stijn
Posts: 642
Joined: Thu Apr 24, 2014 9:13 am

Re: [MIPS] Garbage collection crash during Unix port running on MIPS

Post by stijn » Wed Jun 09, 2021 9:16 am

No problem :)

Post Reply