[SOLVED] mutex deadlock during garbage collection

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
ThomasK
Posts: 5
Joined: Wed Nov 22, 2017 1:56 pm

[SOLVED] mutex deadlock during garbage collection

Post by ThomasK » Wed Feb 21, 2018 9:02 am

Hi,

I'm currently facing a fairly interesting problem with the garbage collection when running the micropython kernel inside of an RTOS.
as micropython runs inside of an RTOS, I'm making use of GC_ENTER and GC_EXIT to lock & unlock mutexes.

I'm wondering now if this is due to something I did in my port, or if this is an issue that exists within the standard micropython builds too?
Has anyone else seen this behavior, or am I missing something / did I make a mistake when implementing an alternative filesystem?

I started from MicroPython 1.9.3

Scenario:

Code: Select all

call gc_alloc()
  GC_ENTER()
  ...
  GC_EXIT()
  call gc_collect()
    call gc_collect_start()
      GC_ENTER()
      call gc_collect_end()
        call gc_sweep()
          call "__del__" method of object if it exists (in this example an old file pointer)
            file_close() throws a "file not open" OS Exception
              OS Exception calls gc_alloc()
                gc_alloc() calls GC_ENTER() 
Result: The thread waits for the mutex to be released by itself.
Last edited by ThomasK on Mon Feb 26, 2018 8:11 am, edited 1 time in total.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: mutex deadlock during garbage collection

Post by jickster » Wed Feb 21, 2018 7:10 pm

ThomasK wrote:
Wed Feb 21, 2018 9:02 am
Hi,

I'm currently facing a fairly interesting problem with the garbage collection when running the micropython kernel inside of an RTOS.
as micropython runs inside of an RTOS, I'm making use of GC_ENTER and GC_EXIT to lock & unlock mutexes.

I'm wondering now if this is due to something I did in my port, or if this is an issue that exists within the standard micropython builds too?
Has anyone else seen this behavior, or am I missing something / did I make a mistake when implementing an alternative filesystem?

I started from MicroPython 1.9.3

Scenario:

Code: Select all

call gc_alloc()
  GC_ENTER()
  ...
  GC_EXIT()
  call gc_collect()
    call gc_collect_start()
      GC_ENTER()
      call gc_collect_end()
        call gc_sweep()
          call "__del__" method of object if it exists (in this example an old file pointer)
            file_close() throws a "file not open" OS Exception
              OS Exception calls gc_alloc()
                gc_alloc() calls GC_ENTER() 
Result: The thread waits for the mutex to be released by itself.
Either you're not supposed to throw an exception while you're cleaning up (i.e. in __del__) OR
it's a bug in uPy.

m_malloc_maybe appears as if it's supposed to be a conditional allocation but its implementation does not actually provide that feature; it just calls malloc and does a debug print.

Code: Select all

void *m_malloc_maybe(size_t num_bytes) {
    void *ptr = malloc(num_bytes);
#if MICROPY_MEM_STATS
    MP_STATE_MEM(total_bytes_allocated) += num_bytes;
    MP_STATE_MEM(current_bytes_allocated) += num_bytes;
    UPDATE_PEAK();
#endif
    DEBUG_printf("malloc %d : %p\n", num_bytes, ptr);
    return ptr;
}
https://github.com/micropython/micropython/issues/3627

Damien
Site Admin
Posts: 605
Joined: Mon Dec 09, 2013 5:02 pm

Re: mutex deadlock during garbage collection

Post by Damien » Fri Feb 23, 2018 7:08 am

The GC_ENTER/GC_EXIT macros are really only intended to be used by the GC for the case that multithreading is enabled and the GIL is disabled. It's probably going to be safer in general to enable the GIL so that things like list or dict update appear atomic to MicroPython scripts.

Also, according to standard CPython behaviour, a file should be allowed to be closed multiple times without error. That really helps the case at hand, where the user may have closed the file explicitly but then it also has the finaliser called when the memory for the file is cleaned up.

But apart from those two points, it does look like there is a bug in uPy that you've exposed: when doing a garbage collection the memory allocator is locked and so it shouldn't try to allocate, but in this case it does.

ThomasK
Posts: 5
Joined: Wed Nov 22, 2017 1:56 pm

Re: mutex deadlock during garbage collection

Post by ThomasK » Fri Feb 23, 2018 9:43 am

Great, thank you for the feedback, jickster & Damien!

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: mutex deadlock during garbage collection

Post by jickster » Fri Feb 23, 2018 9:39 pm

ThomasK wrote:
Fri Feb 23, 2018 9:43 am
Great, thank you for the feedback, jickster & Damien!
Please update your question title by prepending [SOLVED] to make the forum easier to read

ThomasK
Posts: 5
Joined: Wed Nov 22, 2017 1:56 pm

Re: [SOLVED] mutex deadlock during garbage collection

Post by ThomasK » Mon Feb 26, 2018 8:12 am

Done, as the part that directly applied to my question was resolved,

although I don't quite agree that it's solved when Damien comes to the conclusion that "there indeed appears to be a bug".

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: [SOLVED] mutex deadlock during garbage collection

Post by jickster » Mon Feb 26, 2018 6:01 pm

ThomasK wrote:
Mon Feb 26, 2018 8:12 am
Done, as the part that directly applied to my question was resolved,

although I don't quite agree that it's solved when Damien comes to the conclusion that "there indeed appears to be a bug".
Personally, I'm not clear what the GIL is and how it works and its relationship to GC_ENTER.

User avatar
pythoncoder
Posts: 3404
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: [SOLVED] mutex deadlock during garbage collection

Post by pythoncoder » Tue Feb 27, 2018 9:13 am

GIL stands for Global Interpreter Lock and is a rather contentious aspect of Python. Most implementations have a GIL. Google is your friend here ;)
Peter Hinch

Post Reply