Embedding with frozen code

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
baddourr
Posts: 1
Joined: Tue Oct 11, 2016 12:48 pm

Embedding with frozen code

Post by baddourr » Mon Oct 24, 2016 9:36 am

Hey all,
This is my first post here, I hope I am not repeating things and wasting others time ;)

Basically, I am working with Ericsson Research on another open source project named Calvin, and there we want to use uPy to execute pre-compiled (frozen) pieces of python code that some should be statically integrated with the uPy (precompiled), and others will be dynamically loaded during the execution. The goal is to have uPy as a submoule in the Calvin structure and use it as a static library (using the examples/embedded way), then call the necessary functionalities to execute/instantiate the needed functionalities/objects.

I managed to understand and execute each of the previous tasks individually, i.e.: compiling frozen models, loading the models, calling functions from c_main and building a library from uPy, but when I try to compile the library with inclusion of frozen code it doesn't compile. Basically, I could not manage to make the right order in the Makefile(s) to pre-generate the QSTR dependencies before compiling the _frozen_mpy.c file. with the following error message:
No rule to make target `/_frozen_mpy.c', needed by `build/genhdr/qstr.i.last

Can someone please advice what changes are needed to the Makefile (or to Makefile.upylib) to include the necessary frozen files.
On top of this, it would be highly preferable to keep the frozen files compiled AFTER the uPy library is created, would this be possible ?

Thanks in advance

Damien
Site Admin
Posts: 647
Joined: Mon Dec 09, 2013 5:02 pm

Re: Embedding with frozen code

Post by Damien » Thu Oct 27, 2016 5:34 am

Frozen bytecode was designed so it could be the last step in the compilation phase, and nothing earlier depends on it. So, yes, you should be able to build the uPy library and then do frozen bytecode at the end (and if any of the frozen scripts change you should only need to rebuild frozen_mpy.c/o).

Look at how stmhal/Makefile does it to get an understanding. Basically you don't want to have frozen_mpy.c in your SRC_C file list, otherwise it'll be scanned for qstrs (which it shouldn't).

If you compile the uPy library with -DMICROPY_QSTR_EXTRA_POOL=mp_qstr_frozen_const_pool and -DMICROPY_MODULE_FROZEN_MPY, then that's enough to support frozen bytecode. Once compiled like this there will be a few outstanding linker symbols (mp_qstr_frozen_const_pool, mp_frozen_mpy_names, mp_frozen_mpy_content) which will be provided by frozen_mpy.o.

To correctly build frozen_mpy.o you use mpy-cross to turn ..py files into .mpy files, and then tools/mpy-tool.py to combine those .mpy files into the frozen_mpy.c file. mpy-tool.py depends on the qstrdefs.preprocessed.h file from building the uPy library, in order to append new interned strings (qstrs) to the existing list.

Hope that helps!

cduran
Posts: 80
Joined: Thu Mar 17, 2016 4:52 pm

Re: Embedding with frozen code

Post by cduran » Tue Feb 28, 2017 10:22 pm

I'm using the minimal build, how would I go about loading the frozen module?

So far I have the following:

1) Compile my frozen py file into an mpy with mpy-cross
2) Used the make-frozen script on the mpy file
3) Included the C file into my project (I named it frozen-helpers, does it need some specific name?)
4) Defined the MICROPY_MODULE_FROZEN in the mpconfigport.h
.
.
n) I don't know what to do next...

Keep in mind that I'm not using the Makefile, I included the project into existing firmware I have in AtmelStudio.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Embedding with frozen code

Post by dhylands » Tue Feb 28, 2017 10:57 pm

In your main.c file you can call

Code: Select all

pyexec_frozen_module("main.py");
replacing main.py with the name of your frozen module. This is what I do on the teensy. I haven't tried with frozen bytecode, just frozen python code.

cduran
Posts: 80
Joined: Thu Mar 17, 2016 4:52 pm

Re: Embedding with frozen code

Post by cduran » Fri Mar 03, 2017 9:01 pm

dhylands wrote:In your main.c file you can call

Code: Select all

pyexec_frozen_module("main.py");
replacing main.py with the name of your frozen module. This is what I do on the teensy. I haven't tried with frozen bytecode, just frozen python code.
Thanks that worked well for source code, however it didn't work with byte code (mpy file).

A) Is there another function for that, I couldn't find one, or do I need to roll my own (which leads to part B of my question)?
B) Are mpy files the same as the output of the mp lexer/compiler or do I need to make changes that make-frozen.py doesn't?


I just realized (forgot about them because it's been a while since I built the mpy-cross) there are some #defines for mpy-cross, do I need to alter any of those?

This is what's currently in there, I need to compile my byte code to run on mpy running on an ARM:

Code: Select all

#define MICROPY_ALLOC_PATH_MAX      (PATH_MAX)
#define MICROPY_PERSISTENT_CODE_LOAD (0)
#define MICROPY_PERSISTENT_CODE_SAVE (1)

#define MICROPY_EMIT_X64            (0)
#define MICROPY_EMIT_X86            (0)
#define MICROPY_EMIT_THUMB          (0)
#define MICROPY_EMIT_INLINE_THUMB   (0)
#define MICROPY_EMIT_INLINE_THUMB_ARMV7M (0)
#define MICROPY_EMIT_INLINE_THUMB_FLOAT (0)
#define MICROPY_EMIT_ARM            (0)

#define MICROPY_DYNAMIC_COMPILER    (1)
#define MICROPY_COMP_CONST_FOLDING  (1)
#define MICROPY_COMP_MODULE_CONST   (1)
#define MICROPY_COMP_CONST          (1)
#define MICROPY_COMP_DOUBLE_TUPLE_ASSIGN (1)
#define MICROPY_COMP_TRIPLE_TUPLE_ASSIGN (1)

#define MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE (0)

#define MICROPY_ENABLE_RUNTIME      (0)
#define MICROPY_ENABLE_GC           (1)
#define MICROPY_STACK_CHECK         (1)
#define MICROPY_HELPER_LEXER_UNIX   (1)
#define MICROPY_LONGINT_IMPL        (MICROPY_LONGINT_IMPL_MPZ)
#define MICROPY_ENABLE_SOURCE_LINE  (1)
#define MICROPY_ENABLE_DOC_STRING   (0)
#define MICROPY_ERROR_REPORTING     (MICROPY_ERROR_REPORTING_DETAILED)
#define MICROPY_WARNINGS            (1)

#define MICROPY_FLOAT_IMPL          (MICROPY_FLOAT_IMPL_DOUBLE)
#define MICROPY_CPYTHON_COMPAT      (1)

#define MICROPY_PY_BUILTINS_STR_UNICODE (1)

// Define to 1 to use undertested inefficient GC helper implementation
// (if more efficient arch-specific one is not available).
#ifndef MICROPY_GCREGS_SETJMP
    #ifdef __mips__
        #define MICROPY_GCREGS_SETJMP (1)
    #else
        #define MICROPY_GCREGS_SETJMP (0)
    #endif
#endif

#define MICROPY_PY___FILE__         (0)
#define MICROPY_PY_ARRAY            (0)
#define MICROPY_PY_ATTRTUPLE        (0)
#define MICROPY_PY_COLLECTIONS      (0)
#define MICROPY_PY_MATH             (0)
#define MICROPY_PY_CMATH            (0)
#define MICROPY_PY_GC               (0)
#define MICROPY_PY_IO               (0)
#define MICROPY_PY_SYS              (0)

// type definitions for the specific machine

#ifdef __LP64__
typedef long mp_int_t; // must be pointer size
typedef unsigned long mp_uint_t; // must be pointer size
#else
// These are definitions for machines where sizeof(int) == sizeof(void*),
// regardless for actual size.
typedef int mp_int_t; // must be pointer size
typedef unsigned int mp_uint_t; // must be pointer size
#endif

#define BYTES_PER_WORD sizeof(mp_int_t)

// Cannot include <sys/types.h>, as it may lead to symbol name clashes
#if _FILE_OFFSET_BITS == 64 && !defined(__LP64__)
typedef long long mp_off_t;
#else
typedef long mp_off_t;
#endif

typedef void *machine_ptr_t; // must be of pointer size
typedef const void *machine_const_ptr_t; // must be of pointer size

#define MP_PLAT_PRINT_STRN(str, len) (void)0

#ifndef MP_NOINLINE
#define MP_NOINLINE __attribute__((noinline))
#endif

// We need to provide a declaration/definition of alloca()
#ifdef __FreeBSD__
#include <stdlib.h>
#else
#include <alloca.h>
#endif

#include <stdint.h>

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Embedding with frozen code

Post by dhylands » Sat Mar 04, 2017 3:58 am

Looking at the code you also need MICROPY_MODULE_FROZEN_MPY defined.

I tested it on stmhal and using pyexec_exec_frozen("main.py") works with bytecode, so it looks like its just a case of enabling the correct configuration options.

cduran
Posts: 80
Joined: Thu Mar 17, 2016 4:52 pm

Re: Embedding with frozen code

Post by cduran » Mon Mar 06, 2017 2:07 pm

Do you mean the code I posted? (Which is the cross compilers code) or the micopython build?

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Embedding with frozen code

Post by dhylands » Mon Mar 06, 2017 7:24 pm

I meant the micropython code you're running. If you look at pyexec_frozen_module: https://github.com/micropython/micropyt ... #L518-L539
then you need MICROPY_FROZEN_MODULE and MICROPY_MODULE_FROZEN_MPY defined in order to have .mpy files executed. Although IIRC you need to specify the name of the .py file

Post Reply