[Windows] New frozen module not present in executable

[Hitman1O1]

Following up viewtopic.php?f=12&t=8224 I am trying to add a new module to the executable. I added the file "ports/windows/manifest.py" with the content:

Code: Select all

freeze('$(MPY_DIR)/ports/windows/modules/python', 'mypackage.py')

And of course I created "ports/windows/modules/python/mypackage.py" with just an empty class.

Then in "ports/windows/Makefile" I added the line: (right after "PROG=...")

Code: Select all

# File containing description of content to be frozen into firmware.
FROZEN_MANIFEST ?= manifest.py

Building now gave an error about "MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE", so I modified "ports/windows/mpconfigport.h" to read:

Code: Select all

#define MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE (0)

Now it compiles fine. However, if I run the new micropython.exe and run "import mypackage" I get a 'Package not found' error.

I am compiling from the Linux subsystem on Windows 10 with the command "make CROSS_COMPILE=i686-w64-mingw32-".

Any ideas what I'm doing wrong?

[Hitman1O1]

If I try the same on the unix port (from inside the Linux subsystem) the new package does work. This port already as a manifest.py, so I just added a line pointing to "ports/windows/modules/python/mypackage.py". After compiling and running I can run "import mypackage" with no problems.

[jimmo]

Any ideas what I'm doing wrong?

I'm not terribly familiar with the Windows port, but it's not clear to me whether the Windows port supports frozen modules (either the old way or the new way with manifest.py). The way to check would be to deliberately add a syntax error to manifest.py and see if the build complains.

[Hitman1O1]

So the Windows port as it is in the repository does not support the frozen modules as there is no manifest.py. I added it and updated the Makefile with the frozen modules line.

In my version if I introduce a syntax error in manifest.py the build fails:

Code: Select all

Traceback (most recent call last):
  File "../../tools/makemanifest.py", line 344, in <module>
    main()
  File "../../tools/makemanifest.py", line 254, in main
    include(input_manifest)
  File "../../tools/makemanifest.py", line 56, in include
    exec(f.read())
  File "<string>", line 1
    <dnex9r
    ^
SyntaxError: invalid syntax
../../py/mkrules.mk:126: recipe for target 'build/frozen_content.c' failed
make: *** [build/frozen_content.c] Error 1

Also the complete build log (without intentional syntax errors):

Code: Select all

Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
mkdir -p build/genhdr
GEN build/genhdr/mpversion.h
GEN build/genhdr/moduledefs.h
GEN build/genhdr/qstr.i.last
GEN build/genhdr/qstr.split
GEN build/genhdr/qstrdefs.collected.h
QSTR updated
GEN build/genhdr/qstrdefs.generated.h
mkdir -p build/build/
mkdir -p build/extmod/
mkdir -p build/lib/embed/
mkdir -p build/lib/mp-readline/
mkdir -p build/lib/utils/
mkdir -p build/ports/unix/
mkdir -p build/py/
CC ../../py/mpstate.c
CC ../../py/nlr.c
CC ../../py/nlrx86.c
CC ../../py/nlrx64.c
CC ../../py/nlrthumb.c
CC ../../py/nlrpowerpc.c
CC ../../py/nlrxtensa.c
CC ../../py/nlrsetjmp.c
CC ../../py/malloc.c
CC ../../py/gc.c
CC ../../py/pystack.c
CC ../../py/qstr.c
CC ../../py/vstr.c
CC ../../py/mpprint.c
CC ../../py/unicode.c
CC ../../py/mpz.c
CC ../../py/reader.c
CC ../../py/lexer.c
CC ../../py/parse.c
CC ../../py/scope.c
CC ../../py/compile.c
CC ../../py/emitcommon.c
CC ../../py/emitbc.c
CC ../../py/asmbase.c
CC ../../py/asmx64.c
CC ../../py/emitnx64.c
CC ../../py/asmx86.c
CC ../../py/emitnx86.c
CC ../../py/asmthumb.c
CC ../../py/emitnthumb.c
CC ../../py/emitinlinethumb.c
CC ../../py/asmarm.c
CC ../../py/emitnarm.c
CC ../../py/asmxtensa.c
CC ../../py/emitnxtensa.c
CC ../../py/emitinlinextensa.c
CC ../../py/emitnxtensawin.c
CC ../../py/formatfloat.c
CC ../../py/parsenumbase.c
CC ../../py/parsenum.c
CC ../../py/emitglue.c
CC ../../py/persistentcode.c
CC ../../py/runtime.c
CC ../../py/runtime_utils.c
CC ../../py/scheduler.c
CC ../../py/nativeglue.c
CC ../../py/pairheap.c
CC ../../py/ringbuf.c
CC ../../py/stackctrl.c
CC ../../py/argcheck.c
CC ../../py/warning.c
CC ../../py/profile.c
CC ../../py/map.c
CC ../../py/obj.c
CC ../../py/objarray.c
CC ../../py/objattrtuple.c
CC ../../py/objbool.c
CC ../../py/objboundmeth.c
CC ../../py/objcell.c
CC ../../py/objclosure.c
CC ../../py/objcomplex.c
CC ../../py/objdeque.c
CC ../../py/objdict.c
CC ../../py/objenumerate.c
CC ../../py/objexcept.c
CC ../../py/objfilter.c
CC ../../py/objfloat.c
CC ../../py/objfun.c
CC ../../py/objgenerator.c
CC ../../py/objgetitemiter.c
CC ../../py/objint.c
CC ../../py/objint_longlong.c
CC ../../py/objint_mpz.c
CC ../../py/objlist.c
CC ../../py/objmap.c
CC ../../py/objmodule.c
CC ../../py/objobject.c
CC ../../py/objpolyiter.c
CC ../../py/objproperty.c
CC ../../py/objnone.c
CC ../../py/objnamedtuple.c
CC ../../py/objrange.c
CC ../../py/objreversed.c
CC ../../py/objset.c
CC ../../py/objsingleton.c
CC ../../py/objslice.c
CC ../../py/objstr.c
CC ../../py/objstrunicode.c
CC ../../py/objstringio.c
CC ../../py/objtuple.c
CC ../../py/objtype.c
CC ../../py/objzip.c
CC ../../py/opmethods.c
CC ../../py/sequence.c
CC ../../py/stream.c
CC ../../py/binary.c
CC ../../py/builtinimport.c
CC ../../py/builtinevex.c
CC ../../py/builtinhelp.c
CC ../../py/modarray.c
CC ../../py/modbuiltins.c
CC ../../py/modcollections.c
CC ../../py/modgc.c
CC ../../py/modio.c
CC ../../py/modmath.c
CC ../../py/modcmath.c
CC ../../py/modmicropython.c
CC ../../py/modstruct.c
CC ../../py/modsys.c
CC ../../py/moduerrno.c
CC ../../py/modthread.c
CC ../../py/vm.c
CC ../../py/bc.c
CC ../../py/showbc.c
CC ../../py/repl.c
CC ../../py/smallint.c
CC ../../py/frozenmod.c
CC ../../extmod/moduasyncio.c
CC ../../extmod/moductypes.c
CC ../../extmod/modujson.c
CC ../../extmod/modure.c
CC ../../extmod/moduzlib.c
CC ../../extmod/moduheapq.c
CC ../../extmod/modutimeq.c
CC ../../extmod/moduhashlib.c
CC ../../extmod/moducryptolib.c
CC ../../extmod/modubinascii.c
CC ../../extmod/virtpin.c
CC ../../extmod/machine_mem.c
CC ../../extmod/machine_pinbase.c
CC ../../extmod/machine_signal.c
CC ../../extmod/machine_pulse.c
CC ../../extmod/machine_i2c.c
CC ../../extmod/machine_spi.c
CC ../../extmod/modbluetooth.c
CC ../../extmod/modussl_axtls.c
CC ../../extmod/modussl_mbedtls.c
CC ../../extmod/modurandom.c
CC ../../extmod/moduselect.c
CC ../../extmod/moduwebsocket.c
CC ../../extmod/modwebrepl.c
CC ../../extmod/modframebuf.c
CC ../../extmod/vfs.c
CC ../../extmod/vfs_blockdev.c
CC ../../extmod/vfs_reader.c
CC ../../extmod/vfs_posix.c
CC ../../extmod/vfs_posix_file.c
CC ../../extmod/vfs_fat.c
CC ../../extmod/vfs_fat_diskio.c
CC ../../extmod/vfs_fat_file.c
CC ../../extmod/vfs_lfs.c
CC ../../extmod/utime_mphal.c
CC ../../extmod/uos_dupterm.c
CC ../../lib/embed/abort_.c
CC ../../lib/utils/printf.c
MPY mypackage.py
GEN build/frozen_content.c
CC build/frozen_content.c
CC ../../lib/utils/gchelper_generic.c
CC ../../ports/unix/main.c
CC ../../ports/unix/input.c
CC ../../ports/unix/modos.c
CC ../../ports/unix/modmachine.c
CC ../../ports/unix/modtime.c
CC ../../ports/unix/gccollect.c
CC windows_mphal.c
CC realpath.c
CC init.c
CC sleep.c
CC fmode.c
CC ../../lib/mp-readline/readline.c
LINK micropython.exe
   text	   data	    bss	    dec	    hex	filename
    320	      0	      0	    320	    140	build/build/frozen_content.o
 272160	   3756	   8632	 284548	  45784	micropython.exe

EDIT: What is the old way of including frozen packages? Maybe that will work.

[jimmo]

So it looks like it built frozen_content.o which means it's doing the right thing there and actually freezing and linking the module. Possibly just support for loading frozen modules isn't being enabled in the build?

I added it and updated the Makefile with the frozen modules line.

Can you describe what you mean here? Did you add all the other bits from unix/Makefile -- in particular this section:

Code: Select all

ifneq ($(FROZEN_MANIFEST)$(FROZEN_MPY_DIR),)
# To use frozen code create a manifest.py file with a description of files to
# freeze, then invoke make with FROZEN_MANIFEST=manifest.py (be sure to build from scratch).
CFLAGS += -DMICROPY_QSTR_EXTRA_POOL=mp_qstr_frozen_const_pool
CFLAGS += -DMICROPY_MODULE_FROZEN_MPY
CFLAGS += -DMPZ_DIG_SIZE=16 # force 16 bits to work on both 32 and 64 bit archs
MPY_CROSS_FLAGS += -mcache-lookup-bc
endif

[Hitman1O1]

I only added the line:

Code: Select all

FROZEN_MANIFEST ?= manifest.py

After including your bit into the Makefile too I get the bytecode error again:

Code: Select all

MPY mypackage.py
GEN build/frozen_content.c
CC build/frozen_content.c
build/frozen_content.c:21:2: error: #error "incompatible MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE"
 #error "incompatible MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE"
  ^~~~~
../../py/mkrules.mk:63: recipe for target 'build/build/frozen_content.o' failed
make: *** [build/build/frozen_content.o] Error 1

[Hitman1O1]

I got it working! Thank you for your help, jimmo

The problem from my last post relates to the first, I changed "#define MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE (0)" in 'mpconfigport.h' and now just simply changed it back.

For completeness:

• The new Makefile:

Code: Select all

include ../../py/mkenv.mk
-include mpconfigport.mk

# File containing description of content to be frozen into firmware.
FROZEN_MANIFEST ?= manifest.py

# define main target
PROG = micropython.exe

# qstr definitions (must come before including py.mk)
QSTR_DEFS = ../unix/qstrdefsport.h

# include py core make definitions
include $(TOP)/py/py.mk

INC += -I.
INC += -I$(TOP)
INC += -I$(BUILD)

# compiler settings
CFLAGS = $(INC) -Wall -Wpointer-arith -Wdouble-promotion -Werror -std=gnu99 -DUNIX -D__USE_MINGW_ANSI_STDIO=1 $(CFLAGS_MOD) $(COPT) $(CFLAGS_EXTRA)
LDFLAGS = $(LDFLAGS_MOD) -lm $(LDFLAGS_EXTRA)

# Debugging/Optimization
ifdef DEBUG
CFLAGS += -g
COPT = -O0
else
COPT = -Os #-DNDEBUG
endif

# source files
SRC_C = \
	lib/utils/gchelper_generic.c \
	lib/utils/printf.c \
	ports/unix/main.c \
	ports/unix/input.c \
	ports/unix/modos.c \
	ports/unix/modmachine.c \
	ports/unix/modtime.c \
	ports/unix/gccollect.c \
	windows_mphal.c \
	realpath.c \
	init.c \
	sleep.c \
	fmode.c \
	$(SRC_MOD)

OBJ = $(PY_O) $(addprefix $(BUILD)/, $(SRC_C:.c=.o))

ifeq ($(MICROPY_USE_READLINE),1)
CFLAGS_MOD += -DMICROPY_USE_READLINE=1
SRC_C += lib/mp-readline/readline.c
endif

# Additional settings for frozen modules
ifneq ($(FROZEN_MANIFEST)$(FROZEN_MPY_DIR),)
# To use frozen code create a manifest.py file with a description of files to
# freeze, then invoke make with FROZEN_MANIFEST=manifest.py (be sure to build from scratch).
CFLAGS += -DMICROPY_QSTR_EXTRA_POOL=mp_qstr_frozen_const_pool
CFLAGS += -DMICROPY_MODULE_FROZEN_MPY
CFLAGS += -DMPZ_DIG_SIZE=16 # force 16 bits to work on both 32 and 64 bit archs
MPY_CROSS_FLAGS += -mcache-lookup-bc
endif

LIB += -lws2_32

# List of sources for qstr extraction
SRC_QSTR += $(SRC_C)
# Append any auto-generated sources that are needed by sources listed in
# SRC_QSTR
SRC_QSTR_AUTO_DEPS +=

include $(TOP)/py/mkrules.mk

• Create manifest.py:

Code: Select all

freeze('$(MPY_DIR)/ports/windows/modules/python', 'mypackage.py')

• Create modules/python/mypackage.py

[Hitman1O1]

Hm, looks like I "cheered too soon" as we say.

I can execute the generated micropython.exe and run something like:

Code: Select all

MicroPython v1.13-70-g7c1511b1d-dirty on 2020-09-08 (fork); win32 version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import mypackage
>>> led = mypackage.MyClass()
>>>

However, if I create a script:

Code: Select all

# main.py
import mypackage
led = mypackage.MyClass()

And try to run this script:

Code: Select all

micropython.exe ./main.py

I get:

Code: Select all

Traceback (most recent call last):
  File ".\main.py", line 1, in <module>
ImportError: no module named 'mypackage'

Why this is happening is beyond me. The way commands are executed cannot be different from the way a script is, right?
Note that I am positive I am using the right executable.

EDIT: Hm, just noticed the same happens with the unix port on the official micropython branch: import on CLI is fine but fails from script. So not specific to my Windows port.

EDIT 2: It gets better. So I cannot run the script, but if I import the script as a module instead the new packages do work:

Code: Select all

PS C:\Users\name\source\repos\micropython\ports\windows> .\micropython.exe .\script.py
Traceback (most recent call last):
  File ".\script.py", line 1, in <module>
ImportError: no module named 'mypackage'
PS C:\Users\name\source\repos\micropython\ports\windows> .\micropython.exe -m script
Hello world!

[stijn]

Can you doublecheck that the imported mypackage is effectively the frozen module, i.e. that it's not finding mypackage.py on the module search path? I assume this can be checked with mypackage.__file__, not sure though.

[Hitman1O1]

"mypackage.__file__" says "mypackage.py".
I am not sure whether this points to a frozen (baked) module or a local module.

But I copied my new micropython.exe to a virtual machine running an empty Windows and it ran there, with the same symptoms: importing from the REPL works but importing from a script does not. So I guess this confirms the freezing did work.

Thank you for your input!