How to add modules like ujson, upip and etc?

Discussion and questions about boards that can run MicroPython but don't have a dedicated forum.
Target audience: Everyone interested in running MicroPython on other hardware.
houneie
Posts: 8
Joined: Thu Sep 17, 2020 5:22 am

How to add modules like ujson, upip and etc?

Post by houneie » Thu Sep 17, 2020 5:36 am

Hello Everyone,

I am new to micropython world so advance apologies.

So basically I am an Intern in a company and they have ported basic support on Micropython to a board which they are developing right now. But they don't have modules like ujson, upip, urequest.

Can you please point me out to resources that I can use to port the above-mentioned modules.

Thanks in advance, any help will be appreciated.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: How to add modules like ujson, upip and etc?

Post by jimmo » Thu Sep 17, 2020 7:28 am

houneie wrote:
Thu Sep 17, 2020 5:36 am
So basically I am an Intern in a company and they have ported basic support on Micropython to a board which they are developing right now. But they don't have modules like ujson, upip, urequest.
Modules get added to the firmware in different ways depending on whether they're written in C or Python...

ujson is written in C, it's usually part of the main firmware and is enabled with the MICROPY_PY_UJSON macro in mpconfigport.h (or mpconfigboard.h). This enables extmod/modujson.c and registers the module in py/objmodule.c

upip and urequests are written in Python and are usually "frozen" into the firmware at compile time via the manifest. See (for example) esp32/boards/manifest.py and esp32/boards/manifest_release.py If your port is an existing architecture (e.g. stm32, esp32, etc) then you can use the existing board config mechanism to add your own manifest. If you've done a complete port from scratch then you will need to add some stuff to your makefile (or build process) to run make-manifest.py and add the generated frozen_content.c into the compile.

Alternatively for the written-in-Python modules like upip and urequests you can just copy over the Python implementations to the filesystem. upip comes from microython/micropython/tools/upip.py and urequests from micropython/micropython-lib/urequests/urequests.py

houneie
Posts: 8
Joined: Thu Sep 17, 2020 5:22 am

Re: How to add modules like ujson, upip and etc?

Post by houneie » Thu Sep 17, 2020 8:22 am

jimmo wrote:
Thu Sep 17, 2020 7:28 am
houneie wrote:
Thu Sep 17, 2020 5:36 am
So basically I am an Intern in a company and they have ported basic support on Micropython to a board which they are developing right now. But they don't have modules like ujson, upip, urequest.
Modules get added to the firmware in different ways depending on whether they're written in C or Python...

ujson is written in C, it's usually part of the main firmware and is enabled with the MICROPY_PY_UJSON macro in mpconfigport.h (or mpconfigboard.h). This enables extmod/modujson.c and registers the module in py/objmodule.c

upip and urequests are written in Python and are usually "frozen" into the firmware at compile time via the manifest. See (for example) esp32/boards/manifest.py and esp32/boards/manifest_release.py If your port is an existing architecture (e.g. stm32, esp32, etc) then you can use the existing board config mechanism to add your own manifest. If you've done a complete port from scratch then you will need to add some stuff to your makefile (or build process) to run make-manifest.py and add the generated frozen_content.c into the compile.

Alternatively for the written-in-Python modules like upip and urequests you can just copy over the Python implementations to the filesystem. upip comes from microython/micropython/tools/upip.py and urequests from micropython/micropython-lib/urequests/urequests.py
Thank you for a detailed reply. I enabled MICROPY_PY_UJSON by adding following line to mpconfigport.h:

#define MICROPY_PY_UJSON (1)

After compiling ujson wasn't in the list of modules. Do I have to add modujson.c myself? In my extmod folder there are only machine_mem.o, machine_mem.P, machine_pulse.o, machine_pulse.P, utime_mphal.o and utime_mphal.P


Update:
Basically I wasn't doing make clean before compiling again. I was in the build folder (my bad).

So basically, I placed modujson.c to /source/extmod. Now when I make then compilation fails with the following error:

Code: Select all

undefined reference to `mp_module_ujson'
collect2: error: ld returned 1 exit status
make: *** [Makefile:110: micropython.elf] Error 1

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: How to add modules like ujson, upip and etc?

Post by jimmo » Thu Sep 17, 2020 11:51 pm

houneie wrote:
Thu Sep 17, 2020 8:22 am
Do I have to add modujson.c myself?
Yeah, look at how (for example) the stm32 port pulls in extmod. (PY_O in your makefile should include the extmod files, via py.mk)

houneie
Posts: 8
Joined: Thu Sep 17, 2020 5:22 am

Re: How to add modules like ujson, upip and etc?

Post by houneie » Fri Sep 18, 2020 4:50 am

Will check and get back to you. Thank you very much for your support.

houneie
Posts: 8
Joined: Thu Sep 17, 2020 5:22 am

Re: How to add modules like ujson, upip and etc?

Post by houneie » Mon Sep 28, 2020 9:58 am

I have added https://github.com/micropython/micropyt ... modujson.c in my /extmod directory.
I have also updated py.mk file.

But when I make it throws me the following errors:

Code: Select all

Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
make: Warning: File 'modhelloworld.c' has modification time 1563275 s in the future
GEN build/genhdr/qstr.i.last
GEN build/genhdr/qstr.split
GEN build/genhdr/qstrdefs.collected.h
QSTR not updated
CC ../py/../extmod/modujson.c
../py/../extmod/modujson.c: In function 'mod_ujson_load':
../py/../extmod/modujson.c:187:24: error: too few arguments to function 'mp_obj_new_str'
                 next = mp_obj_new_str(vstr.buf, vstr.len);
                        ^~~~~~~~~~~~~~
In file included from ../../inc/py/objlist.h:29,
                 from ../py/../extmod/modujson.c:29:
../../inc/py/obj.h:642:10: note: declared here
 mp_obj_t mp_obj_new_str(const char* data, size_t len, bool make_qstr_if_not_already);
          ^~~~~~~~~~~~~~
../py/../extmod/modujson.c:298:25: error: implicit declaration of function 'MP_ERROR_TEXT'; did you mean 'MP_ROM_INT'? [-Werror=implicit-function-declaration]
     mp_raise_ValueError(MP_ERROR_TEXT("syntax error in JSON"));
                         ^~~~~~~~~~~~~
                         MP_ROM_INT
../py/../extmod/modujson.c:298:25: error: passing argument 1 of 'mp_raise_ValueError' makes pointer from integer without a cast [-Werror=int-conversion]
     mp_raise_ValueError(MP_ERROR_TEXT("syntax error in JSON"));
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../py/../extmod/modujson.c:32:
../../inc/py/runtime.h:150:47: note: expected 'const char *' but argument is of type 'int'
 NORETURN void mp_raise_ValueError(const char *msg);
                                   ~~~~~~~~~~~~^~~
cc1: all warnings being treated as errors
make: *** [../py/mkrules.mk:59: build/py/../extmod/modujson.o] Error 1

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: How to add modules like ujson, upip and etc?

Post by jimmo » Thu Oct 01, 2020 12:59 pm

houneie wrote:
Mon Sep 28, 2020 9:58 am
But when I make it throws me the following errors:
It sounds like maybe your fork is based on an older version of MicroPython (before we added string compression via the MP_ERROR_TEXT macro) but you've grabbed the very latest version of modujson.c ?

houneie
Posts: 8
Joined: Thu Sep 17, 2020 5:22 am

Re: How to add modules like ujson, upip and etc?

Post by houneie » Fri Oct 02, 2020 6:13 am

Can you point me out the commit version of which I should use modujson?

Thank you very much for your support.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: How to add modules like ujson, upip and etc?

Post by stijn » Fri Oct 02, 2020 7:09 am

You should already have a modujson.c in your repository (well, unless you have a version which is over 6 years old), that's the one you have to use. It's a bit hard to give advice because we don't know what your source tree looks like - normally modujson.c is in the extmod directory but you say you put it there. Does this mean you're using a custom version where this file was deleted? Is the repository still in version control? What does 'git log -n1' give as output? In any case you have to figure out which version of MicroPython this is based on, then use the corresponding modujson.c.

houneie
Posts: 8
Joined: Thu Sep 17, 2020 5:22 am

Re: How to add modules like ujson, upip and etc?

Post by houneie » Fri Oct 02, 2020 12:14 pm

Basically this a custom version fro RISCV based board (hifive1-revb). Yes, files were deleted. The repository is in version control and the directory structure is different when compared to official micropython's.

Post Reply