ulab for ESP8266

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
imaditya
Posts: 1
Joined: Thu Jul 23, 2020 10:36 am

ulab for ESP8266

Post by imaditya » Thu Jul 23, 2020 10:41 am

Hi,

I just got an ESP8266 and flashed micropython firmware on it. My main goal was to try out the ulab(port of numpy) before ordering a D series PyBoard.

Does anyone have any pointers on how to install ulab on ESP8266 micrpython.

Thanks :)

User avatar
rcolistete
Posts: 336
Joined: Thu Dec 31, 2015 3:12 pm
Location: Brazil
Contact:

Re: ulab for ESP8266

Post by rcolistete » Thu Jul 23, 2020 5:44 pm

Tried compiling ulab on ESP8266, but there are issues with ESP8266 partitions :

Code: Select all

LINK build-GENERIC/firmware.elf
xtensa-lx106-elf-ld: build-GENERIC/firmware.elf section `.text' will not fit in region `iram1_0_seg'
xtensa-lx106-elf-ld: region `iram1_0_seg' overflowed by 64796 bytes
Need to try changing the flash layout, which is not simple at all on ESP8266.
My "MicroPython Samples". My "MicroPython Firmwares" with many options (double precision, ulab, etc).

User avatar
rcolistete
Posts: 336
Joined: Thu Dec 31, 2015 3:12 pm
Location: Brazil
Contact:

Re: ulab for ESP8266

Post by rcolistete » Thu Jul 23, 2020 10:30 pm

There are, for the moment, 3 issues when building MicroPython v1.12 + ulab on ESP8266 :
1) the one cited above, `iram1_0_seg' overflowed, I think the fix is to changing the flash layout and/or place ulab in "irom0_0_seg", i. e., changing the MicroPython source code;
2) "'out' may be used uninitialized error in function 'numerical_sort_helper' of 'numerical.c'", solved by this ulab PR#153;
3) some math functions are not visible in MicroPython on ESP8266, but they are used by ulab :

Code: Select all

build-GENERIC/code/vectorise.o:(.text.vectorise_acosh+0x0): undefined reference to `acoshf'
build-GENERIC/code/vectorise.o:(.text.vectorise_asinh+0x0): undefined reference to `asinhf'
build-GENERIC/code/vectorise.o:(.text.vectorise_atanh+0x0): undefined reference to `atanhf'
build-GENERIC/code/vectorise.o:(.text.vectorise_erf+0x0): undefined reference to `erff'
build-GENERIC/code/vectorise.o:(.text.vectorise_erfc+0x0): undefined reference to `erfcf'
build-GENERIC/code/vectorise.o:(.text.vectorise_gamma+0x0): undefined reference to `tgammaf'
build-GENERIC/code/vectorise.o:(.text.vectorise_lgamma+0x0): undefined reference to `lgammaf'
build-GENERIC/code/vectorise.o:(.text.vectorise_log2+0x0): undefined reference to `log2f'
the solution seems to be modifying ".../ports/ESP8266/Makefile" to include this functions, i. e., changing the MicroPython source code.
Last edited by rcolistete on Fri Jul 24, 2020 3:44 am, edited 2 times in total.
My "MicroPython Samples". My "MicroPython Firmwares" with many options (double precision, ulab, etc).

User avatar
rcolistete
Posts: 336
Joined: Thu Dec 31, 2015 3:12 pm
Location: Brazil
Contact:

Re: ulab for ESP8266

Post by rcolistete » Thu Jul 23, 2020 10:43 pm

imaditya wrote:
Thu Jul 23, 2020 10:41 am
I just got an ESP8266 and flashed micropython firmware on it. My main goal was to try out the ulab(port of numpy) before ordering a D series PyBoard.
Which ESP8266 ? How much flash size ? ulab takes a lot of space (> 40 kB) for a ESP8266 module, so I think that ESP8266 with smaller flash sizes (512 kB or 1 MB) can't have ulab at all.

About ordering a Pyboard (D SF2 ?), I recommed, there are many Pyboard D features worth buying one.
imaditya wrote:
Thu Jul 23, 2020 10:41 am
Does anyone have any pointers on how to install ulab on ESP8266 micrpython.
It is needed to compile MicroPython source code to build a firmware with ulab included. Follow :
- MicroPython port to ESP8266 instructions, e. g., using the docker commands;
- MicroPython ulab instructions, i. e. :
* put ulab folder in the same folder where micropython folder is;
* add "USER_C_MODULES=../../../ulab all" in the end of make commands.

For example, on Linux terminal :

Code: Select all

[micropython]$ cd mpy-cross
[mpy-cross]$ docker run --rm -v $HOME:$HOME -u $UID -w $PWD larsks/esp-open-sdk make 
[mpy-cross]$ cd ../ports/esp8266
[esp8266]$ docker run --rm -v $HOME:$HOME -u $UID -w $PWD larsks/esp-open-sdk make submodules
[esp8266]$ docker run --rm -v $HOME:$HOME -u $UID -w $PWD larsks/esp-open-sdk make clean
[esp8266]$ docker run --rm -v $HOME:$HOME -u $UID -w $PWD larsks/esp-open-sdk make -j8 USER_C_MODULES=../../../ulab all
If I succed in building MicroPython + ulab on ESP866, I will post here a link to the firmware.
My "MicroPython Samples". My "MicroPython Firmwares" with many options (double precision, ulab, etc).

User avatar
rcolistete
Posts: 336
Joined: Thu Dec 31, 2015 3:12 pm
Location: Brazil
Contact:

Re: ulab for ESP8266

Post by rcolistete » Fri Jul 24, 2020 1:45 am

rcolistete wrote:
Thu Jul 23, 2020 10:30 pm
There are, for the moment, 3 issues when building MicroPython v1.12 + ulab on ESP8266 :
1) the one cited above, `iram1_0_seg' overflowed, I think the fix is to changing the flash layout and/or place ulab in "irom0_0_seg", i. e., changing the MicroPython source code;
This issue (1) seems to be solved. Edit the file "../ports/esp8266/boards/esp8266_common.ld", and add the line

Code: Select all

        *code/*.o*(.literal* .text*)   /* ulab module */
for example, in line 183, after :

Code: Select all

        */frozen.o(.rodata.mp_frozen_content) /* frozen modules */
My "MicroPython Samples". My "MicroPython Firmwares" with many options (double precision, ulab, etc).

User avatar
rcolistete
Posts: 336
Joined: Thu Dec 31, 2015 3:12 pm
Location: Brazil
Contact:

Re: ulab for ESP8266

Post by rcolistete » Fri Jul 24, 2020 3:07 am

Solved the issue (3), succeeded building MicroPython firmware with ulab v0.54.0 for ESP8266 ! Yeah, the new version of ulab, release some hours ago.

I'll later (time to sleep...) describe the needed changes in '...esp8266/Makefile' and '...esp8266/mpconfigport.h', make a "README.md" in the repository below, etc.

Link for the firmware MicroPython daily build v1.12-652-gcaaaa2b1f, with ulab v0.54.0 + sp (single precision), for Generic ESP8266 module, 2M or more of flash.

Code: Select all

MicroPython v1.12-652-gcaaaa2b1f-dirty on 2020-07-24; ESP module with ESP8266
Type "help()" for more information.
>>> import gc
>>> gc.collect()
>>> gc.mem_free()
36656
>>> import ulab as np
>>> np.__version__
'0.54.0'
Tried fft with 1024 and 2048 points, it works ! So preliminary benchmarks for fft of 1024 points, sp (single precision), without threads, :
- ESP8266 (WeMos D1 Mini) + MicroPython v1.12-652-gcaaaa2b1f-dirty on 2020-07-24 with ulab, default clock (80 MHz) :
* 57.509 ms;
- ESP8266 (WeMos D1 Mini) + MicroPython v1.12-652-gcaaaa2b1f-dirty on 2020-07-24 with ulab, overclock (160 MHz)
* 28.936 ms;
- Pyboard D SF2 + MicroPython v1.12-624-g57333f6ea on 2020-07-13 with ulab, default clock (120 MHz) :
* 1.873 ms;
- Pyboard D SF2 + MicroPython v1.12-652-gcaaaa2b1f-dirty on 2020-07-23 with ulab in QSPI flash, default clock (120 MHz) :
* 2.070 ms.
My "MicroPython Samples". My "MicroPython Firmwares" with many options (double precision, ulab, etc).

Online
User avatar
pythoncoder
Posts: 4917
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ulab for ESP8266

Post by pythoncoder » Fri Jul 24, 2020 5:23 am

Those benchmarks are interesting, notably the minimal (~10%) penalty associated with using QSPI Flash on the Pyboard D.

I'm impressed you got it to run on an ESP8266.
Peter Hinch

v923z
Posts: 155
Joined: Mon Dec 28, 2015 6:19 pm

Re: ulab for ESP8266

Post by v923z » Fri Jul 24, 2020 2:16 pm

rcolistete wrote:
Fri Jul 24, 2020 3:07 am
Solved the issue (3), succeeded building MicroPython firmware with ulab v0.54.0 for ESP8266 ! Yeah, the new version of ulab, release some hours ago.

I'll later (time to sleep...) describe the needed changes in '...esp8266/Makefile' and '...esp8266/mpconfigport.h', make a "README.md" in the repository below, etc.
I am not against posting the instructions on https://github.com/v923z/micropython-ulab/. Raise an issue, if you want to do that!
rcolistete wrote:
Fri Jul 24, 2020 3:07 am
Tried fft with 1024 and 2048 points, it works !
The FFT takes place in place, so as long as you can fit your data (watch out for the imaginary part!) in RAM, you should be fine. I only use a couple of bytes for temporary variables. 4096 points might be a bit tight, though.

User avatar
rcolistete
Posts: 336
Joined: Thu Dec 31, 2015 3:12 pm
Location: Brazil
Contact:

Re: ulab for ESP8266

Post by rcolistete » Sat Jul 25, 2020 8:00 am

rcolistete wrote:
Fri Jul 24, 2020 3:07 am
Solved the issue (3), succeeded building MicroPython firmware with ulab v0.54.0 for ESP8266 ! Yeah, the new version of ulab, release some hours ago.

I'll later (time to sleep...) describe the needed changes in '...esp8266/Makefile' and '...esp8266/mpconfigport.h', make a "README.md" in the repository below, etc.
I've made a MicroPython branch (in my fork), "esp8266_compatible_ulab", with commit "ESP8266 compatible with ulab module : math functions and ulab in irom", and also submitted the MicroPython PR#6284 "ESP8266 made compatible with ulab module", which solves the issues (1) and (3) cited above, by changing files :
- "../ports/esp8266/boards/esp8266_common.ld" so ulab is moved from IRAM1 to IROM0, fixing the "`iram1_0_seg' overflowed" error;
- '.../esp8266/Makefile' and '.../esp8266/mpconfigport.h' to define some math functions used by ulab, fixing the building/linking errors "undefined reference to acoshf, asinhf, atanhf, erff, erfcf, tgammaf, lgammaf, log2f."

Issue (2) was merged in ulab master after PR#153, "numerical.c : fixed 'out' may be used uninitialized error in function…".

So, to compile ulab + MicroPython for ESP8266, use :
- ulab source code from July 24th 2020 onwards;
- temporary my MicroPython branch, "esp8266_compatible_ulab" while the PR#6284 isn't merged in MicroPython upstream/master.
My "MicroPython Samples". My "MicroPython Firmwares" with many options (double precision, ulab, etc).

User avatar
rcolistete
Posts: 336
Joined: Thu Dec 31, 2015 3:12 pm
Location: Brazil
Contact:

Re: ulab for ESP8266

Post by rcolistete » Sat Jul 25, 2020 8:08 am

v923z wrote:
Fri Jul 24, 2020 2:16 pm
I am not against posting the instructions on https://github.com/v923z/micropython-ulab/. Raise an issue, if you want to do that!
If the PR#6284 is merged, then the instructions to compile MicroPython for ESP8266 + ulab will be very simple : use ulab >= 0.54.0 & MicroPython newer than 'some date'.
v923z wrote:
Fri Jul 24, 2020 2:16 pm
The FFT takes place in place, so as long as you can fit your data (watch out for the imaginary part!) in RAM, you should be fine. I only use a couple of bytes for temporary variables. 4096 points might be a bit tight, though.
After FFT with 2048 points, the free RAM on ESP8266 (WeMos D1 Mini) & MicroPython v1.12 2020-07-24 is very small, just 2800 bytes (from 35968 bytes of free RAM after boot/restart).
So no way to run FFT with 4096 bytes on MicroPython for ESP8266.
My "MicroPython Samples". My "MicroPython Firmwares" with many options (double precision, ulab, etc).

Post Reply