MicroPython on ESP-WROVER with 4MB of psRAM

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
loboris
Posts: 50
Joined: Fri Oct 02, 2015 6:19 pm

MicroPython on ESP-WROVER with 4MB of psRAM

Postby loboris » Sat Jul 08, 2017 10:45 am

MicroPython works great on ESP32, but the most serious issue is still (as on most other MicroPython boards) limited amount of free memory.

Recently, I've got ESP-WROVER-KIT with ESP-WROVER module with 4MB psRAM, and I've tried MicroPython on it.
Some modifications were needed to make it work with external SPIRAM, but finaly I've got it runnig.

It is huge difference between MicroPython running with less than 100KB of free memory and running with 4MB of free memory :!:

The MicroPython task stack size is increased to 32 KB (64 KB or greater should also be possible).

I have also implemented internal Flash FAT file system using esp-idf wear leveling driver, so there is less danger to wear the flash out with frequent writes :D

Here is an example of running on ESP-WROVER:

Code: Select all

I (1404) cpu_start: PSRAM mode: flash 40m sram 40m
I (1405) cpu_start: PSRAM initialized, cache is in normal (1-core mode.
I (1409) cpu_start: Pro cpu up.
I (1421) cpu_start: Single core mode
I (4046) heap_alloc_caps: SPI SRAM memory test OK
I (4046) heap_alloc_caps: Initializing. RAM available for dynamic allocation:
I (4052) heap_alloc_caps: At 3F800000 len 00400000 (4096 KiB): SPIRAM
I (4074) heap_alloc_caps: At 3FFAE2A0 len 00001D60 (7 KiB): DRAM
I (4094) heap_alloc_caps: At 3FFC1318 len 0001ECE8 (123 KiB): DRAM
I (4115) heap_alloc_caps: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (4137) heap_alloc_caps: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (4158) heap_alloc_caps: At 40093AFC len 0000C504 (49 KiB): IRAM
I (4179) cpu_start: Pro cpu start user code
I (4240) cpu_start: Starting scheduler on PRO CPU.

Allocating uPY heap (4194048 bytes) in SPIRAM using malloc

FlashBdev init: Size=2093056, Blocks=511
Using wear leveling FAT file system
MicroPython v1.9.1-219-g3580284e-dirty on 2017-07-08; ESP32 module with ESP32
Type "help()" for more information.
>>> import os
>>> os.listdir()
['boot.py']
>>> import micropython
>>> micropython.mem_info()
stack: 736 out of 31744
GC: total: 4097984, used: 6064, free: 4091920
 No. of 1-blocks: 58, 2-blocks: 12, max blk sz: 264, max free sz: 254991
>>>
>>> ll = ['esp32'] * 200000
>>>
>>> micropython.mem_info()
stack: 736 out of 31744
GC: total: 4097984, used: 807760, free: 3290224
 No. of 1-blocks: 65, 2-blocks: 16, max blk sz: 50000, max free sz: 204991
>>>
>>> ll[123456]
'esp32'
>>>


ESP-WROVER module is still hard to get, but it is expected to be available in August/September.
Pycom is also offering the boards and OEM modules with 4MB of psRAM, to be available in August/September.


AnalogLamb is offering
ALB32-WROVER module with 8MB Flash and [b]4MB psRAM in ESP-WROOM-32 footprint package. ESP32-WROVER is also available for pre-order.

Available on GitHub.
Visit also ESP32 Forum for more detailes.
Last edited by loboris on Sun Aug 20, 2017 1:03 pm, edited 2 times in total.

loboris
Posts: 50
Joined: Fri Oct 02, 2015 6:19 pm

Re: MicroPython on ESP-WROVER with 4MB of psRAM

Postby loboris » Sun Jul 16, 2017 12:17 pm

GitHub repository with all necessary tools and souces to build the MicroPython firmware is now available:
https://github.com/loboris/MicroPython_ESP32_psRAM_LoBo

ESP32 Forum: https://www.esp32.com/viewtopic.php?f=17&t=2446

User avatar
pythoncoder
Posts: 1790
Joined: Fri Jul 18, 2014 8:01 am

Re: MicroPython on ESP-WROVER with 4MB of psRAM

Postby pythoncoder » Sun Jul 16, 2017 4:16 pm

That sounds like a major achievement. Can you tell more about how a psRAM build works? How does heap allocation "decide" between internal RAM and psRAM? Is bytecode preferentially assigned to RAM? Has performance been quantified compared to a build using internal RAM only?
Peter Hinch

loboris
Posts: 50
Joined: Fri Oct 02, 2015 6:19 pm

Re: MicroPython on ESP-WROVER with 4MB of psRAM

Postby loboris » Sun Jul 16, 2017 7:23 pm

At this time the complete MicroPython heap is allocated on psRAM.
Operations from psRAM are slower then from internal RAM, as it is accessed over SPI interface, with cache in internal RAM (32KB if I remember well). So, if we are operating with the data already in the cache, the speed will be the same. If the new block of data has to be placed in cache,there will be a delay. It works the same way the SPI Flash is accessed.

I've done some simple tests, keep in mind that for now I couldn't enable psRAM operating at 80MHz (only with MicroPython, it works with esp-idf applications) and that the psRAM support in eps-idf is at very early stage (not even officially supported yet).

Here are the rusults:

Running without psRAM support:

Code: Select all

FreeRTOS running on BOTH CORES, MicroPython task started on App Core.

Allocating uPY stack: size=16384 bytes
Allocating uPY heap:  size=94208 bytes

Reset reason: Power on reset Wakeup: Power on wake
OSError: [Errno 2] ENOENT
MicroPython b94fddd-dirty on 2017-07-16; ESP-WROVER module with ESP32+psRAM
Type "help()" for more information.
>>> import micropython,time
>>> t1=time.ticks_us(); x=['esp32'] * 15000; t2=time.ticks_us(); print(t2-t1)
3554
>>> micropython.mem_info()
stack: 736 out of 15360
GC: total: 92032, used: 68640, free: 23392
 No. of 1-blocks: 43, 2-blocks: 6, max blk sz: 3750, max free sz: 1428
>>>

==================

>>> t1=time.ticks_us(); x=['esp32'] * 1000; t2=time.ticks_us(); print(t2-t1)
383
>>> micropython.mem_info()
stack: 736 out of 15360
GC: total: 92032, used: 12640, free: 79392
 No. of 1-blocks: 43, 2-blocks: 6, max blk sz: 264, max free sz: 4928
>>>


With psRAM enabled:

Code: Select all

FreeRTOS running on BOTH CORES, MicroPython task started on App Core.

Allocating uPY stack: size=65536 bytes
Allocating uPY heap:  size=4194048 bytes (in SPIRAM using pvPortMallocCaps)

Reset reason: Power on reset Wakeup: Power on wake
OSError: [Errno 2] ENOENT
MicroPython b94fddd-dirty on 2017-07-16; ESP-WROVER module with ESP32+psRAM
Type "help()" for more information.
>>> import micropython,time
>>> t1=time.ticks_us(); x=['esp32'] * 15000; t2=time.ticks_us(); print(t2-t1)
17655
>>> micropython.mem_info()
stack: 736 out of 64512
GC: total: 4097984, used: 68640, free: 4029344
 No. of 1-blocks: 43, 2-blocks: 6, max blk sz: 3750, max free sz: 251800
>>>

==================

>>> t1=time.ticks_us(); x=['esp32'] * 1000; t2=time.ticks_us(); print(t2-t1)
692
>>> micropython.mem_info()
stack: 736 out of 64512
GC: total: 4097984, used: 12640, free: 4085344
 No. of 1-blocks: 43, 2-blocks: 6, max blk sz: 264, max free sz: 255300
>>>

==================

>>> t1=time.ticks_us(); x=['esp32'] * 150000; t2=time.ticks_us(); print(t2-t1)
184038
>>> micropython.mem_info()
stack: 736 out of 64512
GC: total: 4097984, used: 608848, free: 3489136
 No. of 1-blocks: 49, 2-blocks: 8, max blk sz: 37500, max free sz: 218037

Note the small difference with small amount of data.
With 80Mhz psRAM clock the times should be half of the measured values.

My measurements with simple memset function (on 2MB block allocated on psRAM) in esp-idf based application shows that the speed is ~7.5 MB/s with 40MHZ psRAM clock and ~14.3 MB/s with 80MHz psRAM clock.

How does heap allocation "decide" between internal RAM and psRAM?

There is a configuration option → Component config → ESP32-specific → Capability allocator can allocate SPI RAM memory (CONFIG_MEMMAP_SPIRAM_ENABLE).
This option is checked in app_main() in main.c to decide if psRAM can be used, see main.c for detailes.

I think all this is very promising, we jus have to wait a little for full esp-idf support and modules availability to take full advantage of ESP32 with psRAM capabilities. Of course, MicroPython itself could/should be optimized to work with psRAM.

User avatar
pythoncoder
Posts: 1790
Joined: Fri Jul 18, 2014 8:01 am

Re: MicroPython on ESP-WROVER with 4MB of psRAM

Postby pythoncoder » Mon Jul 17, 2017 8:17 am

Impressive, especially when you get 80MHz working.
Peter Hinch

loboris
Posts: 50
Joined: Fri Oct 02, 2015 6:19 pm

Re: MicroPython on ESP-WROVER with 4MB of psRAM

Postby loboris » Tue Jul 18, 2017 9:00 am

This is the test with SPI Flash and psRAM running at 80MHz:
x=['esp32'] * 150000
40MHz: 184038 usec
80MHz: 115515 usec
Not exactly half the time, but significant improvement (~63%).

Code: Select all

I (44) boot: ESP-IDF b94fddd-dirty 2nd stage bootloader
I (45) boot: compile time 10:43:52
I (71) boot: Enabling RNG early entropy source...
I (72) qio_mode: Enabling QIO for flash chip GD
I (76) boot: SPI Speed      : 80MHz
I (89) boot: SPI Mode       : QIO
I (101) boot: SPI Flash Size : 4MB
I (114) boot: Partition Table:
I (125) boot: ## Label            Usage          Type ST Offset   Length
I (148) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (171) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (194) boot:  2 factory          factory app      00 00 00010000 00100000
I (218) boot: End of partition table
I (231) boot: Disabling RNG early entropy source...
I (248) boot: Loading app partition at offset 00010000
I (1535) boot: segment 0: paddr=0x00010018 vaddr=0x00000000 size=0x0ffe8 ( 65512)
I (1536) boot: segment 1: paddr=0x00020008 vaddr=0x3f400010 size=0x2e4b0 (189616) map
I (1552) boot: segment 2: paddr=0x0004e4c0 vaddr=0x3ffb0000 size=0x03078 ( 12408) load
I (1580) boot: segment 3: paddr=0x00051540 vaddr=0x40080000 size=0x00400 (  1024) load
I (1605) boot: segment 4: paddr=0x00051948 vaddr=0x40080400 size=0x17084 ( 94340) load
I (1652) boot: segment 5: paddr=0x000689d4 vaddr=0x400c0000 size=0x00070 (   112) load
I (1658) boot: segment 6: paddr=0x00068a4c vaddr=0x00000000 size=0x075bc ( 30140)
I (1683) boot: segment 7: paddr=0x00070010 vaddr=0x400d0018 size=0xb1ce0 (728288) map
I (1710) cpu_start: PSRAM mode: flash 80m sram 80m
I (1726) cpu_start: PSRAM initialized, cache is in even/odd (2-core) mode.
I (1749) cpu_start: Pro cpu up.
I (1761) cpu_start: Starting app cpu, entry point is 0x400810a8
I (0) cpu_start: App cpu up.
I (3230) heap_alloc_caps: SPI SRAM memory test OK
I (3231) heap_alloc_caps: Initializing. RAM available for dynamic allocation:
I (3236) heap_alloc_caps: At 3F800000 len 00400000 (4096 KiB): SPIRAM
I (3258) heap_alloc_caps: At 3FFAE2A0 len 00001D60 (7 KiB): DRAM
I (3279) heap_alloc_caps: At 3FFB9170 len 00006E90 (27 KiB): DRAM
I (3299) heap_alloc_caps: At 3FFC0000 len 00010000 (64 KiB): DMAONLY
I (3321) heap_alloc_caps: At 3FFD0000 len 00010000 (64 KiB): DRAM
I (3342) heap_alloc_caps: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (3363) heap_alloc_caps: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (3385) heap_alloc_caps: At 40097484 len 00008B7C (34 KiB): IRAM
I (3406) cpu_start: Pro cpu start user code
I (3462) cpu_start: Starting scheduler on PRO CPU.
I (1683) cpu_start: Starting scheduler on APP CPU.

FreeRTOS running on BOTH CORES, MicroPython task started on App Core.

Allocating uPY stack: size=65536 bytes
Allocating uPY heap:  size=4194048 bytes (in SPIRAM using malloc)

Reset cause: 0
MicroPython 4ad81dd-dirty on 2017-07-15; ESP-WROVER module with ESP32+psRAM
Type "help()" for more information.
>>> import time,micropython
>>> micropython.mem_info()
stack: 736 out of 64512
GC: total: 4097984, used: 320, free: 4097664
 No. of 1-blocks: 6, 2-blocks: 7, max blk sz: 2, max free sz: 256095
>>> t1=time.ticks_us(); x=['esp32'] * 150000; t2=time.ticks_us(); print(t2-t1)
115515
>>> micropython.mem_info()
stack: 736 out of 64512
GC: total: 4097984, used: 600944, free: 3497040
 No. of 1-blocks: 13, 2-blocks: 7, max blk sz: 37500, max free sz: 218528
>>>

loboris
Posts: 50
Joined: Fri Oct 02, 2015 6:19 pm

Re: MicroPython on ESP-WROVER with 4MB of psRAM

Postby loboris » Fri Jul 21, 2017 3:48 pm

:!: Important update:

  • The MicroPython firmware is now built as esp-idf component.
  • menuconfig can be used for ESP32 configuration as well as for configuration of some MicroPython options.
  • The repository can be now used to build MicroPython for modules with psRAM as well as for ESP32 modules without psRAM. Standard esp-idf and Xtensa toolchain can be used if building without psRAM support.
  • micropython is no longer submodule, it's core is based on latest build (1.9.1) from main MicroPython repository, with some modifications needed for the new build system.
  • All ESP32 working modes (unicore/dualcore) and speeds, as well as all flash/psRAM speeds and modes are supported.
  • BUILD.sh script improved with more build options.
  • BTree module is included, can be Enabled/Disabled via menuconfig.
  • The repository README-md is updated with new information.

GitHub repository: https://github.com/loboris/MicroPython_ESP32_psRAM_LoBo
ESP32 Forum: https://www.esp32.com/viewtopic.php?f=17&t=2446
Last edited by loboris on Sat Jul 22, 2017 7:49 pm, edited 1 time in total.

User avatar
pythoncoder
Posts: 1790
Joined: Fri Jul 18, 2014 8:01 am

Re: MicroPython on ESP-WROVER with 4MB of psRAM

Postby pythoncoder » Sat Jul 22, 2017 6:18 am

Both the board and the work you've done look awesome :D You might want to tell Adafruit about it given that they sell the board and support MicroPython.
Peter Hinch

loboris
Posts: 50
Joined: Fri Oct 02, 2015 6:19 pm

Re: MicroPython on ESP-WROVER with 4MB of psRAM

Postby loboris » Sat Jul 22, 2017 7:57 pm

:D Good news for everybody wanting to work with ESP32+psRAM.

AnalogLamb is offering ALB32-WROVER module with 8MB Flash and 4MB psRAM in ESP-WROOM-32 footprint package. Available for pre-order, will be released on August 8, 2017.
https://www.analoglamb.com/product/alb32-wrover-esp32-module-with-64mb-flash-and-32mb-psram
Image

User avatar
pythoncoder
Posts: 1790
Joined: Fri Jul 18, 2014 8:01 am

Re: MicroPython on ESP-WROVER with 4MB of psRAM

Postby pythoncoder » Sun Jul 23, 2017 9:45 am

Interesting. I'd assumed that the Adafruit/Espressif boards had psRAM but looking at the schematic https://dl.espressif.com/dl/schematics/ESP-WROVER-KIT_SCH-3.pdf I can't see any. An unfortunate omission.

AAMOI why the ESD protection diodes on the SD card connector? Is there a possibility of hardware other than an SD card being plugged in?
Peter Hinch


Return to “ESP32 boards”

Who is online

Users browsing this forum: No registered users and 2 guests