Tool uPyLoader for MP on esp32

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
profra
Posts: 39
Joined: Sat Jan 03, 2015 12:23 am

Tool uPyLoader for MP on esp32

Post by profra » Sat Nov 04, 2017 3:23 pm

@Damien I have been using uPyLoader (uPL) (https://github.com/BetaRavener/uPyLoader) to work on esp8266 with MP for a long time. It is a handy tool for beginners (like me)... has GUI and works on both platforms WIN and NIX. A few days ago I started playing with MP on esp32 and I wanted to do it with the same tool... BUT it doesn't work quite right. It is not possible to upload two basic scripts (__upload.py, __download.py) to esp32. Without them is uPL useless. It seems that the culprit is UART(0) which is blocked only for REPL...throwed error 'ValueError: UART(0) is disabled (dedicated to REPL)'...look at the attached file. uPL needs this channel for his (custom) communication.
The reasons I have described above lead me to ask you whether you can help somehow ... either modify something within the MP or give a good advice on how to deal with this problem. I communicate with the author of uPL and he will surely be able to do anything needed on his side. Many thanks in advance for your help.

Code: Select all

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
Falling back to built-in command interpreter.
OK
>ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:4332
load:0x40078000,len:0
load:0x40078000,len:10992
entry 0x4007a6c4
[0;32mI (204) cpu_start: Pro cpu up.[0m
[0;32mI (204) cpu_start: Single core mode[0m
[0;32mI (205) heap_init: Initializing. RAM available for dynamic allocation:[0m
[0;32mI (208) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM[0m
[0;32mI (214) heap_init: At 3FFDCD60 len 000032A0 (12 KiB): DRAM[0m
[0;32mI (220) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM[0m
[0;32mI (227) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM[0m
[0;32mI (233) heap_init: At 4008FC7C len 00010384 (64 KiB): IRAM[0m
[0;32mI (239) cpu_start: Pro cpu start user code[0m
[0;32mI (33) cpu_start: Starting scheduler on PRO CPU.[0m
OSError: [Errno 2] ENOENT
MicroPython v1.9.2-443-g236297f4 on 2017-11-04; ESP32 module with ESP32
Type "help()" for more information.
>>> 
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== #V1
=== from machine import UART
=== import time
=== 
=== def read_timeout(uart, cnt, retries=1000):
===     data = b""
===     for i in range(0, retries):
===         rec = uart.read(cnt - len(data))
===         if rec:
===             data += rec
===             if len(data) == cnt:
===                 return data
===         time.sleep(0.01)
===     return None
=== 
=== def main():
===     uart = UART(0, 115200)
===     suc = False
===     with open("__upload.py", "wb") as f:
===         while True:
===             d = read_timeout(uart, 2)
===             if not d or d[0] != ord("#"):
===                 x = uart.write(b"#2")
===                 break
===             cnt = d[1] & 0x7F
===             if cnt == 0:
===                 suc = True
===                 break
===             d = read_timeout(uart, cnt)
===             if d:
===                 esc = False
===                 for c in d:
===                     if c == 0:
===                         esc = True
===                         continue
===                     x = f.write(bytes([c & 0x0F if esc else c]))
===                     esc = False
===                 x = uart.write(b"#1")
===             else:
===                 x = uart.write(b"#3")
===                 break
===     x = uart.write(b"#1#" if suc else b"#0#")
=== 
=== main()
=== 
Traceback (most recent call last):
  File "<stdin>", line 44, in <module>
  File "<stdin>", line 17, in main
ValueError: UART(0) is disabled (dedicated to REPL)
>>> V1from machine import UARTimport timedef read_timeout(uart,file_name="__download.py"
... with open("__upload.py") as f:
...         exec(f.read(), globals())
...         V1from machine import UARTimport timedef read_timeout(uart
>>> 
>>> 
>>> 

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Tool uPyLoader for MP on esp32

Post by Roberthh » Sat Nov 04, 2017 5:40 pm

Looking into the mcu scripts in the link you gave, it seems that you could replace UART.read() by sys.stdin.buffer.read(), and UART.write() by sys.stdout.buffer.write(). You have to disable keyboard interrupt with micropython.kbd_intr(-1). You can re-enable it, when the script closes, with micropython.kbd_intr(3). This method works on ESP8266 too.
The major difference: sys.stdin.buffer.read() is blocking. If no data arrives, it will wait forever, whereas uart.read() returns immediately with the data it has buffered. So that may lead to a locked device, especially if Keyboard-interrupt is disabled.

And yes: the place in the code which prevent using UART 0 is at line 196 in machine_uart.c. Changing that could be a try.

profra
Posts: 39
Joined: Sat Jan 03, 2015 12:23 am

Re: Tool uPyLoader for MP on esp32

Post by profra » Sat Nov 04, 2017 8:39 pm

@Roberthh Thank you very much for the exhausting answer ... we will try all the suggested options.

Beta_Ravener
Posts: 35
Joined: Tue Aug 09, 2016 6:56 pm

Re: Tool uPyLoader for MP on esp32

Post by Beta_Ravener » Sun Nov 05, 2017 1:36 pm

Hi, thanks for the useful info. I will try to experiment a little and see if I can make it work with sys module. I would just like to know, is there some reasoning why the UART became reserved?

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Tool uPyLoader for MP on esp32

Post by Roberthh » Sun Nov 05, 2017 3:01 pm

That's what the source code tells:

Code: Select all

    // Attempts to use UART0 from Python has resulted in all sorts of fun errors.
    // FIXME: UART0 is disabled for now.
    if (uart_num == UART_NUM_0) {
        nlr_raise(mp_obj_new_exception_msg_varg(&mp_type_ValueError, "UART(%d) is disabled (dedicated to REPL)", uart_num));
    }
SO no details. You might try yourself.
Edit: I gave it a try. No fun -> immediate call for the Guru == hard crash.
Besides that: You might look at tools like rshell and ampy, how they handle file transfer. Both work with ESP32. AFAIK, rshell uses tool/pyboard.py

profra
Posts: 39
Joined: Sat Jan 03, 2015 12:23 am

Re: Tool uPyLoader for MP on esp32

Post by profra » Sun Nov 05, 2017 8:32 pm

@Roberthh, @Beta_Ravener Hi, I did the same try according to your suggestion and I can only confirm... Guru in action!
I should have read your comment first :D
I have installed MP from Loboris, the installation is SO EASY...
but I have checked (compared) machine_uart.c from both (Damien and Loboris)...
the differences were not (IMO) significant for this try. The matter is definitely more complicated than I can do.

Code: Select all

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:5676
load:0x40078000,len:0
load:0x40078000,len:12532
entry 0x40078f84
[0;32mI (28) boot: ESP-IDF ESP32_LoBo_v1.9.1-37-g405fcaa-dirty 2nd stage bootloader[0m
[0;32mI (28) boot: compile time 22:11:14[0m
[0;32mI (30) boot: Enabling RNG early entropy source...[0m
[0;32mI (36) boot: SPI Speed      : 40MHz[0m
[0;32mI (40) boot: SPI Mode       : DIO[0m
[0;32mI (44) boot: SPI Flash Size : 4MB[0m
[0;32mI (48) boot: Partition Table:[0m
[0;32mI (52) boot: ## Label            Usage          Type ST Offset   Length[0m
[0;32mI (59) boot:  0 nvs              WiFi data        01 02 00009000 00006000[0m
[0;32mI (66) boot:  1 phy_init         RF data          01 01 0000f000 00001000[0m
[0;32mI (74) boot:  2 MicroPython      factory app      00 00 00010000 00270000[0m
[0;32mI (81) boot:  3 internalfs       Unknown data     01 81 00280000 00140000[0m
[0;32mI (89) boot: End of partition table[0m
[0;32mI (93) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x481b8 (295352) map[0m
[0;32mI (160) esp_image: segment 1: paddr=0x000581e0 vaddr=0x3ffb0000 size=0x07e30 ( 32304) load[0m
[0;32mI (166) esp_image: segment 2: paddr=0x00060018 vaddr=0x400d0018 size=0xbbc94 (769172) map[0m
[0;32mI (317) esp_image: segment 3: paddr=0x0011bcb4 vaddr=0x3ffb7e30 size=0x04a2c ( 18988) load[0m
[0;32mI (321) esp_image: segment 4: paddr=0x001206e8 vaddr=0x40080000 size=0x00400 (  1024) load[0m
[0;32mI (324) esp_image: segment 5: paddr=0x00120af0 vaddr=0x40080400 size=0x14fc8 ( 85960) load[0m
[0;32mI (352) esp_image: segment 6: paddr=0x00135ac0 vaddr=0x400c0000 size=0x00064 (   100) load[0m
[0;32mI (352) esp_image: segment 7: paddr=0x00135b2c vaddr=0x50000000 size=0x00904 (  2308) load[0m
[0;32mI (365) boot: Loaded app from partition at offset 0x10000[0m
[0;32mI (365) boot: Disabling RNG early entropy source...[0m
[0;32mI (370) cpu_start: Pro cpu up.[0m
[0;32mI (374) cpu_start: Starting app cpu, entry point is 0x40081240[0m
[0;32mI (0) cpu_start: App cpu up.[0m
[0;32mI (384) heap_init: Initializing. RAM available for dynamic allocation:[0m
D (391) heap_init: New heap initialised at 0x3ffae6e0[0m
[0;32mI (396) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM[0m
D (402) heap_init: New heap initialised at 0x3ffcb360[0m
[0;32mI (407) heap_init: At 3FFCB360 len 00014CA0 (83 KiB): DRAM[0m
[0;32mI (413) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM[0m
[0;32mI (420) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM[0m
D (426) heap_init: New heap initialised at 0x400953c8[0m
[0;32mI (431) heap_init: At 400953C8 len 0000AC38 (43 KiB): IRAM[0m
[0;32mI (438) cpu_start: Pro cpu start user code[0m
D (450) clk: RTC_SLOW_CLK calibration value: 3530573[0m
D (12) intr_alloc: Connected src 46 to int 2 (cpu 0)[0m
D (12) intr_alloc: Connected src 57 to int 3 (cpu 0)[0m
D (13) intr_alloc: Connected src 16 to int 9 (cpu 0)[0m
D (17) intr_alloc: Connected src 24 to int 12 (cpu 0)[0m
[0;32mI (23) cpu_start: Starting scheduler on PRO CPU.[0m
D (0) intr_alloc: Connected src 25 to int 2 (cpu 1)[0m
[0;32mI (4) cpu_start: Starting scheduler on APP CPU.[0m
D (47) heap_init: New heap initialised at 0x3ffe0440[0m
D (53) heap_init: New heap initialised at 0x3ffe4350[0m
D (58) nvs: nvs_flash_init_custom partition=nvs start=9 count=6[0m

FreeRTOS running on BOTH CORES, MicroPython task started on App Core.

uPY stack size = 19456 bytes
uPY  heap size = 81920 bytes

D (111) intr_alloc: Connected src 34 to int 3 (cpu 1)[0m
Reset reason: Power on reset
D (119) intr_alloc: Connected src 22 to int 4 (cpu 1)[0m
MicroPython ESP32_LoBo_v2.0.7 - 2017-11-01 on ESP32 board with ESP32
Type "help()" for more information.
>>> 
>>> 
>>> 
>>> 
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== #V1
=== from machine import UART
=== import time
=== 
=== def read_timeout(uart, cnt, retries=1000):
===     data = b""
===     for i in range(0, retries):
===         rec = uart.read(cnt - len(data))
===         if rec:
===             data += rec
===             if len(data) == cnt:
===                 return data
===         time.sleep(0.01)
===     return None
=== 
=== def main():
===     uart = UART(0, 115200)
===     suc = False
===     with open("__upload.py", "wb") as f:
===         while True:
===             d = read_timeout(uart, 2)
===             if not d or d[0] != ord("#"):
===                 x = uart.write(b"#2")
===                 break
===             cnt = d[1] & 0x7F
===             if cnt == 0:
===                 suc = True
===                 break
===             d = read_timeout(uart, cnt)
===             if d:
===                 esc = False
===                 for c in d:
===                     if c == 0:
===                         esc = True
===                         continue
===                     x = f.write(bytes([c & 0x0F if esc else c]))
===                     esc = False
===                 x = uart.write(b"#1")
===             else:
===                 x = uart.write(b"#3")
===                 break
===     x = uart.write(b"#1#" if suc else b"#0#")
=== 
=== main()
=== 
Guru Meditation Error of type LoadProhibited occurred on core  1. Exception was unhandled.
Register dump:
PC      : 0x400d1aaa  PS      : 0x00060d30  A0      : 0x8015caea  A1      : 0x3ffbace0  
A2      : 0x00790070  A3      : 0x00000000  A4      : 0x4015b08c  A5      : 0x3ffcbfa4  
A6      : 0x3ffcbfb0  A7      : 0x00000003  A8      : 0x80089094  A9      : 0x3ffbacd0  
A10     : 0x00000003  A11     : 0x00060f23  A12     : 0x00060f20  A13     : 0x00000000  
A14     : 0x4015b08c  A15     : 0x3ffcbfa4  SAR     : 0x0000001c  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00790070  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000000  

Backtrace: 0x400d1aaa:0x3ffbace0 0x4015cae7:0x3ffbad00 0x4015cecb:0x3ffbad30 0x4012d8a5:0x3ffbad80 0x401227d3:0x3ffbadd0 0x4011a255:0x3ffbadf0 0x40126bd3:0x3ffbae10 0x4011e4f2:0x3ffbaeb0 0x4011a255:0x3ffbaee0 0x40126bd3:0x3ffbaf00 0x4011e4f2:0x3ffbafa0 0x4011a255:0x3ffbb000 0x4011a282:0x3ffbb020 0x4010e58b:0x3ffbb040 0x4010e7e9:0x3ffbb0e0 0x401083f1:0x3ffbb120

CPU halted.

profra
Posts: 39
Joined: Sat Jan 03, 2015 12:23 am

Re: Tool uPyLoader for MP on esp32

Post by profra » Tue Nov 07, 2017 10:29 pm

@Roberthh, @Beta_Ravener FYI... I spent a little more time on the problem and found the next thing... I have flashed WEMOS Lolin32 with binaries for WiPy 2 and the uPL is working normally (in addition, it is still possible to connect via ftp). I compared the files machine_uart.c (Damien's MP) and machuart.c (PYCOM)... it is clear that massive modifications have been made on the PYCOM side but the original module is obviously from Damien's MP... the modifications made are beyond my knowledge :(
P.S. @Roberthh It seems that you have a lot of experience with MP. Do you see some problems/obstacles if one will use SW for WiPy 2 on the development modules bought on Aliexpress, Bangood, eBay...? I would be grateful for your opinion.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Tool uPyLoader for MP on esp32

Post by Roberthh » Wed Nov 08, 2017 6:18 am

I use the WiPY 2 software myself on a Lolin32 module, because it is much more convenient to use (built-in USB). WiPy is not just a ESP32, power supply, antenna switch and RGB led on a module. The only physical difference to the Lolin32 is the external antenna switch, which you can ignore on the LoLino32. It uses GPIO16, which means, you cannot use that in your code, unless you change the firmware. And obviously the RGB led is missing.
The Pin numbers are not matching, but if you want, you can make your own board description file. It look not so difficult, I just didn't do that yet.

profra
Posts: 39
Joined: Sat Jan 03, 2015 12:23 am

Re: Tool uPyLoader for MP on esp32

Post by profra » Wed Nov 08, 2017 10:27 am

@Roberthh Thank you for your opinion. I also thought about all of from your side mentioned things, and I am glad that there is nothing unknown to me. Yes, the Pin numbers are not matching... do you mean that one needs some remapping module in MP or some in C and build new bin?

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Tool uPyLoader for MP on esp32

Post by Roberthh » Wed Nov 08, 2017 11:26 am

in the esp32/boards subdirectory there is a path for each board, including a file called pins.csv. From what I understand, a new board directory has to be made (e.g. LOLIN32) as the copy of WIPY and the respective files have to be edited. Their content seems pretty obvious.

Post Reply