A simple shell for pyboard
-
- Posts: 16
- Joined: Sun Dec 17, 2017 3:46 pm
Re: A simple shell for pyboard
Thanks Robert and Dave!
I'm a beginner, so it may or may not be "easy" for me to write a cp method, but yea, you've both given me a bunch of examples to work from (and encouragement!) so I might give it a try...
Yes, using cat with redirection will work...
Thanks much!
Keith
I'm a beginner, so it may or may not be "easy" for me to write a cp method, but yea, you've both given me a bunch of examples to work from (and encouragement!) so I might give it a try...
Yes, using cat with redirection will work...
Thanks much!
Keith
Re: A simple shell for pyboard
upysh is really cool, I have placed it on all my MicroPython modules, and do "from upysh import *" when needed.
Two changelists, this is the last one fixing n<=0 corner cases.
I want to add "wc" as well. Counting lines and bytes is easy, but for counting words as Linux "wc" does an "isalpha()" is needed that works for complete unicode character set [there are 99537 unicode characters for which isalpha() returns True]. But MicroPython "isalpha()" only works for ASCII characters:
viewtopic.php?f=2&t=5396
Last, but not least, this is "tail()":
I did add "tail" command, not sure whether I should raise PR.
Two changelists, this is the last one fixing n<=0 corner cases.
I want to add "wc" as well. Counting lines and bytes is easy, but for counting words as Linux "wc" does an "isalpha()" is needed that works for complete unicode character set [there are 99537 unicode characters for which isalpha() returns True]. But MicroPython "isalpha()" only works for ASCII characters:
viewtopic.php?f=2&t=5396
Last, but not least, this is "tail()":
Code: Select all
def tail(f, n=10):
with open(f) as f:
if n<=0: return
a = [ "" for i in range(n) ]
i = 0
while True:
l = f.readline()
if not l: break
a[i % n] = l
i += 1
if i>0 and i<n:
for j in range(i+1):
sys.stdout.write(a[j])
else:
for j in range(n):
sys.stdout.write(a[(i+j)%n])
Pico-W Access Point static file webserver:
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
Re: A simple shell for pyboard
PRs are definitely the preferred approach.
Note that upysh (from Micropython-lib) was written by pfalcon. The simple shell that I wrote is found here: https://github.com/dhylands/upy-shell
I find I tend to use rshell https://github.com/dhylands/rshell now rather than upy-shell
Note that upysh (from Micropython-lib) was written by pfalcon. The simple shell that I wrote is found here: https://github.com/dhylands/upy-shell
I find I tend to use rshell https://github.com/dhylands/rshell now rather than upy-shell
Re: A simple shell for pyboard
Thanks.
I went a step further and was able to implement isalpha() for whole unicode. Then I found out that isalpha() is not what is needed to generate the "word" count value for "wc".
So I did run this command to extract what Linux "wc" thinks is a word character:
Find the details on how isword.py is created from "out" in this posting.
Then I implemented upysh "wc", it imports "isword" module which takes 10KB ram, and needs 3KB in addition.
upysh.py as well as isword.py, as well as auxiliary files can be found here:
https://github.com/Hermann-SW/micropyth ... ster/upysh
Fork mission statement is here:
https://github.com/Hermann-SW/micropyth ... -statement
Here you can see that even on ESP8266 with only 26KB free ram wc() does work:
On ESP32 even bigger files can be processed with wc():
For comparison on Linux:
I went a step further and was able to implement isalpha() for whole unicode. Then I found out that isalpha() is not what is needed to generate the "word" count value for "wc".
So I did run this command to extract what Linux "wc" thinks is a word character:
Code: Select all
$ time ( for((i=0; i<17*65536; ++i)); do python3 -c "import sys;sys.stdout.write(chr($i))" | wc -w; done > out )
Then I implemented upysh "wc", it imports "isword" module which takes 10KB ram, and needs 3KB in addition.
upysh.py as well as isword.py, as well as auxiliary files can be found here:
https://github.com/Hermann-SW/micropyth ... ster/upysh
Fork mission statement is here:
https://github.com/Hermann-SW/micropyth ... -statement
Here you can see that even on ESP8266 with only 26KB free ram wc() does work:
Code: Select all
MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> from upysh_ import *
upysh is intended to be imported using:
...
>>> wc('upysh_.py')
123 281 2440 upysh_.py
>>>
Code: Select all
>>>
MicroPython v1.9.4-623-g34af10d2e on 2018-10-03; ESP32 module with ESP32
Type "help()" for more information.
>>> wc('isword.py')
64 895 10781 isword.py
>>>
Code: Select all
$ wc isword.py
64 895 10781 isword.py
$
Pico-W Access Point static file webserver:
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
Re: A simple shell for pyboard
Implementing wc is surely a great work, but I personally would not use it. I have upysh.py on every device imported by main.py, and what I need & use is ls, rm, rmdir, mkdir, cd, cat. df might be the only option missing. So the whole module takes like 2k of RAM, what I think is the amount I can miss, at least on devices w/o SPIRAM.
Re: A simple shell for pyboard
OK, so maybe 10KB is too nuch.
I thought about using a bytearray(768+76*32), that will cost 3200 bytes RAM only for the same isword information. The first 768 bytes have 0x00 or 0x01 for False and True, and 76 numbers 0x02-0x4D "pointing" to 32 byte portion offsets at 768+(n-2)*32. Will update when it is available on github.
I thought about using a bytearray(768+76*32), that will cost 3200 bytes RAM only for the same isword information. The first 768 bytes have 0x00 or 0x01 for False and True, and 76 numbers 0x02-0x4D "pointing" to 32 byte portion offsets at 768+(n-2)*32. Will update when it is available on github.
Pico-W Access Point static file webserver:
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
Re: A simple shell for pyboard
It can be done with far less than 3200 bytes.
Normally I use "uniq -c" on sorted files.
But it helps here immensely to identify the runs of consecutive 0s and 1s.
Starting with the file "out" from the other posting pointed to, the first 3 planes need to be handled:
The best is plane 2, very few "if"s needed to handle (as for planes e, f and 10):
The other two planes have more runs:
But even plane 0 allows to identify isword by binary search with at most 10 comparisons.
Since a plane consists of 2**16=65536 unicode characters the indices of 01 and 10 flips can be stord by UINT16 values.
Therefore bytearray((743+140)*2) space is needed only (1766 bytes, time vs. space).
I will have to learn how to address n UINT16 values in bytearray(2*n) ...
P.S:
Easier with array instead of bytearray:
Normally I use "uniq -c" on sorted files.
But it helps here immensely to identify the runs of consecutive 0s and 1s.
Starting with the file "out" from the other posting pointed to, the first 3 planes need to be handled:
Code: Select all
$ head -65536 out > 0
$ head -$((2*65536)) out | tail -65536 > 1
$ head -$((3*65536)) out | tail -65536 > 2
$
Code: Select all
$ uniq -c 2
42711 1
20777 0
542 1
1506 0
$
Code: Select all
$ uniq -c 0 | wc --lines
743
$ uniq -c 1 | wc --lines
140
$
Since a plane consists of 2**16=65536 unicode characters the indices of 01 and 10 flips can be stord by UINT16 values.
Therefore bytearray((743+140)*2) space is needed only (1766 bytes, time vs. space).
I will have to learn how to address n UINT16 values in bytearray(2*n) ...
P.S:
Easier with array instead of bytearray:
Code: Select all
MicroPython v1.9.4-623-g34af10d2e on 2018-10-03; ESP32 module with ESP32
Type "help()" for more information.
>>> gc.collect(); gc.mem_free()
100864
>>> import array
>>> a=array.array('H',bytearray(10000))
>>> gc.collect(); gc.mem_free()
90688
>>> len(a)
5000
>>>
Pico-W Access Point static file webserver:
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
Re: A simple shell for pyboard
It is 2464 bytes, and with below only 2928 bytes get added to that for adding tail() and wc().
I implemented the new binary search paradigm for isword.py:
viewtopic.php?f=2&t=5396&p=31563#p31563
It reduced RAM usage for isword.py from 10KB by more than 2/3rd to 3052 bytes!
Instead of just importing isword.py as before into upysh.py, this time I copied over the needed stuff only, not the test functions. This reduces upysh.py RAM usage by an additional 600 bytes. Instead of 13KB before upysh.py now uses 5392 bytes only(!), 2928 bytes more than the 2464 bytes for upysh.py without tail() and wc(). Available through this commit:
Code: Select all
28656 after soft reset
26192 old upysh
25888 + tail()
25072 + wc(), _w(), _s(), with _0 and _1 None
23264 + wc() complete, with _0 and _1 arrays (1808)
old: 2464
+tail(): +304 = 2768
+ wc(): +2624 = 5392
Code: Select all
$ webrepl_client.py -p abcd 192.168.4.1
...
MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> from upysh_ import *
...
>>> wc('upysh_.py')
161 1251 8715 upysh_.py
>>> exit
### closed ###
$ wc upysh.py
161 1251 8715 upysh.py
$
Pico-W Access Point static file webserver:
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
Re: A simple shell for pyboard
I could not resist and added copy command as well, at price of 136 bytes more RAM usage with this commit.
Then I thought about grep() command, and implemented as well for another 360 bytes more RAM usage with this commit, total upysh.py RAM usage now 5888 bytes. Currently only "i" and "v" option are implemented for grep, here are some samples:
Then I thought about grep() command, and implemented as well for another 360 bytes more RAM usage with this commit, total upysh.py RAM usage now 5888 bytes. Currently only "i" and "v" option are implemented for grep, here are some samples:
Code: Select all
>>> grep('', '\)$', 'tst.txt')
Fourth()
>>> grep('', '\)', 'tst.txt')
second().
Fourth()
>>> grep('', 't', 'tst.txt')
first
Fourth()
>>> grep('i', 't', 'tst.txt')
first
ThirD
Fourth()
>>> grep('iv', 'T', 'tst.txt')
second().
>>>
upysh commands:
pwd, cd("new_dir"), ls, ls(...), head(...), tail(...), wc(...), cat(...),
newfile(...), mv("old", "new"), cp("src", "tgt"), rm(...),
grep("opt", "regex", "file"), mkdir(...), rmdir(...), clear
Pico-W Access Point static file webserver:
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
Re: A simple shell for pyboard
There was one command missing I use often, octal dump command od("opt", "file") added with this commit.
I did reduce RAM usage by 32 bytes by previous commit, now increase by 512 bytes to 6368 bytes in total for upysh.py
od("", "file") is like "od -Ax -tx1":
od("c", "file") is like "od -Ax -tcx1":
I did reduce RAM usage by 32 bytes by previous commit, now increase by 512 bytes to 6368 bytes in total for upysh.py
od("", "file") is like "od -Ax -tx1":
Code: Select all
>>> od('','256.dat')
000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
000010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
000020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
000030 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f
000040 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f
000050 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f
000060 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f
000070 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f
000080 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f
000090 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f
0000a0 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af
0000b0 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf
0000c0 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf
0000d0 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df
0000e0 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef
0000f0 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff
000100
>>>
Code: Select all
>>> od('c','256.dat')
000000 \0 001 002 003 004 005 006 \a \b \t \n \v \f \r 016 017
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
000010 020 021 022 023 024 025 026 027 030 031 032 033 034 035 036 037
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
000020 ! " # $ % & ' ( ) * + , - . /
20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
000030 0 1 2 3 4 5 6 7 8 9 : ; < = > ?
30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f
000040 @ A B C D E F G H I J K L M N O
40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f
000050 P Q R S T U V W X Y Z [ \ ] ^ _
50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f
000060 ` a b c d e f g h i j k l m n o
60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f
000070 p q r s t u v w x y z { | } ~ 177
70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f
000080 200 201 202 203 204 205 206 207 210 211 212 213 214 215 216 217
80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f
000090 220 221 222 223 224 225 226 227 230 231 232 233 234 235 236 237
90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f
0000a0 240 241 242 243 244 245 246 247 250 251 252 253 254 255 256 257
a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af
0000b0 260 261 262 263 264 265 266 267 270 271 272 273 274 275 276 277
b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf
0000c0 300 301 302 303 304 305 306 307 310 311 312 313 314 315 316 317
c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf
0000d0 320 321 322 323 324 325 326 327 330 331 332 333 334 335 336 337
d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df
0000e0 340 341 342 343 344 345 346 347 350 351 352 353 354 355 356 357
e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef
0000f0 360 361 362 363 364 365 366 367 370 371 372 373 374 375 376 377
f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff
000100
>>>
Examples for all 5 new upysh commands as well as memory usage calculation for ESP8266 in fork-mission-statement.upysh commands:
pwd, cd("new_dir"), ls, ls(...), head(...), tail(...), wc(...), cat(...),
newfile(...), mv("old", "new"), cp("src", "tgt"), rm(...),
grep("opt", "regex", "file"), od("opt", "file"), mkdir(...), rmdir(...), clear
Pico-W Access Point static file webserver:
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell
https://github.com/Hermann-SW/pico-w
Tiny MicroPython robots (the PCB IS the robot platform)
viewtopic.php?f=5&t=11454
webrepl_client.py
https://github.com/Hermann-SW/webrepl#webrepl-shell