A simple shell for pyboard

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
keithostertag
Posts: 9
Joined: Sun Dec 17, 2017 3:46 pm

Re: A simple shell for pyboard

Post by keithostertag » Tue Feb 20, 2018 12:46 am

Thanks Robert and Dave!

I'm a beginner, so it may or may not be "easy" for me to write a cp method, but yea, you've both given me a bunch of examples to work from (and encouragement!) so I might give it a try...

Yes, using cat with redirection will work...

Thanks much!
Keith

HermannSW
Posts: 144
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Wed Oct 17, 2018 8:31 am

upysh is really cool, I have placed it on all my MicroPython modules, and do "from upysh import *" when needed.
dhylands wrote:
Tue Feb 20, 2018 12:15 am
PRs welcome
I did add "tail" command, not sure whether I should raise PR.
Two changelists, this is the last one fixing n<=0 corner cases.

I want to add "wc" as well. Counting lines and bytes is easy, but for counting words as Linux "wc" does an "isalpha()" is needed that works for complete unicode character set [there are 99537 unicode characters for which isalpha() returns True]. But MicroPython "isalpha()" only works for ASCII characters:
viewtopic.php?f=2&t=5396

Last, but not least, this is "tail()":

Code: Select all

def tail(f, n=10):
    with open(f) as f:
        if n<=0: return
        a = [ "" for i in range(n) ]
        i = 0
        while True:
            l = f.readline()
            if not l: break
            a[i % n] = l
            i += 1
        if i>0 and i<n:
            for j in range(i+1):
                sys.stdout.write(a[j])
        else:
            for j in range(n):
                sys.stdout.write(a[(i+j)%n])

User avatar
dhylands
Posts: 3382
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: A simple shell for pyboard

Post by dhylands » Wed Oct 17, 2018 2:35 pm

PRs are definitely the preferred approach.

Note that upysh (from Micropython-lib) was written by pfalcon. The simple shell that I wrote is found here: https://github.com/dhylands/upy-shell

I find I tend to use rshell https://github.com/dhylands/rshell now rather than upy-shell

HermannSW
Posts: 144
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Wed Oct 24, 2018 4:16 pm

Thanks.

I went a step further and was able to implement isalpha() for whole unicode. Then I found out that isalpha() is not what is needed to generate the "word" count value for "wc".

So I did run this command to extract what Linux "wc" thinks is a word character:

Code: Select all

$ time ( for((i=0; i<17*65536; ++i)); do python3 -c "import sys;sys.stdout.write(chr($i))" | wc -w; done > out )
Find the details on how isword.py is created from "out" in this posting.

Then I implemented upysh "wc", it imports "isword" module which takes 10KB ram, and needs 3KB in addition.

upysh.py as well as isword.py, as well as auxiliary files can be found here:
https://github.com/Hermann-SW/micropyth ... ster/upysh

Fork mission statement is here:
https://github.com/Hermann-SW/micropyth ... -statement

Here you can see that even on ESP8266 with only 26KB free ram wc() does work:

Code: Select all

MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> from upysh_ import *

upysh is intended to be imported using:
...
>>> wc('upysh_.py')
123 281 2440 upysh_.py
>>> 
On ESP32 even bigger files can be processed with wc():

Code: Select all

>>> 
MicroPython v1.9.4-623-g34af10d2e on 2018-10-03; ESP32 module with ESP32
Type "help()" for more information.

>>> wc('isword.py')
64 895 10781 isword.py
>>> 
For comparison on Linux:

Code: Select all

$ wc isword.py 
   64   895 10781 isword.py
$ 

User avatar
Roberthh
Posts: 1996
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: A simple shell for pyboard

Post by Roberthh » Wed Oct 24, 2018 6:48 pm

Implementing wc is surely a great work, but I personally would not use it. I have upysh.py on every device imported by main.py, and what I need & use is ls, rm, rmdir, mkdir, cd, cat. df might be the only option missing. So the whole module takes like 2k of RAM, what I think is the amount I can miss, at least on devices w/o SPIRAM.

HermannSW
Posts: 144
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Thu Oct 25, 2018 11:55 pm

OK, so maybe 10KB is too nuch.
I thought about using a bytearray(768+76*32), that will cost 3200 bytes RAM only for the same isword information. The first 768 bytes have 0x00 or 0x01 for False and True, and 76 numbers 0x02-0x4D "pointing" to 32 byte portion offsets at 768+(n-2)*32. Will update when it is available on github.

HermannSW
Posts: 144
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Fri Oct 26, 2018 9:32 pm

It can be done with far less than 3200 bytes.

Normally I use "uniq -c" on sorted files.
But it helps here immensely to identify the runs of consecutive 0s and 1s.
Starting with the file "out" from the other posting pointed to, the first 3 planes need to be handled:

Code: Select all

$ head -65536 out > 0
$ head -$((2*65536)) out | tail -65536 > 1
$ head -$((3*65536)) out | tail -65536 > 2
$ 
The best is plane 2, very few "if"s needed to handle (as for planes e, f and 10):

Code: Select all

$ uniq -c 2 
  42711 1
  20777 0
    542 1
   1506 0
$
The other two planes have more runs:

Code: Select all

$ uniq -c 0 | wc --lines
743
$ uniq -c 1 | wc --lines
140
$ 
But even plane 0 allows to identify isword by binary search with at most 10 comparisons.

Since a plane consists of 2**16=65536 unicode characters the indices of 01 and 10 flips can be stord by UINT16 values.
Therefore bytearray((743+140)*2) space is needed only (1766 bytes, time vs. space).

I will have to learn how to address n UINT16 values in bytearray(2*n) ...

P.S:
Easier with array instead of bytearray:

Code: Select all

MicroPython v1.9.4-623-g34af10d2e on 2018-10-03; ESP32 module with ESP32
Type "help()" for more information.
>>> gc.collect(); gc.mem_free()
100864
>>> import array
>>> a=array.array('H',bytearray(10000))
>>> gc.collect(); gc.mem_free()
90688
>>> len(a)
5000
>>> 

HermannSW
Posts: 144
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Wed Oct 31, 2018 6:00 pm

Roberthh wrote:
Wed Oct 24, 2018 6:48 pm
So the whole module takes like 2k of RAM, what I think is the amount I can miss, at least on devices w/o SPIRAM.
It is 2464 bytes, and with below only 2928 bytes get added to that for adding tail() and wc().

I implemented the new binary search paradigm for isword.py:
viewtopic.php?f=2&t=5396&p=31563#p31563

It reduced RAM usage for isword.py from 10KB by more than 2/3rd to 3052 bytes!

Instead of just importing isword.py as before into upysh.py, this time I copied over the needed stuff only, not the test functions. This reduces upysh.py RAM usage by an additional 600 bytes. Instead of 13KB before upysh.py now uses 5392 bytes only(!), 2928 bytes more than the 2464 bytes for upysh.py without tail() and wc(). Available through this commit:

Code: Select all

28656 after soft reset

26192 old upysh
25888 + tail()
25072 + wc(), _w(), _s(), with _0 and _1 None
23264 + wc() complete, with _0 and _1 arrays (1808)

    old:         2464
+tail():  +304 = 2768
+  wc(): +2624 = 5392
With new implementation even ESP8266 can do wc() for bigger files:

Code: Select all

$ webrepl_client.py -p abcd 192.168.4.1
...
MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> from upysh_ import *
...
>>> wc('upysh_.py')
161 1251 8715 upysh_.py
>>> exit
### closed ###
$ wc upysh.py
 161 1251 8715 upysh.py
$ 

HermannSW
Posts: 144
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Wed Oct 31, 2018 9:21 pm

I could not resist and added copy command as well, at price of 136 bytes more RAM usage with this commit.

Then I thought about grep() command, and implemented as well for another 360 bytes more RAM usage with this commit, total upysh.py RAM usage now 5888 bytes. Currently only "i" and "v" option are implemented for grep, here are some samples:

Code: Select all

>>> grep('', '\)$', 'tst.txt')
  Fourth()
>>> grep('', '\)', 'tst.txt')
second().
  Fourth()
>>> grep('', 't', 'tst.txt')
first
  Fourth()
>>> grep('i', 't', 'tst.txt')
first
ThirD
  Fourth()
>>> grep('iv', 'T', 'tst.txt')
second().
>>>

upysh commands:
pwd, cd("new_dir"), ls, ls(...), head(...), tail(...), wc(...), cat(...),
newfile(...), mv("old", "new"), cp("src", "tgt"), rm(...),
grep("opt", "regex", "file"), mkdir(...), rmdir(...), clear

HermannSW
Posts: 144
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Thu Nov 01, 2018 1:17 am

There was one command missing I use often, octal dump command od("opt", "file") added with this commit.
I did reduce RAM usage by 32 bytes by previous commit, now increase by 512 bytes to 6368 bytes in total for upysh.py

od("", "file") is like "od -Ax -tx1":

Code: Select all

>>> od('','256.dat')
000000  00  01  02  03  04  05  06  07  08  09  0a  0b  0c  0d  0e  0f
000010  10  11  12  13  14  15  16  17  18  19  1a  1b  1c  1d  1e  1f
000020  20  21  22  23  24  25  26  27  28  29  2a  2b  2c  2d  2e  2f
000030  30  31  32  33  34  35  36  37  38  39  3a  3b  3c  3d  3e  3f
000040  40  41  42  43  44  45  46  47  48  49  4a  4b  4c  4d  4e  4f
000050  50  51  52  53  54  55  56  57  58  59  5a  5b  5c  5d  5e  5f
000060  60  61  62  63  64  65  66  67  68  69  6a  6b  6c  6d  6e  6f
000070  70  71  72  73  74  75  76  77  78  79  7a  7b  7c  7d  7e  7f
000080  80  81  82  83  84  85  86  87  88  89  8a  8b  8c  8d  8e  8f
000090  90  91  92  93  94  95  96  97  98  99  9a  9b  9c  9d  9e  9f
0000a0  a0  a1  a2  a3  a4  a5  a6  a7  a8  a9  aa  ab  ac  ad  ae  af
0000b0  b0  b1  b2  b3  b4  b5  b6  b7  b8  b9  ba  bb  bc  bd  be  bf
0000c0  c0  c1  c2  c3  c4  c5  c6  c7  c8  c9  ca  cb  cc  cd  ce  cf
0000d0  d0  d1  d2  d3  d4  d5  d6  d7  d8  d9  da  db  dc  dd  de  df
0000e0  e0  e1  e2  e3  e4  e5  e6  e7  e8  e9  ea  eb  ec  ed  ee  ef
0000f0  f0  f1  f2  f3  f4  f5  f6  f7  f8  f9  fa  fb  fc  fd  fe  ff
000100
>>> 
od("c", "file") is like "od -Ax -tcx1":

Code: Select all

>>> od('c','256.dat')
000000  \0 001 002 003 004 005 006  \a  \b  \t  \n  \v  \f  \r 016 017
        00  01  02  03  04  05  06  07  08  09  0a  0b  0c  0d  0e  0f
000010 020 021 022 023 024 025 026 027 030 031 032 033 034 035 036 037
        10  11  12  13  14  15  16  17  18  19  1a  1b  1c  1d  1e  1f
000020       !   "   #   $   %   &   '   (   )   *   +   ,   -   .   /
        20  21  22  23  24  25  26  27  28  29  2a  2b  2c  2d  2e  2f
000030   0   1   2   3   4   5   6   7   8   9   :   ;   <   =   >   ?
        30  31  32  33  34  35  36  37  38  39  3a  3b  3c  3d  3e  3f
000040   @   A   B   C   D   E   F   G   H   I   J   K   L   M   N   O
        40  41  42  43  44  45  46  47  48  49  4a  4b  4c  4d  4e  4f
000050   P   Q   R   S   T   U   V   W   X   Y   Z   [   \   ]   ^   _
        50  51  52  53  54  55  56  57  58  59  5a  5b  5c  5d  5e  5f
000060   `   a   b   c   d   e   f   g   h   i   j   k   l   m   n   o
        60  61  62  63  64  65  66  67  68  69  6a  6b  6c  6d  6e  6f
000070   p   q   r   s   t   u   v   w   x   y   z   {   |   }   ~ 177
        70  71  72  73  74  75  76  77  78  79  7a  7b  7c  7d  7e  7f
000080 200 201 202 203 204 205 206 207 210 211 212 213 214 215 216 217
        80  81  82  83  84  85  86  87  88  89  8a  8b  8c  8d  8e  8f
000090 220 221 222 223 224 225 226 227 230 231 232 233 234 235 236 237
        90  91  92  93  94  95  96  97  98  99  9a  9b  9c  9d  9e  9f
0000a0 240 241 242 243 244 245 246 247 250 251 252 253 254 255 256 257
        a0  a1  a2  a3  a4  a5  a6  a7  a8  a9  aa  ab  ac  ad  ae  af
0000b0 260 261 262 263 264 265 266 267 270 271 272 273 274 275 276 277
        b0  b1  b2  b3  b4  b5  b6  b7  b8  b9  ba  bb  bc  bd  be  bf
0000c0 300 301 302 303 304 305 306 307 310 311 312 313 314 315 316 317
        c0  c1  c2  c3  c4  c5  c6  c7  c8  c9  ca  cb  cc  cd  ce  cf
0000d0 320 321 322 323 324 325 326 327 330 331 332 333 334 335 336 337
        d0  d1  d2  d3  d4  d5  d6  d7  d8  d9  da  db  dc  dd  de  df
0000e0 340 341 342 343 344 345 346 347 350 351 352 353 354 355 356 357
        e0  e1  e2  e3  e4  e5  e6  e7  e8  e9  ea  eb  ec  ed  ee  ef
0000f0 360 361 362 363 364 365 366 367 370 371 372 373 374 375 376 377
        f0  f1  f2  f3  f4  f5  f6  f7  f8  f9  fa  fb  fc  fd  fe  ff
000100
>>> 
upysh commands:
pwd, cd("new_dir"), ls, ls(...), head(...), tail(...), wc(...), cat(...),
newfile(...), mv("old", "new"), cp("src", "tgt"), rm(...),
grep("opt", "regex", "file"), od("opt", "file"), mkdir(...), rmdir(...), clear
Examples for all 5 new upysh commands as well as memory usage calculation for ESP8266 in fork-mission-statement.

Post Reply