A simple shell for pyboard

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
HermannSW
Posts: 99
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Wed Nov 07, 2018 11:35 pm

From current help():

Code: Select all

...
upysh commands:
pwd, cd("new_dir"), ls, ls(...), head(...), tail(...), wc(...), cat(...),
newfile(...), mv("old", "new"), cp("src", "tgt"), rm(...), clear
grep("file", "regex" [, "opt"]), od("file" [, "opt"]), mkdir(...), rmdir(...)
...
Analyzing which commands can be pipe generator/consumer (of generator|consumer) gives:

Code: Select all

          none:  pwd, cd(), newfile(), mv(), rm(), clear, mkdir(), rmdir()
generator only:  man, ls, ls()[, pipe(filename)]
          both:  head(), tail(), cat(), grep(), od()
 consumer only:  wc(), cp()[, done]
Commands missing that could be useful (all for "both"): cut() -b, sort() -r/-n, uniq() -c/-u

HermannSW
Posts: 99
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Thu Nov 08, 2018 12:17 am

After previous removal of lambda pipeing code, the previous use of "cp(lambda: head('tst.txt',3), 'x')" as "head -3 tst.txt > x" is not possible anymore. Pipeing cp() now leaves a Pipe_ class object:

Code: Select all

>>> pipe("tst.txt") | (cp,"x") 
<_Pipe object at 3fff1ae0>
>>> 
Therefore cp() has been changed (in streaming mode) to be the "tee" command:

Code: Select all

>>> pipe("tst.txt") | (head,3) | (cp,"3.txt") | (tail,2) | done
second().
ThirD
>>> cat("3.txt")
first
second().
ThirD
>>> 
This is current cp():

Code: Select all

def cp(s, t):
    S = _openlambda(s)
    with open(t, "w") as t:
        while True:
            l = S.readline()
            if not l: break
            t.write(l)

    if type(s) != str:
        S.seek(0,0)
        return S

HermannSW
Posts: 99
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Thu Nov 08, 2018 12:48 am

I remembered the usecase of "micropython.qstr_info()" output being processed by grep() [or sort(), uniq()].
I had to bring in back lambda processing, but this time only in pipe() arg:

Code: Select all

...
grep("file", "regex" [, "opt"]), od("file" [, "opt"]), mkdir(...), rmdir(...)

>>> import micropython
>>> pipe(lambda: micropython.qstr_info(1)) | (head,5) | done
qstr pool: n_pool=1, n_qstr=65, n_str_data_bytes=494, n_total_bytes=2110
Q(webrepl_cfg.py)
Q(abcd)
Q(upysh_)
Q(upysh_.py)
>>> micropython.qstr_info()
qstr pool: n_pool=1, n_qstr=65, n_str_data_bytes=494, n_total_bytes=2110
>>> 
In class _Pipe(object):

Code: Select all

    def __init__(self, val):
        if type(val) == type(lambda: x):
            s = bytearray()
            prev = os.dupterm(_DUP(s))
            val()
            os.dupterm(prev)
            self.val = io.StringIO(s)
        else:
            self.val = open(val)
P.S:
This lambda processing alllows to grep "ls" and "man" command output:

Code: Select all

>>> pipe(lambda: print(man)) | (grep,"grep") | done
upysh commands head/cat/tail/wc/cp/grep/od allow for lambda pipeing:
grep("file", "regex" [, "opt"]), od("file" [, "opt"]), mkdir(...), rmdir(...)
>>> 
>>> pipe(lambda: print(ls)) | (grep,"\d\d\d\d\d") | done
   12353 upysh_.py
>>> pipe(lambda: print(ls)) | (grep,"^   \d") | done
   12353 upysh_.py
>>> 

HermannSW
Posts: 99
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Thu Nov 08, 2018 1:52 am

I solved the out of memory error on import into ESP8266(ESP-01s) MicroPython which became real again after latest additions. It seems that doing lazy import of normally unneeded stuff is the solution to help MicroPython being able to import. The big arrays _0 and _1 are only needed when wc() command gets executed, and splitting them away into module "wc.py" resolves the issue completely!

Code: Select all

MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> m0=gc.mem_free()
>>> from upysh_ import *
...
>>> pwd
/
>>> m0, gc.mem_free()
(26464, 22192)
>>> wc("tst.txt")
4 4 33 tst.txt
>>> gc.mem_free()
19520
>>> 
This is split module:

Code: Select all

$ cat wc.py 
import array
a0=array.array('H', [32, 126, 159, 887, ...
a1=array.array('H', [11, 12, 38, 39, 58, ...
$ 
And this is the wc() code initializing the two global variables _0 and _1 that have values None before executing wc() for the first time:

Code: Select all

def wc(fn):
    global _0
    global _1
    if _0 == None:
        import wc
        _0 = wc.a0
        _1 = wc.a1

    out = sys.stdout if type(fn) == str else io.StringIO(bytearray())

    with _openlambda(fn) as f:
        c=r=w=0
        while True:
            l = f.readline()
            if not l: break
            r=r+1
            c=c+len(bytes(l,'utf-8'))
            w=w+words(l)
    out.write("{0:d} {1:d} {2:d} {3:s}\n".format(r,w,c,fn))

    if out != sys.stdout:
        out.seek(0,0)
        return out
P.S:
upysh.py runs fine with "from upysh.py import *" in python3.
Use of "io.StringIO(bytearray())" results in this error message in python3:

Code: Select all

TypeError: argument should be string, bytes or integer, not _io.StringIO
I replaced all occurences by io.StringIO('') and now it works in MicroPython as well as in python3.
Only upysh.py feature I found sofar that does not run in python3 is pipe() lambda evaluation, because os.dupterm() is unknown.

HermannSW
Posts: 99
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Sun Nov 11, 2018 9:33 pm

I completed the pipeing work for all commands affected, and updated man as well as README.txt fork-mission-statement:
https://github.com/Hermann-SW/micropyth ... -statement

Now RAM usage is 6352 byte as long as "wc()" command does not get used. That command does lazy import of split out module "words.py". On ESP-01s even that runs out of memory because temporarily the import does need too much RAM. For such systems preloading words.py before importing upysh_ works. Both modules together need 8880 bytes of RAM:

Code: Select all

...
MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> gc.collect(); m0=gc.mem_free()
>>> import words
>>> from upysh_ import *
>>> gc.collect(); m1=gc.mem_free()
>>> print(m0, m1, m0-m1)
28656 19776 8880
>>> 
...
"tee()" command has been added in addtion to "cp()" command.
New pipeing based on parts of pipeto 0.2.1, with added multi argument processing, is available.
From upysh_ man:

Code: Select all

...
Most upysh_ commands allow for "producer|consumer" pipeing:
  >>> pipe("tst.txt") | (head,3) | (tee,"3.txt") | (grep,'t','i') | done
  first
  ThirD
  >>> pipe(lambda: micropython.qstr_info(1)) | (head,2) | done
  qstr pool: n_pool=1, n_qstr=69, n_str_data_bytes=525, n_total_bytes=2141
  Q(webrepl_cfg.py)
  >>>

producer only:  man, ls, ls()[, pipe()]
         both:  head(), tail(), cat(), grep(), od(), tee()[, wc(), cp()]
consumer only:  [done]
...
You can install "normal" upysh and upysh_ with words.py on a module. If ever memory consumption of upysh_ becomes a problem, just do CTRL-D soft reset and use upysh.

Have fun with upysh_ pipeing ...

Code: Select all

>>> pipe(lambda: ls()) | (grep,'^    [0-9]') | done
    6633 upysh_.py
    5543 wc.py
    6464 words.py
>>>

Post Reply