A simple shell for pyboard

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
HermannSW
Posts: 138
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Wed Nov 07, 2018 11:35 pm

From current help():

Code: Select all

...
upysh commands:
pwd, cd("new_dir"), ls, ls(...), head(...), tail(...), wc(...), cat(...),
newfile(...), mv("old", "new"), cp("src", "tgt"), rm(...), clear
grep("file", "regex" [, "opt"]), od("file" [, "opt"]), mkdir(...), rmdir(...)
...
Analyzing which commands can be pipe generator/consumer (of generator|consumer) gives:

Code: Select all

          none:  pwd, cd(), newfile(), mv(), rm(), clear, mkdir(), rmdir()
generator only:  man, ls, ls()[, pipe(filename)]
          both:  head(), tail(), cat(), grep(), od()
 consumer only:  wc(), cp()[, done]
Commands missing that could be useful (all for "both"): cut() -b, sort() -r/-n, uniq() -c/-u

HermannSW
Posts: 138
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Thu Nov 08, 2018 12:17 am

After previous removal of lambda pipeing code, the previous use of "cp(lambda: head('tst.txt',3), 'x')" as "head -3 tst.txt > x" is not possible anymore. Pipeing cp() now leaves a Pipe_ class object:

Code: Select all

>>> pipe("tst.txt") | (cp,"x") 
<_Pipe object at 3fff1ae0>
>>> 
Therefore cp() has been changed (in streaming mode) to be the "tee" command:

Code: Select all

>>> pipe("tst.txt") | (head,3) | (cp,"3.txt") | (tail,2) | done
second().
ThirD
>>> cat("3.txt")
first
second().
ThirD
>>> 
This is current cp():

Code: Select all

def cp(s, t):
    S = _openlambda(s)
    with open(t, "w") as t:
        while True:
            l = S.readline()
            if not l: break
            t.write(l)

    if type(s) != str:
        S.seek(0,0)
        return S

HermannSW
Posts: 138
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Thu Nov 08, 2018 12:48 am

I remembered the usecase of "micropython.qstr_info()" output being processed by grep() [or sort(), uniq()].
I had to bring in back lambda processing, but this time only in pipe() arg:

Code: Select all

...
grep("file", "regex" [, "opt"]), od("file" [, "opt"]), mkdir(...), rmdir(...)

>>> import micropython
>>> pipe(lambda: micropython.qstr_info(1)) | (head,5) | done
qstr pool: n_pool=1, n_qstr=65, n_str_data_bytes=494, n_total_bytes=2110
Q(webrepl_cfg.py)
Q(abcd)
Q(upysh_)
Q(upysh_.py)
>>> micropython.qstr_info()
qstr pool: n_pool=1, n_qstr=65, n_str_data_bytes=494, n_total_bytes=2110
>>> 
In class _Pipe(object):

Code: Select all

    def __init__(self, val):
        if type(val) == type(lambda: x):
            s = bytearray()
            prev = os.dupterm(_DUP(s))
            val()
            os.dupterm(prev)
            self.val = io.StringIO(s)
        else:
            self.val = open(val)
P.S:
This lambda processing alllows to grep "ls" and "man" command output:

Code: Select all

>>> pipe(lambda: print(man)) | (grep,"grep") | done
upysh commands head/cat/tail/wc/cp/grep/od allow for lambda pipeing:
grep("file", "regex" [, "opt"]), od("file" [, "opt"]), mkdir(...), rmdir(...)
>>> 
>>> pipe(lambda: print(ls)) | (grep,"\d\d\d\d\d") | done
   12353 upysh_.py
>>> pipe(lambda: print(ls)) | (grep,"^   \d") | done
   12353 upysh_.py
>>> 

HermannSW
Posts: 138
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Thu Nov 08, 2018 1:52 am

I solved the out of memory error on import into ESP8266(ESP-01s) MicroPython which became real again after latest additions. It seems that doing lazy import of normally unneeded stuff is the solution to help MicroPython being able to import. The big arrays _0 and _1 are only needed when wc() command gets executed, and splitting them away into module "wc.py" resolves the issue completely!

Code: Select all

MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> m0=gc.mem_free()
>>> from upysh_ import *
...
>>> pwd
/
>>> m0, gc.mem_free()
(26464, 22192)
>>> wc("tst.txt")
4 4 33 tst.txt
>>> gc.mem_free()
19520
>>> 
This is split module:

Code: Select all

$ cat wc.py 
import array
a0=array.array('H', [32, 126, 159, 887, ...
a1=array.array('H', [11, 12, 38, 39, 58, ...
$ 
And this is the wc() code initializing the two global variables _0 and _1 that have values None before executing wc() for the first time:

Code: Select all

def wc(fn):
    global _0
    global _1
    if _0 == None:
        import wc
        _0 = wc.a0
        _1 = wc.a1

    out = sys.stdout if type(fn) == str else io.StringIO(bytearray())

    with _openlambda(fn) as f:
        c=r=w=0
        while True:
            l = f.readline()
            if not l: break
            r=r+1
            c=c+len(bytes(l,'utf-8'))
            w=w+words(l)
    out.write("{0:d} {1:d} {2:d} {3:s}\n".format(r,w,c,fn))

    if out != sys.stdout:
        out.seek(0,0)
        return out
P.S:
upysh.py runs fine with "from upysh.py import *" in python3.
Use of "io.StringIO(bytearray())" results in this error message in python3:

Code: Select all

TypeError: argument should be string, bytes or integer, not _io.StringIO
I replaced all occurences by io.StringIO('') and now it works in MicroPython as well as in python3.
Only upysh.py feature I found sofar that does not run in python3 is pipe() lambda evaluation, because os.dupterm() is unknown.

HermannSW
Posts: 138
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Sun Nov 11, 2018 9:33 pm

I completed the pipeing work for all commands affected, and updated man as well as README.txt fork-mission-statement:
https://github.com/Hermann-SW/micropyth ... -statement

Now RAM usage is 6352 byte as long as "wc()" command does not get used. That command does lazy import of split out module "words.py". On ESP-01s even that runs out of memory because temporarily the import does need too much RAM. For such systems preloading words.py before importing upysh_ works. Both modules together need 8880 bytes of RAM:

Code: Select all

...
MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> gc.collect(); m0=gc.mem_free()
>>> import words
>>> from upysh_ import *
>>> gc.collect(); m1=gc.mem_free()
>>> print(m0, m1, m0-m1)
28656 19776 8880
>>> 
...
"tee()" command has been added in addtion to "cp()" command.
New pipeing based on parts of pipeto 0.2.1, with added multi argument processing, is available.
From upysh_ man:

Code: Select all

...
Most upysh_ commands allow for "producer|consumer" pipeing:
  >>> pipe("tst.txt") | (head,3) | (tee,"3.txt") | (grep,'t','i') | done
  first
  ThirD
  >>> pipe(lambda: micropython.qstr_info(1)) | (head,2) | done
  qstr pool: n_pool=1, n_qstr=69, n_str_data_bytes=525, n_total_bytes=2141
  Q(webrepl_cfg.py)
  >>>

producer only:  man, ls, ls()[, pipe()]
         both:  head(), tail(), cat(), grep(), od(), tee()[, wc(), cp()]
consumer only:  [done]
...
You can install "normal" upysh and upysh_ with words.py on a module. If ever memory consumption of upysh_ becomes a problem, just do CTRL-D soft reset and use upysh.

Have fun with upysh_ pipeing ...

Code: Select all

>>> pipe(lambda: ls()) | (grep,'^    [0-9]') | done
    6633 upysh_.py
    5543 wc.py
    6464 words.py
>>>

HermannSW
Posts: 138
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: A simple shell for pyboard

Post by HermannSW » Tue Nov 13, 2018 8:02 pm

I went a step further, not more features for upysh_ yet, but being able to execute a single MicroPython command remotely for being able to use all the Linux bash shell power for working on single command MicroPython output.

Here is a first simple sample, just demonstrating what it is:

Code: Select all

$ ./micropython mp.py 192.168.4.1 5**4**3
542101086242752217003726400434970855712890625
$ 
In this posting I did describe how to remote control MicroPython from another MicroPython, using Danni's uwebsocket module. I tried to make use of that with python3, and after many changes to get uwebsocket module (because that is much easier to use than my webrepl_client.py python[23] remote console) to run with python 3 I ended up with this python3 error on difference to MicroPython:

Code: Select all

$ python3 echo_websocket_org.py 
Traceback (most recent call last):
  File "echo_websocket_org.py", line 2, in <module>
    websocket = client.connect("ws://echo.websocket.org/")
  File "/home/stammw/uwebsockets/client.py", line 59, in connect
    header = sock.readline()[:-2]
AttributeError: 'socket' object has no attribute 'readline'
$ 

I gave up there and remembered on the unix port of MicroPython. With it uwebsockets module new echo_websockets_org.py example directly worked:

Code: Select all

$ ./micropython 
MicroPython v1.9.4-683-gd94aa57 on 2018-11-13; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import echo_websocket_org
The quick brown fox jumps over the lazy dog

>>> 

At the start of this posting I already showed mp.py in action. It is running in unix port MicroPython, uses uwebsockets module to connect to ESP-01s WebREPL with IP address 192.168.4.1 and executed the passed command. Before I show the simple 25 line only mp.py, I will show the intended application (of mixing Linux bash, unix port MicroPython and ESP8266 MicroPython alltogether).

First my upysh_ module needs to be imported, after the ESP-01s was powered on (the quotes around star character are needed to prevent bash from expanding it):

Code: Select all

$ ./micropython mp.py 192.168.4.1 from upysh_ import '*'

upysh_ is intended to be imported using:
[import words  (on systems with very small RAM)]
from upysh_ import *
...
grep(file, regex [, opt]), od(file [, opt]), mkdir(...), rmdir(...)

$ 

Now that upysh_ module is loaded, its "ls" command can get executed on ESP-01s (that sorts files alphabetically), and Linux bash can be used to sort by size. This should just be a simple example:

Code: Select all

$ ./micropython mp.py 192.168.4.1 ls | sort -n

       0 empty
      11 umlaute.txt
      14 webrepl_cfg.py
      23 rn.txt
      29 numstr.txt
      33 tst.txt
      60 tloop.py
     361 boot.py
     384 256.dat
    1571 upysh.py
    6464 words.py
    6633 upysh_.py
$ 

Now that you have seen how to use mp.py in unix port MicroPython, here it is:

Code: Select all

import uwebsockets.client
import sys

websocket = uwebsockets.client.connect('ws://'+sys.argv[1]+':8266/')

def do(cmd,out=True):
    inp = cmd+"\r\n"
    websocket.send(inp)

    resp = ""
    while not("\n>>> " in resp or resp.startswith(">>> ")):
        resp = websocket.recv()
        while inp != "" and resp != "" and inp[0] == resp[0]:
            inp = inp[1:]
            resp = resp[1:]
        if out and not("\n>>> " in resp or resp.startswith(">>> ")):
            print(resp,end='')


resp = websocket.recv()
assert resp == "Password: ", resp

do("abcd",False)

do(" ".join(sys.argv[2:]))

So for doing a MicroPython session I will use upysh_ on the module itself, in a webrepl_client.py remote session.
And if more than the features and commands implemented in upysh_ is needed, I will use mp.py and Linux bash.

P.S: P.P.S:
This is mixed pipeing example. The mp.py command needs to be quoted for dividing the vertical bars between MicroPython upysh_ processing and Linux bash shell:

Code: Select all

$ ./micropython mp.py 192.168.4.1 'pipe(lambda: micropython.qstr_info(1)) | (tee,"qstr.txt") | done' | sort | tail -6
Q(v)
Q(wc)
Q(webrepl_cfg.py)
Q(wloaded)
Q(words)
Q(x)
$ 
If teeing output into qstr.txt is not needed, the command line looks much easier:

Code: Select all

$ ./micropython mp.py 192.168.4.1 'micropython.qstr_info(1)' | sort | tail -6
Q(v)
Q(wc)
Q(webrepl_cfg.py)
Q(wloaded)
Q(words)
Q(x)
$ 
P.P.P.S:
For another application of mp.py see this posting (1000s of calls):
viewtopic.php?f=2&p=31863#p31863

Post Reply