A simple shell for pyboard

[HermannSW]

Oh wow!

I knew already that MicroPython is cool ...

Today I learned how to redirect stdout to a bytearray in this posting:
viewtopic.php?f=2&t=5446&p=31594#p31594

I wanted to use that for *nix like command line pipe operators between more than one upysh commands.

After I learned how to use StringIO, that it is available in MicroPython and that initialization with bytearray is possible all looked good.

Then I had to learn how to prevent a function with args to run when passed as arg to another function, and learned about lambda.

I only enabled head() for pipeing, just as a proof of concept -- still unbelievable to me how easy this can be done:

Code: Select all

>>> cat("tst.txt")
first
second().
ThirD
  Fourth()
>>> head(lambda: tail("tst.txt", 3), 2)
second().
ThirD
>>> head("tst.txt", 2)
first
second().
>>> 

I used the DUP class from the other posting, and stuffed all else into new function openlambda:

• returns StringIO containing function output in case of lambda passed
• returns open(f) in case of filename passed

I really like this, this is the small diff, here the DIP class from the other posting:

Code: Select all

$ diff upysh.py upyshlambda.py 
139a140,165
> 
> from io import IOBase
> from io import StringIO
> 
> class DUP(IOBase):
> 
>     def __init__(self, s):
>         self.s = s
> 
>     def write(self, data):
>         self.s += data
>         return len(data)
> 
>     def readinto(self, data):
>         return 0
> 
> def openlambda(f)
...

And this is openlambda():

Code: Select all

...
> def openlambda(f):
>     if type(f) == type(lambda: x):
>         s = bytearray()
>         prev = os.dupterm(DUP(s))
>         f()
>         os.dupterm(prev)
>         return StringIO(s)
>     else:
>         return open(f)
> 
141c167
<     with open(f) as f:
---
>     with openlambda(f) as f:
$ 

P.S:
Originally i wanted to enable pipeing for head(), tail(), wc(), grep(), od() and perhaps cat().
Just realized that "cp(lambda: tail('tst.txt', 3), 'newfile')" would make sense as well for "tail -3 tst.txt > newfile" ...

[HermannSW]

Its done, with commit 4abab1e lambda pipeing has been added to 7 upysh commands:

Code: Select all

>>> man

upysh is intended to be imported using:
from upysh_ import *

To see this help text again, type "man".

upysh commands head/cat/tail/wc/cp/grep/od allow for lambda pipeing:
  >>> tail(lambda: head('tst.txt', 3), 2)
  second().
  ThirD
  >>>  

upysh commands:
pwd, cd("new_dir"), ls, ls(...), head(...), tail(...), wc(...), cat(...),
newfile(...), mv("old", "new"), cp("src", "tgt"), rm(...),
grep("opt", "regex", "file"), od("opt", "file"), mkdir(...), rmdir(...), clear

>>> 

upysh RAM usage increased by another 512 bytes to 6880, but it is definitely worth it.

Find more pipeing examples in fork-mission-statement. I only want to show this one from there:

od() allows to debug arbitrary stuff. With it issue #4285 was found and reported:

Code: Select all

>>> od('c', lambda: sys.stdout.write('abc'))
000000   a   b   c
        61  62  63
000003
>>> 

P.S:
Pipeing grep is helpful in looking into interned string usage:

Code: Select all

>>> micropython.qstr_info()
qstr pool: n_pool=1, n_qstr=61, n_str_data_bytes=461, n_total_bytes=2077
>>> grep('', 'tst', lambda: micropython.qstr_info(1))
Q(tst.txt)
Q(tst)
>>> 

[pfalcon]

Cool, but not nearly as cool as https://pypi.org/project/pipeto/

[HermannSW]

Cool, but not nearly as cool as https://pypi.org/project/pipeto/

The vertical bars look like the pipe original, but not the "done"s.
Compose looks good.
Does pipeto run on MicroPython?

[HermannSW]

OK, I did "pip3 install pipeto" and all samples from the project link worked fine under python3.
Then I copied over pipeto.py and operator.py to ESP01s module.
The import of operator.py resulted in errors, these were the differences I had to make to keep MicroPython happy:

Code: Select all

$ diff operator.py.orig operator.py
110c110
<     return a @ b
---
>     return a # @ b
377c377
<     a @= b
---
>     a # @= b
$ 

After those changes importing pipeto was fine, but importing operator afterwards runs out of memory:

Code: Select all

$ webrepl_client.py -p abcd 192.168.4.1
Password: 
WebREPL connected
>>> 
>>> 
MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> gc.collect(); gc.mem_free()
28640
>>> from pipeto import *
>>> import operator as op
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating %u bytes
>>> 

OK, pipeto only needs the "or" operator, so I minimized operator.py for pipeto usage:

Code: Select all

$ cat operator_.py
def or_(a, b):
    "Same as a | b."
    return a | b

__or__ = or_
$ 

Now importing pipeto and minimized operator_ works fine, does cost 1360 bytes RAM ...

Code: Select all

$ webrepl_client.py -p abcd 192.168.4.1
Password: 
WebREPL connected
>>> 
>>> 
MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> gc.collect(); m0=gc.mem_free()
>>> from pipeto import *
>>> import operator_ as op
>>> gc.collect(); m1=gc.mem_free()
>>> print(m0, m1, m0-m1)
28624 27264 1360
...

... and all the examples from pipeto project page work like a charme with MicroPython:

Code: Select all

...
28624 27264 1360
>>> 
>>> inc = lambda x: x + 1
>>> double = lambda x: x + x
>>> 
>>> pipe(1) | float | str | list | done  
['1', '.', '0']
>>> pipe(2) | inc | done 
3
>>> pipe(2) | inc | double | done
6
>>> pipe([1,2,3]) | sum | done
6
>>> newfn = compose(inc) | double
>>> newfn(2) # == double(inc(2))
6
>>> 

A trivial upysh_ sample works with pipeto:

Code: Select all

>>> wc_ = lambda x: wc(x)
>>> pipe('boot.py') | wc_ | done 
16 37 361 boot.py
>>> 

The 1st function arg is passed by pipeto "self.val = fn(self.val)":

Code: Select all

class _Pipe(object):
    def __init__(self, val):
        self.val = val
        
    def __or__(self, fn):
        if fn == done:
            return self.val
        self.val = fn(self.val)
        return self

def pipe(arg):
    return _Pipe(arg)

def done(arg):
    return arg

Since head() command has 2 args, I need to find a way to pass the second arg.

Simplified the problem is starting with this simple example ...

Code: Select all

>>> inc = lambda x: x + 1
>>> pipe(2) | inc | done 
3
>>> 

... the enhanced lambda with two args works:

Code: Select all

>>> inc2 = lambda x,y: x + y
>>> inc2(3,4)
7
>>> 

Not sure how to pass second arg with pipeto, any hints welcome:

Code: Select all

>>> pipe(3) | inc2 ??4?? | done 

[HermannSW]

I had to learn more Python, especially the "*" operator for converting tupel to function args.
It turned out to be really easy:

• do "... | fn | ..." for function fn with 1 (piped) arg
• do "... | (fn, arg2, arg3, ..., argN) | ..." for function fn with N args, 1st arg piped

Here you can see little MicroPython multi arg function pipe demo, it works:

Code: Select all

MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> from pipe_ import *
>>> inc  = lambda x: x+1
>>> inc3 = lambda x,y,z: x+y*z
>>> inc3(4,5,6)
34
>>> pipe(4) | inc | done
5
>>> pipe(4) | (inc3,5,6) | done
34
>>> 

And this is the simple module that does it:

Code: Select all

$ cat pipe_.py 
class _Pipe(object):
    def __init__(self, val):
        self.val = val

    def __or__(self, fn):
        if fn == done:
            return self.val
        elif type(fn) == type( (0,) ):
            self.val = fn[0]( * (self.val,) + fn[1:] )
        else:
            self.val = fn(self.val)
        return self

def pipe(arg):
    return _Pipe(arg)

def done(arg):
    return arg

def or_(a, b):
    "Same as a | b."
    return a | b

__or__ = or_
$ 

The trick was to add

Code: Select all

...
        elif type(fn) == type( (0,) ):
            self.val = fn[0]( * (self.val,) + fn[1:] )
...

to _Pipe class member function __or__!

I can now fully understand pfalcon's previous statement wrt my lambda pipeing implemented for 7 upysh_ commands:

Cool, but not nearly as cool as https://pypi.org/project/pipeto/

Now I need to find the correct way to make use this exciting pipeing feature with upysh_ commands.

P.S:
This is first simple (and wrong) multi arg upysh head() command demo.
What is wrong is that head() command outputs to stdout, instead of pipeing and letting "done" do the output:

Code: Select all

>>> head_ = lambda f,n: head(f,n)
>>> pipe('tst.txt') | (head_,3) | done
first
second().
ThirD
>>> 

[HermannSW]

I wanted to get rid of the "head_" lambda definition for using head() function in pipes.
Because the 7 functions I want this to have for are easy, no inspect module is needed.
I looked up what to take from that module for upysh_ command application.
Here is use of new function L(fn) taking function as input:

Code: Select all

>>> pipe('tst.txt') | (L(head),3) | done
first
second().
ThirD
>>> 

L(fn) takes function fn, determines the variables passed, and then returns exactly the lambda I previously created by hand and assigned to "head_". This is another "oh wow" for me:

Code: Select all

def L(f):
    vs=f.__code__.co_varnames[:f.__code__.co_argcount]
    return lambda *vs: f(*vs)

P.S:
I should got to sleep now -- while L(fn) is correct and cool on its own, it is not needed!
Instead of passing "head_" as earlier, or L(head), just pass "head" and that's it:

Code: Select all

>>> pipe('tst.txt') | (head,3) | done
first
second().
ThirD
>>> pipe('tst.txt') | (tail,3) | done
second().
ThirD
  Fourth()
>>> 

Works for builtin functions as well:

Code: Select all

>>> pipe(-6.3) | abs | done
6.3
>>> 

[HermannSW]

I don't know why pipeto project did import operator module, it is not needed for pipes at least, and removed.
"done" can be any function, is not executed, so I replaced it with minimal function.

I added "__str__" Pipe_ class member, that allows to get all more "shell pipe like" using print():

Code: Select all

>>> from pipe_ import *
>>> pipe(-4) | abs | done
4
>>> print( pipe(-4) | abs )
4
>>> 

Another example:

Code: Select all

>>> def powt(t): return pow(*t)
... 
>>> print( pipe(17) | (divmod,3) )
(5, 2)
>>> print( pipe(17) | (divmod,3) | powt )
25
>>> 

Simplified and enhanced "pipe_.py" ...

Code: Select all

class _Pipe(object):
    def __init__(self, val):
        self.val = val

    def __or__(self, fn):
        if fn == done:
            return self.val
        elif type(fn) == type( (0,) ):
            self.val = fn[0]( * (self.val,) + fn[1:] )
        else:
            self.val = fn(self.val)
        return self
...

Code: Select all

...
    def __str__(self):
        return self.val

def pipe(arg):
    return _Pipe(arg)

def done():
    return

[HermannSW]

Using __str__() the way I did is not good. While MicroPython accepts it, CPython gives an error message:

Code: Select all

>>> sys.version_info
sys.version_info(major=3, minor=4, micro=9, releaselevel='final', serial=0)
>>> print( pipe(-4) | abs )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __str__ returned non-string (type int)
>>> 

As preparation for more pipeing work commit 5085e56 moved file arg to 1st arg of grep() and od() commands. The option string moved to last position for both, with an empty string default:

Code: Select all

>>> grep('tst.txt', 't')
first
  Fourth()
>>> grep('tst.txt', 't', 'i')
first
ThirD
  Fourth()
>>>

[HermannSW]

I made huge progress, integrated the reduced pipe_ module (minus the __str__() member function) into upysh, and made pipe work. This time it is real pipe, that is nothing gets written (as duplicate) to sys.stdout as with lambda pipeing before (lambda pipeing is fine with webrepl.html in browser or webrepl_client.py, but screen session shows all the duplicated lines which is disturbing).

Current upysh.py cannot be imported on ESP8266 anymore because running into out of memory -- temporarily after import 17KB RAM is needed, but gc.collect() shows that really only 8304 bytes are needed:

Code: Select all

...
>>> m1=gc.mem_free()
>>> gc.collect(); m2=gc.mem_free()
>>> print(m0,m1,m2,m0-m1,m0-m2)
100944 84016 92640 16928 8304
>>> 

First, I enhanced what was there, so now both pipeings work.
The most trivial new pipe consists of pipe() with filename and done.
The filename passed to pipe() opens file that is read and printed by done:

Code: Select all

>>> pipe("tst.txt") | done
first
second().
ThirD
  Fourth()
>>> 

I have implemented new pipeing for head() and tail() as proof of concept.
While new pipeing is fine, the result for lambda pipeing now needs an additional step:

Code: Select all

>>> pipe("tst.txt") | (head,3) | (tail,2) | done
second().
ThirD
>>> print( tail(lambda: head("tst.txt",3), 2)  .getvalue(),end="")
second().
ThirD
>>> 

New _Pipe class __init__() prepares the file to be processed first:

Code: Select all

    def __init__(self, val):
        self.val = open(val)

_Pipe class __or__() remains mainly the same, only done handling changed:

Code: Select all

        if fn == done:
            if type(self.val) == TextIOWrapper:
                print("".join(self.val.readlines()),end="")
                return
            elif type(self.val) == StringIO:
                print(self.val.getvalue(),end="")
                return
            else:
                return self.val

_openlambda() keeps being called as before, only this has been inserted to myke new pipeing work:

Code: Select all

    elif type(f) == TextIOWrapper or type(f) == StringIO:
        return f

This is slightly modified head() function:

Code: Select all

def head(f, n=10):
    out = sys.stdout if type(f) == str else StringIO(bytearray())

    with _openlambda(f) as f:
        for i in range(n):
            l = f.readline()
            if not l: break
            out.write(l)

    if out != sys.stdout:
        out.seek(0,0)
        return out

The first and last 3 lines (needed for all upysh commands) look like they can be done as a decorator, but not a simple one.
Instead of always sending output to sys.stdout, now in case of pipeing output gets written into StringIO, which finally gets seeked to first position before returned as input for next pipe command.

This just describes what is there currently, memory on input needs to be brought down, decorators need to be implemented if possible, perhaps the whole lambda pipeing needs to be removed, ...

P.S:
I just removed the lambda pipeing stuff, and now it fits into ESP8266 as well, finally needing 7424 bytes RAM:

Code: Select all

MicroPython v1.9.4-272-g46091b8a on 2018-07-18; ESP module with ESP8266
Type "help()" for more information.
>>> gc.collect(); m0=gc.mem_free()
>>> from upysh_ import *
...
>>> m1=gc.mem_free()
>>> gc.collect(); m2=gc.mem_free()
>>> print(m0,m1,m2,m0-m1,m0-m2)
28656 13136 21232 15520 7424
>>> pipe("tst.txt") | (head,3) | (tail,2) | done
second().
ThirD
>>> 

P.P.S:

Code: Select all

>>> pipe("tst.txt") | (head,3) | (tail,2) | (grep,'t','i') | done
ThirD
>>>