Redis client, packaging questions

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Redis client, packaging questions

Post by dwight.hubbard » Mon May 16, 2016 9:55 pm

I have a partially complete redis client working on micropython at https://github.com/dwighthubbard/micropython-redis the functionality is good enough as a proof of concept (works on both micropython-unix and wipy).

However I have some questions related to packaging.

First some background, I split the client up into separate components by command group. So for example.the redis.lists.RedisLists() class provides a redis client with only the functionality for dealing with lists. Each of the individual classes acts as a mixin so the redis.redis.Redis() class is nothing but a class that pulls in all of the mixins into a single object. This way an app using the redis client in a constrained environment can pull in the specific functionality it uses while in an envirionment with more resources there is a Redis() object that works nearly identically to the redis-py Redis() class.

So my question is, is there packaging support for splitting each component off into it's own package? If so how? If I was doing this with cpython I would put the base stuff in a single package and have each component package define an entrypoint that allows the main package to iterate over the components that are installed. However my understanding is that the micropython packaging doesn't have this functionality.

My second question is how do we use micropython-pip with something other than micropython-unix?? It seems like the packaging documentation is a bit lacking currently.

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Redis client, packaging questions

Post by pfalcon » Mon May 16, 2016 11:10 pm

dwight.hubbard wrote:I have a partially complete redis client working on micropython at https://github.com/dwighthubbard/micropython-redis the functionality is good enough as a proof of concept (works on both micropython-unix and wipy).
Cool, thanks for posting here.
However I have some questions related to packaging.

First some background, I split the client up into separate components by command group. So for example.the redis.lists.RedisLists() class provides a redis client with only the functionality for dealing with lists. Each of the individual classes acts as a mixin so the redis.redis.Redis() class is nothing but a class that pulls in all of the mixins into a single object. This way an app using the redis client in a constrained environment can pull in the specific functionality it uses while in an envirionment with more resources there is a Redis() object that works nearly identically to the redis-py Redis() class.

So my question is, is there packaging support for splitting each component off into it's own package? If so how? If I was doing this with cpython I would put the base stuff in a single package and have each component package define an entrypoint that allows the main package to iterate over the components that are installed. However my understanding is that the micropython packaging doesn't have this functionality.
MicroPython supports that in the most natural way (from my POV). You may have e.g. redis-lists, redis-dicts, dist-packages containing respectively redis/List.py, redis/Dict.py. Each can be install separately, and imported, thanks to Python3 namespace package. Then pull-all package, redis, depends on all of redis-lists, redis-dicts, etc., and contains redis/__init__.py which imports them all. You can see example of this collections and uasyncio packages in micropython-lib.
My second question is how do we use micropython-pip with something other than micropython-unix?? It seems like the packaging documentation is a bit lacking currently.
It's all on the forum, and due to lack of people who'd use it *and* report results, still has "experimental" status, so there's no documentation beyond that. The general idea is that you create "installable tree" using -p switch to upip and deploy to the target in whatever way suitable (rsync, scp, webrepl, ftp, xmodem, ...). https://github.com/pfalcon/micropython-dlman (old, unupdated) and https://github.com/pfalcon/notes-pico give examples of this approach.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Tue May 17, 2016 7:25 am

Great, that was the information I needed.

I split the package up into several smaller packages which I am able to install using pip-micropython using the document links you provided :-)

Then added my notes to the documentation at http://micropython-redis.readthedocs.io/ feedback certainly would be welcome since this is the first time I've packaged anything for micropython.

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Redis client, packaging questions

Post by pfalcon » Tue May 17, 2016 4:55 pm

dwight.hubbard wrote: I split the package up into several smaller packages which I am able to install using pip-micropython using the document links you provided :-)
pip-micropython is deprecated, and replaced with native "upip", that's also on forum: viewtopic.php?f=15&t=524 and in README at https://github.com/micropython/micropython
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Thu May 19, 2016 7:31 am

pfalcon wrote: pip-micropython is deprecated, and replaced with native "upip", that's also on forum: viewtopic.php?f=15&t=524 and in README at https://github.com/micropython/micropython
[/quote]

Ok, I think I got the basic stuff figured out.

The documentation now uses the upip module, I have travis running basic unit tests and coverage reporting. Now I can start working on the fun stuff.

machdisk
Posts: 16
Joined: Sun May 22, 2016 1:35 am

Re: Redis client, packaging questions

Post by machdisk » Sun May 22, 2016 2:02 pm

Dwight,

Thanks for this contribution. I took a quick look at the github repository and see that there are a few tests included but might you provide a quick usage example (a LED blinking example would be awesome where the WiPy reads a variable from the redis server)? I'm still a bit new to microcontroller platform and am not exactly sure how to set things up. Specifically, I'm interested in seeing how to configure the client and have it connect to the redis server running on a PC connected via wifi to the WiPy. My trouble is knowing which IP address to specify for the WiPy. Do I need to know the PC IP address or is it always 127.0.0.1?

My ultimate goal is to have a logging process on the WiPy that can access the redis server to pull in the variables necessary for the process and complete the tasks in order and place the results back to the server. Any help or advice would be appreciated.

Cheers,

Brian

dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Mon May 23, 2016 5:09 am

The main uredis.Redis class pulls in all of the functionality that's been implemented., so it works pretty much like the normal redis-py python bindings, for the functionality that's supported.

Code: Select all

$ micropython
MicroPython v1.8-77-gafce978 on 2016-05-15; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import redis
>>> r=redis.Redis('192.168.4.2', 6666)
>>> r.keys()
[b"temp:esp8266", b'temp1']
>>> r.llen("temp:esp8266")
2769
>>> r.lrange("temp:esp8266", -10,-1)
[b'25.625', b'25.625', b'25.625', b'25.625', b'25.625', b'25.625', b'25.625', b'25.625', b'25.625', b'25.625']
>>> 
The module splits up the functionality for each redis operation group into individual modules so it's possible to generate a Redis class that only has the needed functionality.

This let me get a simple example working on an esp8266 (the esp8266 logged the values that where in the previous example)

I ran this on a nodemcu esp8266 board with the most recent micropython from the master branch. To make this work I had to disable the two imports in the uredis/__init__.py (going to have to do some more work on the packaging it looks like)

However with the above change the following code in main.py takes the temp from an ds18b20 every second and appends it to a list in the redis server running on port 6666 on my laptop which is connected to the esp8266 with the default wifi configuraton.

Code: Select all

from machine import Pin
import onewire
import time
from uredis.list import List

class Redis(List):
	pass

r=Redis('192.168.4.2', 6666)
ow = onewire.OneWire(Pin(2))
sensor = ow.scan()[0]
while True:
    ds = onewire.DS18B20(ow)
    ds.convert_temp()
    current_temp = ds.read_temp(sensor)
    r.rpush('temp:esp8266' % sensor, current_temp)
    print('temp: %s' % current_temp)
    time.sleep(1)

dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Mon May 23, 2016 5:47 am

If you are running the redis server on your laptop with the default port (6397) and using the default configuration for the wipy then the following would implement a simple message queue. Any value added to the "myqueue" list would be pulled pulled off of the list on the wipy and the callback_function() will be called with the value that was on the list.

Code: Select all

import redis

connection = redis.Redis('192.168.1.2', 6397)
redis_key_name = "myqueue'

def callback_function(argument):
    print(argument)

while True:
    value = connection.blpop(redis_key_name, 9999)  # Get and remove the first element in the list and block if there isn't anything there
    callback_function(value)

machdisk
Posts: 16
Joined: Sun May 22, 2016 1:35 am

Re: Redis client, packaging questions

Post by machdisk » Tue May 24, 2016 1:16 pm

Dwight,

This is great! After a little haggling on my part I was able to get it working and have a loop poll the redis queue on my laptop and have it change the frequency of the LED on the WiPy. Though simple, it will serve as the basis for more complicated efforts to follow.

Cheers,

Brian

dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Tue May 31, 2016 6:28 pm

I'm glad to hear you got it working. Once I get some time I'm going to see if I can port the hotqueue queue.worker decorator to micropython.
machdisk wrote:Dwight,

This is great! After a little haggling on my part I was able to get it working and have a loop poll the redis queue on my laptop and have it change the frequency of the LED on the WiPy. Though simple, it will serve as the basis for more complicated efforts to follow.

Cheers,

Brian

Post Reply