Redis client, packaging questions

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Tue Jun 28, 2016 7:45 am

I just did some updates to the redis client to decrease the resource usage.

Now that I have the redis functionality I need working on the esp8266 I created a module that implements a subset of the hotqueue module functionality.
MicroPython v1.8.1-87-g7ddd85f-dirty on 2016-06-27; ESP module with ESP8266
Type "help()" for more information.
>>> from microqueue import MicroQueue
>>> queue = MicroQueue('queuename', host='192.168.1.183', port=6666)
>>>
>>> @queue.worker
... def print_message(message):
... print(message)
...
>>> print_message()
Micropython Rocks!!!
This works pretty well at this point :D
dwight.hubbard wrote:I'm glad to hear you got it working. Once I get some time I'm going to see if I can port the hotqueue queue.worker decorator to micropython.
machdisk wrote:Dwight,

This is great! After a little haggling on my part I was able to get it working and have a loop poll the redis queue on my laptop and have it change the frequency of the LED on the WiPy. Though simple, it will serve as the basis for more complicated efforts to follow.

Cheers,

Brian

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Redis client, packaging questions

Post by pfalcon » Sat Jul 02, 2016 2:51 pm

dwight.hubbard, you may be interested in MicroPython's native btree module which is in development now: http://forum.micropython.org/viewtopic.php?f=15&t=2043 . With it, MicroPython becomes pretty much a full-stack platform now.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

jladd
Posts: 4
Joined: Sun Jul 03, 2016 2:46 am

Re: Redis client, packaging questions

Post by jladd » Sun Jul 03, 2016 2:49 am

Dwight,

How are you able to get the uredis module onto the 8266?

I looked at upip and it looks like that only supports micropython on unix (could have that wrong). I also looked into using webrepl_cli.py to try to copy over a directory, but it looks like it can only handle files. Finally, looked at the Makefile for micropython to see if there was a way of including an extra module, but didn't see anything. Is there a trick to this?

(By the way, tried installing uredis using the line from tox.ini and it worked - appreciate the fix!)

dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Sun Jul 03, 2016 8:25 am

There are multiple ways to do it, but the method I've been using on the esp8266 is:
  1. Clone the micropython repo from github.
  2. Build and install micropython unix.
  3. Go into the root micropython directory and "make clean"
  4. Change into the esp8266 directory.
  5. Install the micropython-redis package to the esp8266 scripts directory using the upip module of the micropython-unix interpreter you just built. I.E.: MICROPYPATH=scripts;micropython -m upip install micropython-redis
  6. Deploy it to the board with "make PORT=/dev/ttyUSB0 deploy"

jladd wrote:Dwight,

How are you able to get the uredis module onto the 8266?

I looked at upip and it looks like that only supports micropython on unix (could have that wrong). I also looked into using webrepl_cli.py to try to copy over a directory, but it looks like it can only handle files. Finally, looked at the Makefile for micropython to see if there was a way of including an extra module, but didn't see anything. Is there a trick to this?

(By the way, tried installing uredis using the line from tox.ini and it worked - appreciate the fix!)

dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Sun Jul 03, 2016 9:40 am

One other thing, if you're running on esp8266 using the full uredis.Redis class will take up most if not all of the esp8266's memory. Take a look at http://micropython-redis.readthedocs.io ... ar-classes for information on how to pull in just the components you need.
jladd wrote:Dwight,

How are you able to get the uredis module onto the 8266?

I looked at upip and it looks like that only supports micropython on unix (could have that wrong). I also looked into using webrepl_cli.py to try to copy over a directory, but it looks like it can only handle files. Finally, looked at the Makefile for micropython to see if there was a way of including an extra module, but didn't see anything. Is there a trick to this?

(By the way, tried installing uredis using the line from tox.ini and it worked - appreciate the fix!)

jladd
Posts: 4
Joined: Sun Jul 03, 2016 2:46 am

Re: Redis client, packaging questions

Post by jladd » Sun Jul 03, 2016 7:31 pm

Cool! Will give it a try.

At this point, I feel like I have some good background on redis and micropython-redis - is there anything you'd like help with?

jladd
Posts: 4
Joined: Sun Jul 03, 2016 2:46 am

Re: Redis client, packaging questions

Post by jladd » Mon Jul 04, 2016 4:48 am

Tried your steps for flashing micropython. (tl;dr - had to work around problem I ran into with upip, but did eventually get it to work).

When I ran MICROPYPATH=scripts;micropython -m upip install micropython-redis, I found that the current working directory was populated with tar files corresponding to the different micropython-redis modules (hash, set, etc.). I tried manually copying the micropython-redis python files from ~/.micropython/lib/uredis_modular directly into the esp8266/scripts directory though and it worked (I only copied over a subset of the files given the space constraints you mentioned). After that, flashing the scripts was successful and I was able to import (and use!) the redis client against a redis server running on my machine.

I didn't dig too much into why the pip install wasn't working for me. I did try setting an absolute path for MICROPYPATH, but no dice. Anyways, that's the latest.

Jim
dwight.hubbard wrote:There are multiple ways to do it, but the method I've been using on the esp8266 is:
  1. Clone the micropython repo from github.
  2. Build and install micropython unix.
  3. Go into the root micropython directory and "make clean"
  4. Change into the esp8266 directory.
  5. Install the micropython-redis package to the esp8266 scripts directory using the upip module of the micropython-unix interpreter you just built. I.E.: MICROPYPATH=scripts;micropython -m upip install micropython-redis
  6. Deploy it to the board with "make PORT=/dev/ttyUSB0 deploy"

dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Tue Jul 05, 2016 5:37 pm

The README.md in the repo has a table that shows what is implemented and what isn't, as well as a rough estimate of the current percentage of the test coverage for each component.

So basically some functionality still needs to be implemented, most of them aren't heavily used with the exception of publish/subscribe and transaction functions.

Also more tests would be really useful, generally in writing tests I've been finding some methods need to handle additional keyword arguments for compatibility with redis-py.

Beyond that the documentation can use a lot of work. Documentation showing the usage of the low level apis that are the most resource efficient is currently completely missing and it needs a lot more detail and examples.

The code could probably also use more optimization. Currently I've been looking to eliminate code that does spurious copy operations. Also looking to as much as possible eliminate encode()/decode() operations as much as possible.

I have been considering doing an implementation that uses a fixed size buffer without doing any memory allocation. To prevent memory fragmentation as well as allowing it to be used within interrupt handlers. Obviously the API for such an implementation would have to be slightly different though.
jladd wrote:Cool! Will give it a try.

At this point, I feel like I have some good background on redis and micropython-redis - is there anything you'd like help with?

jladd
Posts: 4
Joined: Sun Jul 03, 2016 2:46 am

Re: Redis client, packaging questions

Post by jladd » Tue Jul 05, 2016 5:52 pm

Awesome, really appreciate the ideas. Will check out those areas and try to find a place to start banging on.

dwight.hubbard
Posts: 38
Joined: Mon May 16, 2016 6:35 pm

Re: Redis client, packaging questions

Post by dwight.hubbard » Tue Jul 05, 2016 6:25 pm

Oh, I find that module very interesting. If the resource usage was low enough I could see it being a good replacement for json config files. Beyond the obvious use case of storing sensor readings. I could also see it being very interesting if the writes use less power/resources than powering up the network and sending the data to an external cloud database.

It would be particularly interesting if there was an immutable view of the data that could be read directly from flash without the need to allocate memory.
pfalcon wrote:dwight.hubbard, you may be interested in MicroPython's native btree module which is in development now: http://forum.micropython.org/viewtopic.php?f=15&t=2043 . With it, MicroPython becomes pretty much a full-stack platform now.

Post Reply