Simple download manager (example of deployable application)

Showroom for MicroPython related hardware projects.
Target audience: Users wanting to show off their project!
Post Reply
pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Simple download manager (example of deployable application)

Post by pfalcon » Sun Jul 20, 2014 7:14 pm

Of course my NAS has vendor's download manager. Of course it doesn't work as expected. Of course I found open-source download manager webapp in Python instead: https://github.com/clowwindy/PyWebGet . Of course, it had bugs with filename handling similar to vendor's, but as it's open source and Python, it was easy to fix: https://github.com/pfalcon/PyWebGet . But of course, PyWebGet uses threads, and as smart people say, threads == bugs. Usually they manifest by deadlocking, but in case of PyWebGet, after 3-4 days of regular usage, it spawned bunch of threads, which took 99% of CPU time.

So, I decided to finally put MicroPython at real-world usage, and write dead simple download manager, just a frontend to wget. The app is pretty simple (but new ways of uPy usage prioritized some of known issues, so there were some uPy development and refactoring required). But what I tried to achieve is lay out a procedure to full deployment of MicroPython application to a target embedded system (embedded Linux system in this case).

I hope it might be interested to other folks who'd like to put MicroPython into daily real-world usage: https://github.com/pfalcon/micropython-dlman . Note that functionality of download manager is not yet up to par - I found that trivial algo used to poll for new downloads preclude disk from going into suspend mode, which is no-go for a home NAS. But deployment process is there and can be used for other apps too.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

nande
Posts: 13
Joined: Fri Oct 10, 2014 6:11 pm
Location: argentina
Contact:

Re: Simple download manager (example of deployable applicati

Post by nande » Mon Nov 03, 2014 9:25 pm

Threading in python is fairly easy, if you do a correct design it is not that bad.
also using the queue class can solve most of your problems.
however afaik CPython has only one concurrent thread*1 so it-s not that usefull unless you're doing some kind of IO ops (like you).
maybe in upy you will need to use interrupts to interrupts to check the transfers, buffer them, and then polls the buffers towrite them out.
*1 GC Related issue iirc
~namida de ashita ga mienai~

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Simple download manager (example of deployable applicati

Post by pythoncoder » Wed Nov 05, 2014 9:29 am

@pfalcon The criticism of threading in the link you provided seems to be founded on the assumption of true concurrency (pre-emptive multi-threading). Using a cooperative model avoids many of the difficulties - while admittedly introducing at least one of its own: the capability of a badly written thread to fail to yield to other threads and lock the system. In a cooperative multi threaded environment the case where a single routine is concurrently executed by multiple threads can't occur because there is no true concurrency. Unless, of coures, I'm missing something in his argument.

I appreciate his later comments but I think we need to consider the Micro in MicroPython: the language's yield and yield from statements enable a low cost implementation of coroutines ideal for the relatively simple yet nontrivial embedded systems targeted by MicroPython. These can be easier to write using cooperative threaded code rather than coding a single event loop which easily descends into a Python snake tangle. Of course writing MicroPython interrupt handlers is a different kettle of fish and requires due caution ;)

Regards, Pete
Peter Hinch
Index to my micropython libraries.

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Simple download manager (example of deployable applicati

Post by pfalcon » Wed Nov 05, 2014 2:43 pm

In a cooperative multi threaded environment
Well, people usually avoid calling cooperatively running things "threads". They call them coroutines, tasks, and actually, gazillion of other things (for example, Microsoft calls them fibers, Go - goroutines, etc.) showing their authors creativity, but most importantly, avoiding mixing them up with pre-emptive threads, commonly known as just "threads". Doing so avoid confusion an makes for simple rules of thumbs like ' threads are "bad" ' (mind the quotes).

Btw, we have general thread on multitasking - http://forum.micropython.org/viewtopic.php?f=2&t=15 , which may be more suitable for its discussion.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Post Reply