State of MicroPython universe (was: of micropython-lib?)

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: State of micropython-lib?

Post by kevinkk525 » Wed Mar 28, 2018 7:53 am

Oh sorry. than that's my fault.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: State of micropython-lib?

Post by pfalcon » Sat Dec 01, 2018 9:23 am

I'd like to bring up this topic, in a wider context of the "State of MicroPython universe". It's actually good timing, because it has passed a year since the event discussed in this thread (even though the thread was created later).

But before going to that, it makes sense to remind ourselves of (contentious, because otherwise Python community is doing great) things which happened in the wider Python universe:

1. A very few Python core developers came out with a PEP (PEP572) to add a totally atrocious syntax to the language, unheard of before. Mad things get added to the mainline Python all the time, but this time the reaction was largely unanimous. This tweet summarize it very well: https://twitter.com/SerhiyStorchaka/sta ... 4452209665 (more discussion: 1 2).
2. One of the 3 developers behind PEP572 was Guido van Rossum, Python's BDFL. That may explain why that PEP was accepted in the first place. Guido (besides being genius, creator of Python, and nice guy overall) was already criticized enough for contradictory moves by Python community. And this time, it brought the case very close to "the king is mad" situation. He thus decided to step off as a honorary BDFL, leaving decision process to be more open (or maybe void so far, and who knows how it comes out).

Two interesting points here: it's always about decisions, here or there. And it's surely enough subjective, if some call decisions "mad". Yet, there should be some objective criteria how to judge changes/decisions. And I'm speaking as a MicroPython developer. And why am I a MicroPython developer in the first place? Because I know Python and see that it offers more resource-efficient ways to develop applications comparing to other scripting language. I know it for awhile, and that effectively becomes Python's social contract: I (and other people) expect it to be that and stay that, allowing users to rely on that, to make bets on that. Allow project like MicroPython to appear (easy) and continue to exist (much harder). So, any changes which go against that nature of Python, and changes which break the trust and social contract, any changes which undermine MicroPython, to which I dedicated quite a few years of my life already - such changes I call mad.

Second point is - it's always about disagreement. Disagreement must be bad, right? Well, I recently found an insightful talk "The Hard Parts of Open Source" which argues that disagreement is deeply ingrained in the mentality of the current-generation folks who deal with computers. Of course, cooperation and productivity can't exist without agreement, the point is that there will be always disagreement sprinkling around. And of course, agreement needs to be cultivated, given that the "default" state is disagreement. Put it that way, it's human to disagree, agreeing is divine.

(next post will be dedicated solely to MicroPython situation)
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: State of micropython-lib?

Post by pfalcon » Sat Dec 01, 2018 11:13 am

So, let's enumerate what happened in specifically in regard to topics of this thread over the last year:
  • While I had a bunch of changes to further optimize uasyncio (and that work stems from the Kickstarter stretch goals!), based on the feedback in this thread, I put it into the backlog and switched to other project for that time being.
  • I continued to develop and maintain micropython-lib.
  • I might expect that there will be code changes to undermine my work in the MicroPython repository, that didn't happen.
  • I wondered if I will be able to rebase on the mainline, or too much crap will flow in. Again, didn't happen. Over this time, I reverted just a couple of underthought commits by Damien which went in due to lack of peer review.
  • Mainline releases became rather sparse, over last year, there was just 1 release. (Ok, if you look at 13 months, there was 2).
  • Development infrastructure grows unmaintained: #4026, #4282
  • Getting back from summer, looking for new MicroPython usages, I "discovered" a bunch of bugs and non-implemented features. I implemented/fixed them in my fork, submitted as PRs to mainline 34 patches.
  • 10 of them remain unmerged, and majority of those didn't get a reply. Actually, the more recently submitted ones are such. I have a bunch more to submit, oftentimes depending on those unmerged. And what the point to submit more if older are unmerged, and newer will likely pile up the same.
  • Similar slow/none response happens with other PRs. Selectively of course. A serious issues reported may go without response (missed?), while other are responded within hours.
  • On the other hand, micropython-lib project slowly recovers from the damage done by imposed forking, recently there was a few contributions by folks who find their way to the primary repository.
  • Seeing all this, specifically: despite forking, there's still a lot of common infrastructure require a lot of work, and Damien clearly can't keep up with it alone - I submitted 1.5 months a request for reinstating my commit access: https://github.com/micropython/micropython/issues/4239
  • Just as many other submissions, it went unresponded for a while, but finally got a write-off that no, because "there's disagreement".
So, that's as far as it goes for the mainline and its interaction with contributors.

Otherwise, MicroPython ecosystem goes well - this autumn I finally got the feeling that the project aims are achieved. And those aims were always providing a viable Python language implementation for embedded/constrained systems, as alternative to other solutions. Not only us, dedicated MicroPython users know that, other parties with enough choices on their hands do select MicroPython as their system of choice, and parties who selected it previously spot quite an impressive results, growth, and liveness based on this choice.

With the background like this - and I heard that from other people, and now tend to agree - mainline starts to look bleak with its poor response times, poor response content, stubbornness, and lack of of desire to cooperate. Damien just can't get that he's now just one of the player in MicroPython universe, however, as the creator of it all, there's higher responsibility and scrutiny lies on him. But he (or his company) can't be the sole point of tripping in this universe, and he actually isn't for quite some time.

So, let's everyone embrace this pluralistic MicroPython universe, and I'd like to share steps I plan to make with my humble projects in the next message.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: State of micropython-lib?

Post by pfalcon » Sat Dec 01, 2018 11:47 am

1. micropython-lib's has got a new motto: "MicroPython standard library for all ports, forks, and variants". As you know, motto can't convey all the complexity of real-world situations. The fine print is "for any variant which wants to be compatible". If Loboris doesn't want his fork to be compatible - it won't be compatible. If Adafruit brags a desire to remove standard MicroPython APIs from their fork - it won't be compatible. If George Robotics wants their variant to be not compatible - it will be such. I can't control 3rd parties. I can only, as a maintainer of affected subsystems, whenever possible to submit PRs and have offered to continue serving as a maintainer in the repository which initially hosted the main variant.

Otherwise the project remains under the same name, "micropython-lib". Note that the name of project by Damien George is "Micro Python". Its my and other community members insistence which cause it to become "MicroPython" instead. Damien is free to revert back to older name. It's otherwise rather common practice that a language and a library project(s) for it are maintained by different parties. Guide van Rossum doesn't own PyPI. He may want to exterminate Python2 and prohibit packages for it, but they still will be there. LuaRocks is maintained by another guy than the authors of Lua. It definitely hosts packages which don't run on "official" Lua, but require an alternative implementation, like LuaJit. micropython-lib itself is no longer limited to MicroPython, but intended for CPython too.

So, any arguments and attempts to constrain micropython-lib to a particular variant are misguided. Instead, it's up to various variants whether they want to run a particular package or not. It's up to their authors/maintainers/contributors of packages to develop and maintain their packages further.

2. Moratorium on changes to uasyncio is over. There will be changes to optimize it further based on the original aims of the project. Older versions remain available. It's licensed under open-source license, under terms of which you can continue its usage or make your own variant (the license requires preserving of the copyright statement of the original author(s) and license terms).

3. Likewise, there will be changes to optimize upip, and many other things.

4. Besides pure technical purpose, my fork is intended to be vendor-neutral MicroPython variant. I recommend to submit all suitable contributions to Damien's repository first. However, parties which are concerned with vendor neutrality, are welcome to share their concerns e.g. in my repository's tracker.

5. My fork has its own name. But nobody knows it/cares about it. And I won't tell any person "I won't be talking to you unless you call my project by its official name". I myself will be using a colloquial rendition understood by other people. So, it largely depends by the community if a particular name stick. The colloquial rendition in question is again "MicroPython" as established by the community to name ecosystem, don't mix that up with "Micro Python", project by Damien George.
Last edited by pfalcon on Wed Dec 26, 2018 10:07 pm, edited 1 time in total.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: State of MicroPython universe (was: of micropython-lib?)

Post by pfalcon » Sat Dec 01, 2018 11:48 am

Few, that was long ;-). But a whole year piled up a bit of updates, I hope you agree ;-).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
adafruit
Posts: 12
Joined: Sun Oct 30, 2016 11:56 pm
Location: NYC
Contact:

Re: State of micropython-lib?

Post by adafruit » Sat Dec 01, 2018 6:22 pm

pfalcon wrote:
Sat Dec 01, 2018 11:47 am
If Loboris doesn't want his fork to be compatible - it won't be compatible. If Adafruit brags a desire to remove standard MicroPython APIs from their fork - it won't be compatible. If George Robotics wants their variant to be not compatible - it will be such. I can't control 3rd parties.
For Adafruit CircuitPython one of the goals is to have a fully consistent hardware API across all board ports and CPython compatibility for libraries. For example, instead of "utime" or "uos", we implement "time" and "os" libraries. Those libraries may not be full featured, but they are strict subset: all CircuitPython code should be able to run on CPython. We took a look at micropython-lib, good work :) CircuitPython will be compatible as long as you are using standard CPython names and subset implementation.

See -Design for compatibility with CPython
https://circuitpython.readthedocs.io/en ... th-cpython
Last edited by pfalcon on Sat Dec 01, 2018 8:29 pm, edited 1 time in total.
Reason: Removed attention-seeking graphics

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: State of micropython-lib?

Post by pfalcon » Sat Dec 01, 2018 8:28 pm

adafruit wrote:
Sat Dec 01, 2018 6:22 pm
For Adafruit CircuitPython one of the goals is to have a fully consistent hardware API across all board ports and CPython compatibility for libraries. For example, instead of "utime" or "uos", we implement "time" and "os" libraries. Those libraries may not be full featured, but they are strict subset: all CircuitPython code should be able to run on CPython.
Well, this is a wrong place for these discussions, this topic is dedicated to other issues. So, let me be quick: you have rather strange definition of "CPython compatibility". CPython compatibility means being able to run CPython applications on SomethingPython, not the other way around. So, you pretend and misguide that you have "CPython compatibility", whereas actually you complicate it by grabbing namespace which doesn't belong to you. Or put it another way, what you do is as good as teaching kids that the value of Pi is 3.

Also, please avoid posting distracting graphical materials (removed) in each post you make. I understand that your target audience is comics-watchers, but please have feeling of appropriateness.

On the other hand, is you want to tell us something interesting, feel free to tell whether you job title is something like "community manager", or you're from marketing department on part-time assignment. Knowing that would be indeed insightful. Thanks.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: State of MicroPython universe (was: of micropython-lib?)

Post by kevinkk525 » Sat Dec 01, 2018 9:03 pm

Well just looking at this post @pfalcon I can see one problem why you are not part of the official mainline anymore. Your language towards others is often aggressive, inappropriate and you make fun of others because you don't respect them or their work with sentences like
I understand that your target audience is comics-watchers, but please have feeling of appropriateness.
You get very personal and kind of emotional instead of speaking objectively and with respect of others and their work, even if you disagree or think it is not good.
You are obviously a very skilled and genious programmer but with this obvious lack of personal interaction, I can understand that it is difficult to work with you. I'm sorry to say that.

This leaves many of the points you make untouched. These are real problems that partly bother me too and should get worked on.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: State of MicroPython universe (was: of micropython-lib?)

Post by pfalcon » Sun Dec 02, 2018 5:44 am

kevinkk525 wrote:
Sat Dec 01, 2018 9:03 pm
Well just looking at this post @pfalcon I can see one problem why you are not part of the official mainline anymore. Your language towards others is often aggressive
I'm sorry dude, but who's aggressive here is you.
, inappropriate and you make fun of others because you don't respect them or their work
Oh, surely I respect them and their work. But if you expect me to bow down in awe at each appearance of a marketing-ridden post, then surely that's ungrounded. I prefer humanly humorous chit-chat instead.
with sentences like
I understand that your target audience is comics-watchers, but please have feeling of appropriateness.
What you're trying to do is discuss moderator's work (and I'm still a moderator here). On many forums, that's immediate ban or pre-ban, but we have a democracy here. Otherwise, account "adafruit" walks a very fine line between direct advertisement/spam and helpful posts, and I'm surely appreciate that skillfulness, but it's my responsibility to warn them to not cross that line.
You get very personal and kind of emotional instead of speaking objectively and with respect of others and their work, even if you disagree or think it is not good.
Sorry again, but that's nonsense. I surely think that Adafruit's strategic investment in MicroPython is a good thing and want to learn more about it. Again, if you expect me or someone else being mesmerized by their awe and microsoft-like tactics of dealing with OpenSource software, that's ... naive.
This leaves many of the points you make untouched. These are real problems that partly bother me too and should get worked on.
So it seems it's you who got too emotional (you don't hold their stock options, do you?). Please stop your personal attacks and get to work to resolve the real problems.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: State of MicroPython universe (was: of micropython-lib?)

Post by kevinkk525 » Sun Dec 02, 2018 11:15 am

Well I'm not going to answer to that, because I am not interested in any emotional discussion. Everyone can think about this as he wishes. I have no grudge against you, never worked with you and have no reason to personally attack you. I'm just an observer on this forum.
If you think I'm aggressive, this must be the first post that anyone sees that way.

Oh and I don't hold any stock options of Adafruit ;)

I'm here for the problems you pointed out.
- I like that micropython-lib should stay compatible to every fork and that forks should try to stay compatible to those libraries.
- I don't like that mainline takes ages to respond to pull requests and merge improvements. Micropython development is very slow at the moment. It's good to bring that issue up. Mainline developers should think about how to change that, I just don't think the solution is to get you back in as there were obvious disagreements and problems in working together. I of course understand that you'd like to try again, just as I understand that the mainline developers (damien?) does not want to try it again.
- I like that you continue to commit to the micropyhton universe.
- Your thoughts about mad additions to python and the problem of (dis)agreement in opensource were really interesting and do show a problem in the opensource scene. While we may have disagreement in some parts, we certainly agree to many problems you pointed out.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

Post Reply