A Tale of Two APIs

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

A Tale of Two APIs

Post by Turbinenreiter » Fri Jun 02, 2017 8:03 pm

I believe that MicroPythons hardware API is its most important feature. There only few languages that you can use on microcontrollers, mostly by the virtue of their compiler suite - Rust, ADA and D are examples. But getting the language to run on a microcontroller is just the first step, the much more important step is to be able to control the hardware from the language. Most vendors supply C and maybe C++ libraries for their chips, but not for any other languages. You can always call C functions, but then you end up with an Hardware API that is not native to the language you are writing in.

MicroPython solves this with the machine module. The machine module is a pain in the ass to write. You have to come up with an abstraction for the hardware that is universal and that people agree on. Then you have to implement it for every single microcontroller-family individually. Every family has it's own C API, or register layout if you do it like that. You get zero free lunch. By my counting, the machine module is only fully implemented for the STM32F4 family, the CC3200 and for the ESP8266 and ESP32. I'm saying "only", because that is just a very small handful of chips, at the same time it is a pretty huge achievement. Implementing the machine module for a certain platform is hard and long work. But its power is magnificent. When I write a driver for a sensor, it works across different microcontrollers, because the machine module is consistent on all of them.

Except when it isn't, and this finally brings me to the reason I'm writing all this: CircuitPython. I love Adafruit and I love the Feather boards and when they picked up MicroPython, I was extremely happy. The Pyboard is a nice board, too, but it is just one form-factor and one form-factor doesn't fit all. It fits my needs very badly. The Feather boards fit my needs amazingly well. But all the joy kinda went down the drain when Adafruit decided that the didn't want to implement the machine module. They gave a reason for doing so. They are saying that CircuitPython is aimed at beginners and therefore they want a simpler API.

I don't believe that CircuitPythons API is simpler. Actually, it is kinda the same. It's just different enough to break compatibility. Let's look at an example: I2C. I am just looking at the methods here. A sensor driver would take an I2C object as an argument when the object is created, so I don't really need to care about the differences in creating the I2C object. I even think that it may be valid that this object creation is different on different platforms, as there are some different approaches in the hardware: STM has a number of I2C busses on specific pins. Atmel has SERCOM busses that can be used for I2C on different Pins.

MicroPython has the following methods for I2C:

Code: Select all

init
deinit (WiPy)
scan

start (ESP8266)
stop (ESP8266)
readinto
write

readfrom
readfrom_into
writeto

readfrom_mem
readfrom_mem_into
writeto_mem
CircuitPython has these:

Code: Select all

deinit
scan

try_lock
unlock

readfrom_into
writeto
There is a couple of interesting things here. First, machine fails to be consistent, having some methods that are only available on some chips. Second, CircuitPythons API is pretty much a subset of machine. But only almost.

CircuitPython:

Code: Select all

readfrom_into(address, buffer, *, start=0, end=len(buffer))
MicroPython:

Code: Select all

readfrom_into(addr, buf, stop=True)
Here, CircuitPython gives you an extra start and end argument, MicroPython doesn't. Why? Because CircuitPython doesn't have the memory operations MicroPython has. But when writing sensor drivers, you read and write to certain memory registers on the sensor via I2C all the time. So without the the memory operations, you have to manipulate the data in the buffer and write and read the whole buffer. Given that the sensors often use certain registers to trigger measurements, you always have to make sure that you don't actually set one of those when writing the buffer to do something completely unrelated. The start and end arguments are there to allow you to write only to certain memory areas. Just in a different way then in the machine module. Worse, if you ask me, but that isn't the matter. The matter is that I now have to either write my sensor driver twice to target each version or I have to use a subset of a subset of a well thought-out API, making my live miserable and sad.

The APIs are almost the same. The simplification argument falls short. CircuitPython may have less methods for its I2C object, but they are actually harder to use. The differences don't improve anything. They are just differences.

The good thing is, none of this unsolvable and none this is happening in bad faith, so we can actually try to fix it. One, very brutal, way would be to write a machine-wrapper around the CircuitPython API that restores compatibility. We could also fork their fork and rework their API into the machine module. A softer method would be to start at the class level and send them small pull-requests, fixing the compatibility issues of the methods. But I think that is just half of the work. The other half is to figure out how we can make sure that everyone who adds a new target for MicroPython happily implements the machine module, instead of something incompatible. We could also think about how we can get more Pyboard form-factors into the MicroPython shop and make Damien some money.

What do you think?

User avatar
deshipu
Posts: 1388
Joined: Thu May 28, 2015 5:54 pm

Re: A Tale of Two APIs

Post by deshipu » Sat Jun 03, 2017 4:19 am

I have been following the developments of both the machine API and the Adafruit APIs, and I have been asking questions along the way, so I think I can give you at least some answers here. Note that I am not defending the status quo -- I also think that it would be awesome if there would be compatibility, and I think the developers also agree -- I'm merely trying to understand the process that leads all the parties involved to producing this mess despite their best intentions.

So first, the reasons for there being separate APIs in the first place -- Adafruit cites their desire for making it simple and consistent across their products -- and that's the truth, but I don't think it's the whole truth. You see, when you invest heavily in hardware, like Adafruit does, designing and manufacturing the boards, creating documentation for them, lessons, video tutorials, etc. -- you really want to also control the software side of things, or you become vulnerable to the whims of the developers. Just look at the most recent release of MicroPython for the ESP8266, which light-handedly breaks practically all existing libraries for SPI hardware, and practically all existing tutorials. If Adafruit was bound to use MicroPython (or even just the same API), they would be practically paralyzed for a month or two fixing all their code, rewriting all the documentation and re-recording all the videos. And then a couple more months dealing with people who didn't upgrade. For a company like Adafruit that would be a catastrophe. And stuff like that happens randomly all the time, as certain developers strive to leave their mark in the code. Do it several times, and you could easily go bankrupt. So there is simply no choice for Adafruit in this matter.

Ok, you could say, but why did they have to modify it so much, why didn't they at least tried to follow some older version of the machine API? There are multiple reasons for that too. You brought up the I2C API as an example, so let's look at that.

First, there are no low-level "start", "stop", "readinto" and "write" methods -- and there are two reasons for that. The first one is that these are very difficult (or even impossible) to implement on the SAMD family of microcontrollers, which is currently the main target of CircuitPython -- the HAL libraries simply don't have such calls exposed, and it's even possible that the peripheral doesn't yield to such low level control. The second one is that Adafruit doesn't really care about you having full control over the peripheral and the underlying protocol -- they only care about "correct" use of the I2C protocol, so that you can write drivers for all of the Adafruit's stuff. If some weird chinese chip, (like TM1640, for example) implements some broken version of "almost I2C", then the CircuitPython's answer is simple: bit-bang it. And I can see the merit in this approach, especially considering the limited resources that Adafruit has.

Now, why are there no really high-level functions, like the mem_* methods, etc.? That's because CircuitPython's idea is to make as much as the code as possible available in Python, and it implements I2CDevice and I2CRegisters libraries, written in Python, that abstract away a lot of the boilerplate that you need to write for drivers, letting you focus on the actual specifics of the device. Writing a driver often just comes down to creating a class with the right register definitions -- and you are done. This is not only much easier to write, but also easier for the users to read and understand, and easier to write the documentation for.

And you know the best part? If you wanted, you could write the same I2CDevice and i2CRegisters libraries for MicroPython's machine API, and suddenly all the CircuitPython drivers will just magically work. It's just that nobody really cares about this, so nobody bothered.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: A Tale of Two APIs

Post by pythoncoder » Sun Jun 04, 2017 9:37 am

Where can I find source and docs for the I2CDevice and i2CRegisters libraries? So far all I've found is docs/design_guide.rst.
Peter Hinch
Index to my micropython libraries.

User avatar
deshipu
Posts: 1388
Joined: Thu May 28, 2015 5:54 pm

Re: A Tale of Two APIs

Post by deshipu » Sun Jun 04, 2017 10:13 am

https://github.com/adafruit/Adafruit_Ci ... _BusDevice
https://github.com/adafruit/Adafruit_Ci ... n_Register

The docs are in the readme and in docstrings inside the code. If you look for "CircuitPython" in the Adafruit's Github, you will also find some drivers that use those libraries.

Personally I'm not yet happy with how they work -- every register keeps its own buffer, and while it's usually just a few bytes, it does fragment the memory and add up. I'm working on a version that would just keep a common buffer for all registers of a particular device.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: A Tale of Two APIs

Post by pythoncoder » Mon Jun 05, 2017 10:09 am

Thanks for that. There are some clever techniques in use there, but I'm unconvinced of the merits of adding a layer of abstraction into device drivers. Using the adafruit_register class, the code

Code: Select all

my_radio.transmit=1
hides things going on underneath: a device read, a bit shift, an OR operation, and a device write. Arguably the low level details should be transparent in case of hardware or timing issues.

Writing device drivers is never going to be beginner-friendly. For a start it requires the ability to read hardware and API datasheets. It often requires a knowledge of binary, BCD and one's and two's complement notations. An understanding of the use of bit shifting and logical operations is needed. I don't think an abstraction layer negates these requirements.

The use of bus locking is interesting: presumably they intend their drivers to be thread-safe. As a lily-livered user of cooperative scheduling this strikes me as brave.
Peter Hinch
Index to my micropython libraries.

User avatar
deshipu
Posts: 1388
Joined: Thu May 28, 2015 5:54 pm

Re: A Tale of Two APIs

Post by deshipu » Mon Jun 05, 2017 10:16 am

I agree about abuse of properties for things that have side effects and take time -- and I voiced my concerns about it. They are aware of the problem, but they think that for their use case the (seeming) simplicity is worth it, especially since the drivers will be written by experienced programmers and consumed by beginners. I still think they are wrong about it, but it's their API.

As for locking, this may be actually quite useful even without threads, if you consider things like DMA or several devices on a single SPI bus.

Post Reply