MicroPython firmwares with "'MICROPY_PY_REVERSE_SPECIAL_METHODS'" enabled ?

[rcolistete]

The flag "MICROPY_PY_REVERSE_SPECIAL_METHODS" seems to enabled only on Unix, Windows ports of MicroPython.

This feature is useful to allow easier implementation of operations with different types not depending on the order, like in ulab module :
>>> np.arange(10) + 1
>>> 1 + np.arange(10) // only works with "MICROPY_PY_REVERSE_SPECIAL_METHODS" enabled
MicroPython modules, like upyuncertainties module, would also profit from this feature.

I've built some Pycom MicroPython firmwares with "'MICROPY_PY_REVERSE_SPECIAL_METHODS'" enabled. The difference in firmware size is small :
- "WiPy_ulab_sp_thread_v1.20.2.rc10-g87b47d16e_2020-08-06" has 'text' with 1.411.288 bytes;
- without the flag, 'text' has 1.411.256;
- difference = 32 bytes.

So my question : why "MICROPY_PY_REVERSE_SPECIAL_METHODS" is not enabled by default for stm32, esp32, etc ? Any drawbacks ?

[pythoncoder]

Have you checked code size?

The plethora of build options and their presence or absence on different platforms makes writing cross-platform code increasingly difficult. If someone asks whether ulab will run on some arbitrary platform how do you set about answering?

[rcolistete]

Building on Pycom MicroPython non-PyBytes ulab SP version for LOPY, it shows in the end :

Code: Select all

...
  text	   data	    bss	    dec	    hex	filename
1470792	 227600	  61548	1759940	 1adac4	build/LOPY/release/application.elf
...
1720592 bytes => Size OK
...

when the firmware size is ok.
Building Pycom MicroPython PyBytes ulab DP version for LOPY :

Code: Select all

...
   text	   data	    bss	    dec	    hex	filename
1497072	 316024	  61548	1874644	 1c9ad4	build-PYBYTES/LOPY/release/application.elf
...
1835296 bytes => Firmware image size exceeds avialable space on board!
...

when the firmware size doesn't fit the firmware partition.

[rcolistete]

Any comments about drawbacks of "MICROPY_PY_REVERSE_SPECIAL_METHODS" ?

Otherwise I'll always use "MICROPY_PY_REVERSE_SPECIAL_METHODS" in my new firmware builds for Pyboard's, ESP8266, ESP32, Pycom, etc.

[dhylands]

Most features like that have flags just because the code takes up space that smaller devices might not want.

You'll generally find that these options are on by default for the unix build https://github.com/micropython/micropyt ... ort.h#L103 and the stm32 builds (although I see this particular flag isn't enabled in the stm32 builds).

[pythoncoder]

I've just run into this "feature" writing a numeric (quaternion) class.

It is another instance of a general problem: it is almost impossible to write cross-platform MicroPython code because language features vary randomly between platforms in a manner which is entirely undocumented.

I raised this RFC but it has provoked little discussion.

[pythoncoder]

Thinking about this some more, the absence of these special methods is problematic, especially __rtruediv__. If you have a class Foo representing some numeric type, you might ask users to put other types on the RHS:

Code: Select all

f = Foo()
g = f + 4  # uses normal special method
h = 3 - f  # Ask users to avoid this

How do you avoid this:

Code: Select all

x = 7/f  # Calls __rtruediv__()

To save users from compiling a special build, I guess you'd have to implement an inverse() method. This results in revolting syntax like

Code: Select all

y = f.inverse() * 7

This would soon get ugly in an expression of any complexity.

[v923z]

...
when the firmware size doesn't fit the firmware partition.

This problem can be helped in a number of ways.

1. In ulab, you can exclude sub-modules. There are some that are somewhat obscure (e.g., approx, or filter, or compare), and might not hold general appeal. They are there, but it doesn't mean that you need them in all applications.

2. If you want, I can make exclusion customisable at the function level. E.g., vectorise contains a huge number of functions that you are almost never going to use. How many times would you need sinh, or erf on a microcontroller?

3. I am completely re-writing the code (see your PM). In the new version, you can choose the number of dimensions. 1D is quite small now. So, if flash space is an issue, you can compile with 1D arrays, which are still useful for many applications.

4. Many functions accept generic iterables as well as ndarrays as their arguments. We could think about conditionally throwing away the iterable arguments, so that

Code: Select all

import ulab
from ulab import numerical
numerical.std([1, 2, 3])

would no longer be valid. You could still calculate the mean of a list by casting first, like

Code: Select all

import ulab
from ulab import numerical
numerical.std(ulab.array([1, 2, 3]))

This would require more RAM (you would have to hold the intermediate storage ulab.array([1, 2, 3])), but you wouldn't lose functionality.

5. We could conditionally remove the vectorize function https://micropython-ulab.readthedocs.io ... -functions. You only get a factor of two in speed compared to python.

6. When working on the new features, I realised that associative operators (see my reply to @pythoncoder below) allow for flash savings. https://github.com/v923z/micropython-ul ... ay.c#L1208 We could implement those in the old version, though, I would rather work on the new.

These are just a couple of possibilities.

Instead of compiling all possible combinations, I am wondering, whether one could perhaps turn this problem around, and allow users to configure their own firmware. You would need some sort of service that compiles the firmware, and you could choose, what you want to include.

[v923z]

Thinking about this some more, the absence of these special methods is problematic, especially __rtruediv__. If you have a class Foo representing some numeric type, you might ask users to put other types on the RHS:

Code: Select all

f = Foo()
g = f + 4  # uses normal special method
h = 3 - f  # Ask users to avoid this

How do you avoid this:

Code: Select all

x = 7/f  # Calls __rtruediv__()

To save users from compiling a special build, I guess you'd have to implement an inverse() method. This results in revolting syntax like

Code: Select all

y = f.inverse() * 7

This would soon get ugly in an expression of any complexity.

I think the problem is more severe: there are only four associative operators, namely *, +, ==, and !=. All others suffer from the problem that you describe.

By exploiting the symmetries of associative operators, I could actually save some flash space in the new version.

[pythoncoder]

That's a very good point. It calls into question whether there is much merit in MicroPython supporting only this subset.