RFC: How to Test MicroPython

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
djantzen
Posts: 3
Joined: Sat Nov 27, 2021 10:30 pm

RFC: How to Test MicroPython

Post by djantzen » Fri May 13, 2022 11:06 pm

Hi All,

I decided to explore firmware, Micropython and the Pico MCU several months ago. Coming from a pure software background I wanted to have automated test coverage for as much as possible, so I started working with UnitTest. I further wanted: 1) to run tests in MP, not in CPython to minimize divergence 2) have tests run in Continuous Integration 3) Not have have to deploy to the Pico and do manual QA, or at least to do this only for final integration testing.

Right away I ran into the problem that the machine module was not defined on my Ubuntu dev environment, and so I stubbed out the main classes/methods. This seems like a common problem to anyone using UnitTest, for example in the PicoCat project which also emphasizes testing. Doing that much allows one to write unit tests around logic that doesn't do any IO. I wanted to take it further however and start validating that pins were being controlled in the correct way. That led me to mock out a sizeable chunk of the machine module in ways that would capture IO traffic on the pins and support test assertions.

Here's 'machine': https://github.com/djantzen/pico_book/b ... machine.py
Here's a simple example blinking an LED: https://github.com/djantzen/pico_book/b ... er_test.py
Here's a test of an LCD screen: https://github.com/djantzen/pico_book/b ... cd_test.py
Here's an I2C test for a temperature sensor: https://github.com/djantzen/pico_book/b ... x0_test.py
Here's an SPI test for a ESP32 wifi co-processor: https://github.com/djantzen/pico_book/b ... ol_test.py

All of the tests run now locally before I push code to the Pico, and in a GitHub Workflow after a commit and push.

My question is: does anyone else find this useful or is it just me wanting to do this kind of testing during development??

I'd love to hear some feedback from more seasoned firmware developers on this approach.

Thank you for your time,
David

ChrisO
Posts: 48
Joined: Mon Apr 06, 2020 6:16 pm

Re: RFC: How to Test MicroPython

Post by ChrisO » Sat May 14, 2022 7:48 pm

Hi Djantzen,

I've recently started experimenting with this as well and I think your desire to reduce the feedback loop and test ASAP resonates with mine.
Instead of mocking Machine code, I opted to extract important code into functions which do not interact with MP specific libraries.

In my case I would have a very simple function to manage an actuator:

Code: Select all

while self.running:
    next_state = await important_function_I_want_to_test(someSensorInput, someOtherSensorsInput)
    if next_state = 'a':
        somePin.value(1)
    #.... etc.
This allows me to write a lot of tests for that important function with a lot branches, without needing any Machine lib.
I'm definitely not a seasoned micropython developer, but I hope it helps anyway.

Chris
Chris

ChrisO
Posts: 48
Joined: Mon Apr 06, 2020 6:16 pm

Re: RFC: How to Test MicroPython

Post by ChrisO » Sat May 14, 2022 7:54 pm

additionally, a lot of micro-libs are really subsets of their CPython variants so you can also quite easily patch those in your unit tests:

I have a file called monkeypatch.py which is the first thing I import in my unittests and for the most part it works well.

Code: Select all

# replace some libs with their c-python 'equivalents'
sys.modules['uasyncio'] = __import__('asyncio')
sys.modules['ucollections'] = __import__('collections')
sys.modules['utime'] = __import__('time')
sys.modules['ujson'] = __import__('json')
sys.modules['urandom'] = __import__('random')
sys.modules['ubinascii'] = __import__('binascii')
sys.modules['uerrno'] = __import__('errno')
sys.modules['ure'] = __import__('re')
sys.modules['ustruct'] = __import__('struct')
...
# mock some that don't have c-python versions.
sys.modules['network'] = MagicMock()

# this works for any import written as `from micropython import const`
sys.modules['micropython.const'] = lambda x : x
Chris

djantzen
Posts: 3
Joined: Sat Nov 27, 2021 10:30 pm

Re: RFC: How to Test MicroPython

Post by djantzen » Sun May 15, 2022 1:40 am

Hi Chris, I definitely agree with the approach of breaking logic out into functions that can be tested independently of anything MP or 'machine' specific. Makes the code more understandable and maintainable as well.

Early on in my development using MP I ran into some compatibility issues with code that ran in CPython but failed on the Pico, so I've been focused on using MP exclusively. I can see the value in your approach however, especially since it let's you use MagicMock. I think porting MagicMock or FlexMock to MP would be a huge benefit, although it would require a lot of work, like implementing the "types" and "typing" modules for starters.

ChrisO
Posts: 48
Joined: Mon Apr 06, 2020 6:16 pm

Re: RFC: How to Test MicroPython

Post by ChrisO » Sun May 15, 2022 8:18 am

My apologies, I now see that my replies were not really useful. I misunderstood part of your post.
I have not felt the need to take it a step further. For me there was no need to mock Machine code due to breaking out logic as discussed, but I would like to be able to mock the network module and limited memory.
Mqtt and network recovery + running up against memory limits can be quite the balancing act. Hopefully someone else is able to chime in with a more useful response ;)
Chris

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: RFC: How to Test MicroPython

Post by stijn » Sun May 15, 2022 4:18 pm

ChrisO wrote:
Sat May 14, 2022 7:54 pm
# replace some libs with their c-python 'equivalents'
sys.modules['uasyncio'] = __import__('asyncio')
...
Perhaps you're aware already, but a while ago MicroPython impleneted 'automatic discovery' of any module prefixed with a u, so the idea is all code simply can do 'import asyncio' as usual and if MicroPython doesn't find the module it will look for 'uasyncio' automatcially, and hence imports are CPython-compatible (not necessarily feature-wise though) so if that is used everywhere the monkey-patching for modules is not needed.

User avatar
scruss
Posts: 360
Joined: Sat Aug 12, 2017 2:27 pm
Location: Toronto, Canada
Contact:

Re: RFC: How to Test MicroPython

Post by scruss » Sun May 15, 2022 4:34 pm

It does look useful, but for me, MicroPython tends to fail on something one of the hardware sensors didn't quite complete: like DHT sensors that almost always work, but for no discernable reason, sometimes don't.

Also, MicroPython on real hardware tends to have quite narrow floating point, which might cause passing code in test to do something quite other on real hardware.

djantzen
Posts: 3
Joined: Sat Nov 27, 2021 10:30 pm

Re: RFC: How to Test MicroPython

Post by djantzen » Sun May 15, 2022 5:39 pm

Hi scruss,

Same here, intermittent failures on my AHT20 and SPI errors talking to the ESP32. I have way too many try/except clauses scattered around for my liking, and now I need to reorganize my code so that I can catch problems at the right level in the call stack to retry the operation properly

ChrisO
Posts: 48
Joined: Mon Apr 06, 2020 6:16 pm

Re: RFC: How to Test MicroPython

Post by ChrisO » Mon May 16, 2022 8:05 am

stijn wrote:
Sun May 15, 2022 4:18 pm
ChrisO wrote:
Sat May 14, 2022 7:54 pm
# replace some libs with their c-python 'equivalents'
sys.modules['uasyncio'] = __import__('asyncio')
...
Perhaps you're aware already, but a while ago MicroPython impleneted 'automatic discovery' of any module prefixed with a u, so the idea is all code simply can do 'import asyncio' as usual and if MicroPython doesn't find the module it will look for 'uasyncio' automatcially, and hence imports are CPython-compatible (not necessarily feature-wise though) so if that is used everywhere the monkey-patching for modules is not needed.
Thank you Stijn, I was not yet aware of this new feature.
Chris

Post Reply