Need some help how to distribute features between main.py and scripts on sd card

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
User avatar
liudr
Posts: 211
Joined: Tue Oct 17, 2017 5:18 am

Need some help how to distribute features between main.py and scripts on sd card

Post by liudr » Sat Aug 18, 2018 12:48 am

I am trying to find a sensible way to distribute features between main.py and other scripts on sd card. My application is data logger. In doing so, I need a few things. Here is what I came up with:

Flash
• boot.py (don't plan on adding anything here in case it gets messed up)
• main.py (decide which script to run depending on a button state)
• Setup sketches (has various functions to start sd, network, uart, pins etc.)
SD card
• Library modules such as DS3231 from Peter Hinch
• Data logging sketches
• Config file for logging parameters
• Config file for SSID and PASS for both AP and STA and user/pass for FTP and Telnet
Network
• FTP, Telnet
• in STA mode use HTTP socket to upload data
UART
• SDI-12
• XBEE
Pins
• Controls power to sensors
• Push button to determine AP (starts essentials and drop to REPL) or STA (starts and logs right away)

I am not sure how to do this correctly. Here are a few issues I have right away:
• If I import machine in main, then define a few pins, sense one pin for mode, toggle another pin to supply power, then I don't know how to pass the pins to the logging script. Should I import machine again in the logging script and redefine the pins?
• If I start serial port in main, then I can't start it with machine.UART() anymore in logging script. Otherwise the processor would go into guru meditation error or something. Again I don't know how to pass the serial object from main to logging script.

So the most important thing that I don't know whether can be done is passing objects from main to scripts I run. The import is really hard to understand. If I import say X, then code within X runs in a separate name space. Only after it ends, do objects in X get imported into the main name space such as in REPL. Then if I import another thing Y, Y can't access the main namespace until it ends. It's really strange. I wish import would be like # include.

User avatar
liudr
Posts: 211
Joined: Tue Oct 17, 2017 5:18 am

Re: Need some help how to distribute features between main.py and scripts on sd card

Post by liudr » Sat Aug 18, 2018 3:36 am

OK, I am coming up with an idea. Currently my logger script has both functions and code. If I turn it entirely to functions, and call the logging function after I import the logger script, then there is no need to let the code know the namespace of main while import runs the code. I will let main handle all hardware related imports as well.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Duck Typing

Post by pythoncoder » Sat Aug 18, 2018 8:46 am

Python uses Duck Typing. It is normal to do things like:

Code: Select all

import machine
i2c = machine.I2C(1)
import my_module
my_module.run(i2c)
my_module only needs to import machine if "machine" appears in its sourcecode. The module can feely call I2C methods against the passed I2C instance: Python determines the availability of methods at runtime.

As a general piece of advice it's best to keep main.py minimal: typically just a single import statement. This facilitates debugging. Start off with an empty main.py, substituting the one-liner when you want to automatically run your script on boot.
Peter Hinch
Index to my micropython libraries.

Christian Walther
Posts: 169
Joined: Fri Aug 19, 2016 11:55 am

Re: Need some help how to distribute features between main.py and scripts on sd card

Post by Christian Walther » Sat Aug 18, 2018 9:05 pm

I think what’s confusing you is that you think of your Python files as scripts. They are not (apart from boot.py and main.py), they are modules. Defining functions in one module and calling them from another, as you propose, is the right way to go, and as pythoncoder shows, you can pass them anything they need. Since you don’t have great control over when a module runs (at first import), it typically shouldn’t do anything else than define functions and classes, and possibly initialize some things. Any code that needs to run at a determined time should be inside those functions and classes.

OutoftheBOTS_
Posts: 847
Joined: Mon Nov 20, 2017 10:18 am

Re: Need some help how to distribute features between main.py and scripts on sd card

Post by OutoftheBOTS_ » Sat Aug 18, 2018 10:43 pm

Not sure how pythonic this is but here is what I do.

I make a script of my program i.e my_program.py

When I am developing it I just type import my_program into the repl to run the script. Once I make a change to the script while developing it I then either press control-D to soft reset then type import my_program again or just type reload(my_progam) without needing to press control-D.

In my_progam.py I always wrap the main program in a try/finally statements so that I can put all my clean-up code in the finally other wise reload(my_program) will run into problems when trying to re-init resources that wasn't released from last execution.

All my main function I tend to peel off into modules that my_program can import (remember to del these modules in finally).

Once my_program.py has finished development and I want it to auto load on boot-up I add improt my_program at last line of main.py then anytime I want to stop it auto loading on boot-up I just comment out the import my_program in main.py

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Need some help how to distribute features between main.py and scripts on sd card

Post by pythoncoder » Sun Aug 19, 2018 8:22 am

@OutoftheBOTS_ For the benefit of the OP (@liudr) I would point out that reload is not natively supported but is provided by code in main.py.

As to your workflow it depends on how you perform the edit/upload part of the cycle. I guess you are using the mass storage interface. This can be problematic so I turn off MSC and use rshell. After a run I drop back to the rshell prompt, edit the code on my PC then upload it and run it again. This causes a soft reset before it runs, ensuring a clean start.

You can make rshell copy the file(s) to the target, invoke the REPL and run the code in a single line, so all you have to do in each cycle is recall the previous line. I suggest you try it - it's very slick ;)

So I've found no need for reload or for try-finally code.
Peter Hinch
Index to my micropython libraries.

OutoftheBOTS_
Posts: 847
Joined: Mon Nov 20, 2017 10:18 am

Re: Need some help how to distribute features between main.py and scripts on sd card

Post by OutoftheBOTS_ » Sun Aug 19, 2018 10:25 am

I am mainly using MP on ESP32 with Loboris port so I do all my editing via a FTP server. I setup in filezilla that it will open *.py with my selected IDE (note++ ha ha) then I just edit it and click save and filezilla will auto update it over FTP then I just swap to my puTTY terminal window(either a serial connection or telnet server) and press up arrow then enter (to re-run the last command reload(my_program) )

Makes for fast development.

User avatar
liudr
Posts: 211
Joined: Tue Oct 17, 2017 5:18 am

Re: Need some help how to distribute features between main.py and scripts on sd card

Post by liudr » Sun Aug 19, 2018 5:05 pm

OutoftheBOTS_ wrote:
Sun Aug 19, 2018 10:25 am
I am mainly using MP on ESP32 with Loboris port so I do all my editing via a FTP server. I setup in filezilla that it will open *.py with my selected IDE (note++ ha ha) then I just edit it and click save and filezilla will auto update it over FTP then I just swap to my puTTY terminal window(either a serial connection or telnet server) and press up arrow then enter (to re-run the last command reload(my_program) )

Makes for fast development.
Thanks. I didn't know about reload. Will research it since I can't re-import without errors coming up. I'm using Mobaxterm for FTP and Telnet so it's not as automatic as your filezilla. I'll try filezilla with npp sometime. I have both npp and a few other editors set up with my preferred color schemes.

[EDIT] Regarding ctro-D, it doesn't work in my case and causes lockups. I'll post the error message once I'm back working on the system. I also should try machine.reset(), if the system is not locking up.
Last edited by liudr on Sun Aug 19, 2018 9:47 pm, edited 1 time in total.

OutoftheBOTS_
Posts: 847
Joined: Mon Nov 20, 2017 10:18 am

Re: Need some help how to distribute features between main.py and scripts on sd card

Post by OutoftheBOTS_ » Sun Aug 19, 2018 9:03 pm

To use reload you have to create the fuction first as it isn't native in MP.

I place this code in my boot.py

Code: Select all

import gc, sys
def reload(mod):
  mod_name = mod.__name__
  del sys.modules[mod_name]
  gc.collect()
  return __import__(mod_name)

User avatar
liudr
Posts: 211
Joined: Tue Oct 17, 2017 5:18 am

Re: Duck Typing

Post by liudr » Sun Aug 19, 2018 9:43 pm

pythoncoder wrote:
Sat Aug 18, 2018 8:46 am
Python uses Duck Typing. It is normal to do things like:

Code: Select all

import machine
i2c = machine.I2C(1)
import my_module
my_module.run(i2c)
my_module only needs to import machine if "machine" appears in its sourcecode. The module can feely call I2C methods against the passed I2C instance: Python determines the availability of methods at runtime.

As a general piece of advice it's best to keep main.py minimal: typically just a single import statement. This facilitates debugging. Start off with an empty main.py, substituting the one-liner when you want to automatically run your script on boot.
Thanks for the pointer! I have just read about duck typing and found it interesting, coming from a static typing language such as C/C++. So I just don't expose any machine-related modules to my logging script, but passing the relevant objects such as the pin, the serial object etc. Then with duck typing, these object can invoke the necessary methods for logging. I'll keep a oneliner main.py like I did before, importing the script, within which I was doing but didn't know if that should be recommended.

Then in the script that main.py imports, it will import all system modules, and possibly my logging module, which will be all functions, no code outside functions.

What if I need some function calls such as os.chdir() in my logging function? Should I pass os.chdir to the logging function?

Post Reply