Need some help how to distribute features between main.py and scripts on sd card
Need some help how to distribute features between main.py and scripts on sd card
I am trying to find a sensible way to distribute features between main.py and other scripts on sd card. My application is data logger. In doing so, I need a few things. Here is what I came up with:
Flash
• boot.py (don't plan on adding anything here in case it gets messed up)
• main.py (decide which script to run depending on a button state)
• Setup sketches (has various functions to start sd, network, uart, pins etc.)
SD card
• Library modules such as DS3231 from Peter Hinch
• Data logging sketches
• Config file for logging parameters
• Config file for SSID and PASS for both AP and STA and user/pass for FTP and Telnet
Network
• FTP, Telnet
• in STA mode use HTTP socket to upload data
UART
• SDI-12
• XBEE
Pins
• Controls power to sensors
• Push button to determine AP (starts essentials and drop to REPL) or STA (starts and logs right away)
I am not sure how to do this correctly. Here are a few issues I have right away:
• If I import machine in main, then define a few pins, sense one pin for mode, toggle another pin to supply power, then I don't know how to pass the pins to the logging script. Should I import machine again in the logging script and redefine the pins?
• If I start serial port in main, then I can't start it with machine.UART() anymore in logging script. Otherwise the processor would go into guru meditation error or something. Again I don't know how to pass the serial object from main to logging script.
So the most important thing that I don't know whether can be done is passing objects from main to scripts I run. The import is really hard to understand. If I import say X, then code within X runs in a separate name space. Only after it ends, do objects in X get imported into the main name space such as in REPL. Then if I import another thing Y, Y can't access the main namespace until it ends. It's really strange. I wish import would be like # include.
Flash
• boot.py (don't plan on adding anything here in case it gets messed up)
• main.py (decide which script to run depending on a button state)
• Setup sketches (has various functions to start sd, network, uart, pins etc.)
SD card
• Library modules such as DS3231 from Peter Hinch
• Data logging sketches
• Config file for logging parameters
• Config file for SSID and PASS for both AP and STA and user/pass for FTP and Telnet
Network
• FTP, Telnet
• in STA mode use HTTP socket to upload data
UART
• SDI-12
• XBEE
Pins
• Controls power to sensors
• Push button to determine AP (starts essentials and drop to REPL) or STA (starts and logs right away)
I am not sure how to do this correctly. Here are a few issues I have right away:
• If I import machine in main, then define a few pins, sense one pin for mode, toggle another pin to supply power, then I don't know how to pass the pins to the logging script. Should I import machine again in the logging script and redefine the pins?
• If I start serial port in main, then I can't start it with machine.UART() anymore in logging script. Otherwise the processor would go into guru meditation error or something. Again I don't know how to pass the serial object from main to logging script.
So the most important thing that I don't know whether can be done is passing objects from main to scripts I run. The import is really hard to understand. If I import say X, then code within X runs in a separate name space. Only after it ends, do objects in X get imported into the main name space such as in REPL. Then if I import another thing Y, Y can't access the main namespace until it ends. It's really strange. I wish import would be like # include.
Re: Need some help how to distribute features between main.py and scripts on sd card
OK, I am coming up with an idea. Currently my logger script has both functions and code. If I turn it entirely to functions, and call the logging function after I import the logger script, then there is no need to let the code know the namespace of main while import runs the code. I will let main handle all hardware related imports as well.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Duck Typing
Python uses Duck Typing. It is normal to do things like:
my_module only needs to import machine if "machine" appears in its sourcecode. The module can feely call I2C methods against the passed I2C instance: Python determines the availability of methods at runtime.
As a general piece of advice it's best to keep main.py minimal: typically just a single import statement. This facilitates debugging. Start off with an empty main.py, substituting the one-liner when you want to automatically run your script on boot.
Code: Select all
import machine
i2c = machine.I2C(1)
import my_module
my_module.run(i2c)
As a general piece of advice it's best to keep main.py minimal: typically just a single import statement. This facilitates debugging. Start off with an empty main.py, substituting the one-liner when you want to automatically run your script on boot.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
-
- Posts: 169
- Joined: Fri Aug 19, 2016 11:55 am
Re: Need some help how to distribute features between main.py and scripts on sd card
I think what’s confusing you is that you think of your Python files as scripts. They are not (apart from boot.py and main.py), they are modules. Defining functions in one module and calling them from another, as you propose, is the right way to go, and as pythoncoder shows, you can pass them anything they need. Since you don’t have great control over when a module runs (at first import), it typically shouldn’t do anything else than define functions and classes, and possibly initialize some things. Any code that needs to run at a determined time should be inside those functions and classes.
-
- Posts: 847
- Joined: Mon Nov 20, 2017 10:18 am
Re: Need some help how to distribute features between main.py and scripts on sd card
Not sure how pythonic this is but here is what I do.
I make a script of my program i.e my_program.py
When I am developing it I just type import my_program into the repl to run the script. Once I make a change to the script while developing it I then either press control-D to soft reset then type import my_program again or just type reload(my_progam) without needing to press control-D.
In my_progam.py I always wrap the main program in a try/finally statements so that I can put all my clean-up code in the finally other wise reload(my_program) will run into problems when trying to re-init resources that wasn't released from last execution.
All my main function I tend to peel off into modules that my_program can import (remember to del these modules in finally).
Once my_program.py has finished development and I want it to auto load on boot-up I add improt my_program at last line of main.py then anytime I want to stop it auto loading on boot-up I just comment out the import my_program in main.py
I make a script of my program i.e my_program.py
When I am developing it I just type import my_program into the repl to run the script. Once I make a change to the script while developing it I then either press control-D to soft reset then type import my_program again or just type reload(my_progam) without needing to press control-D.
In my_progam.py I always wrap the main program in a try/finally statements so that I can put all my clean-up code in the finally other wise reload(my_program) will run into problems when trying to re-init resources that wasn't released from last execution.
All my main function I tend to peel off into modules that my_program can import (remember to del these modules in finally).
Once my_program.py has finished development and I want it to auto load on boot-up I add improt my_program at last line of main.py then anytime I want to stop it auto loading on boot-up I just comment out the import my_program in main.py
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Need some help how to distribute features between main.py and scripts on sd card
@OutoftheBOTS_ For the benefit of the OP (@liudr) I would point out that reload is not natively supported but is provided by code in main.py.
As to your workflow it depends on how you perform the edit/upload part of the cycle. I guess you are using the mass storage interface. This can be problematic so I turn off MSC and use rshell. After a run I drop back to the rshell prompt, edit the code on my PC then upload it and run it again. This causes a soft reset before it runs, ensuring a clean start.
You can make rshell copy the file(s) to the target, invoke the REPL and run the code in a single line, so all you have to do in each cycle is recall the previous line. I suggest you try it - it's very slick
So I've found no need for reload or for try-finally code.
As to your workflow it depends on how you perform the edit/upload part of the cycle. I guess you are using the mass storage interface. This can be problematic so I turn off MSC and use rshell. After a run I drop back to the rshell prompt, edit the code on my PC then upload it and run it again. This causes a soft reset before it runs, ensuring a clean start.
You can make rshell copy the file(s) to the target, invoke the REPL and run the code in a single line, so all you have to do in each cycle is recall the previous line. I suggest you try it - it's very slick
So I've found no need for reload or for try-finally code.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
-
- Posts: 847
- Joined: Mon Nov 20, 2017 10:18 am
Re: Need some help how to distribute features between main.py and scripts on sd card
I am mainly using MP on ESP32 with Loboris port so I do all my editing via a FTP server. I setup in filezilla that it will open *.py with my selected IDE (note++ ha ha) then I just edit it and click save and filezilla will auto update it over FTP then I just swap to my puTTY terminal window(either a serial connection or telnet server) and press up arrow then enter (to re-run the last command reload(my_program) )
Makes for fast development.
Makes for fast development.
Re: Need some help how to distribute features between main.py and scripts on sd card
Thanks. I didn't know about reload. Will research it since I can't re-import without errors coming up. I'm using Mobaxterm for FTP and Telnet so it's not as automatic as your filezilla. I'll try filezilla with npp sometime. I have both npp and a few other editors set up with my preferred color schemes.OutoftheBOTS_ wrote: ↑Sun Aug 19, 2018 10:25 amI am mainly using MP on ESP32 with Loboris port so I do all my editing via a FTP server. I setup in filezilla that it will open *.py with my selected IDE (note++ ha ha) then I just edit it and click save and filezilla will auto update it over FTP then I just swap to my puTTY terminal window(either a serial connection or telnet server) and press up arrow then enter (to re-run the last command reload(my_program) )
Makes for fast development.
[EDIT] Regarding ctro-D, it doesn't work in my case and causes lockups. I'll post the error message once I'm back working on the system. I also should try machine.reset(), if the system is not locking up.
Last edited by liudr on Sun Aug 19, 2018 9:47 pm, edited 1 time in total.
-
- Posts: 847
- Joined: Mon Nov 20, 2017 10:18 am
Re: Need some help how to distribute features between main.py and scripts on sd card
To use reload you have to create the fuction first as it isn't native in MP.
I place this code in my boot.py
I place this code in my boot.py
Code: Select all
import gc, sys
def reload(mod):
mod_name = mod.__name__
del sys.modules[mod_name]
gc.collect()
return __import__(mod_name)
Re: Duck Typing
Thanks for the pointer! I have just read about duck typing and found it interesting, coming from a static typing language such as C/C++. So I just don't expose any machine-related modules to my logging script, but passing the relevant objects such as the pin, the serial object etc. Then with duck typing, these object can invoke the necessary methods for logging. I'll keep a oneliner main.py like I did before, importing the script, within which I was doing but didn't know if that should be recommended.pythoncoder wrote: ↑Sat Aug 18, 2018 8:46 amPython uses Duck Typing. It is normal to do things like:my_module only needs to import machine if "machine" appears in its sourcecode. The module can feely call I2C methods against the passed I2C instance: Python determines the availability of methods at runtime.Code: Select all
import machine i2c = machine.I2C(1) import my_module my_module.run(i2c)
As a general piece of advice it's best to keep main.py minimal: typically just a single import statement. This facilitates debugging. Start off with an empty main.py, substituting the one-liner when you want to automatically run your script on boot.
Then in the script that main.py imports, it will import all system modules, and possibly my logging module, which will be all functions, no code outside functions.
What if I need some function calls such as os.chdir() in my logging function? Should I pass os.chdir to the logging function?