[Solved] Noob has questions about exec() and import()

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
User avatar
sebi
Posts: 48
Joined: Tue Mar 29, 2016 9:36 pm
Location: France

[Solved] Noob has questions about exec() and import()

Post by sebi » Tue Mar 26, 2019 11:32 pm

Dear all,

My problem has to do with exec, import and globals
For ease of illustration, I create a file named code.py, the content of which being:

Code: Select all

class A:
    pass

a=A()
a.b=1

def toggle():
    a.b*=-1

toggle()
When I type the following commands under MicroPython's prompt I get:

Code: Select all

MicroPython v1.10-131-g952139904 on 2019-02-26; PYBv1.1 with STM32F405RG
Type "help()" for more information.
>>> def do(): exec(open("code.py").read())
...
>>> do()
>>> dir()
['__name__', 'toggle', 'a', 'do', 'A']
>>> a.b
-1
>>>
...which is what I expect, and provides the same results as when typing all the lines of code.py directly under MicroPython's prompt.

However, if I create a file named make.py, the content of which being:

Code: Select all

def do():
    exec(open("code.py").read())

def do2():
    exec(open("code.py").read(), globals())
and I type the following commands under MicroPython's prompt, I get:

Code: Select all

MicroPython v1.10-131-g952139904 on 2019-02-26; PYBv1.1 with STM32F405RG
Type "help()" for more information.
>>> from make import do
>>> do()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "make.py", line 1, in do
  File "<string>", line 10, in <module>
  File "<string>", line 8, in toggle
NameError: name 'a' isn't defined
>>> dir()
['__name__', 'toggle', 'a', 'do', 'A']
>>>
Q: Why do I get this error even thought a is obviously in the global namespace?

If I restart MicroPython and type:

Code: Select all

MicroPython v1.10-131-g952139904 on 2019-02-26; PYBv1.1 with STM32F405RG
Type "help()" for more information.
>>> from make import do2
>>> do2()
>>> dir()
['__name__', 'do2']
>>> a.b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'a' isn't defined
>>>
...which is not what I am looking for, as I would like a.b=-1.

Q: Could someone help me understand the underlying mechanisms of exec and import?

PS: I have noticed that def func(): exec('a=2') under MicroPython does the same job as def func(): exec('a=2', globals()) under CPython 3: executing func() will create a variable a, the value of which is 2, in the global namespace.

Q: Why globals() can be omitted under MicroPython for such an example?
Last edited by sebi on Sat Apr 06, 2019 6:37 pm, edited 1 time in total.

OutoftheBOTS_
Posts: 847
Joined: Mon Nov 20, 2017 10:18 am

Re: Noob has questions about exec() and import()

Post by OutoftheBOTS_ » Tue Mar 26, 2019 11:49 pm

Hopefully I can help and give correct answers and if I am incorrect someone will correct me :)

First it is a little bit of a round about way you loading into RAM your code from code.py the following will do the same thing

Code: Select all

import code
also I think your getting namespace incorrect

Code: Select all

MicroPython v1.10-131-g952139904 on 2019-02-26; PYBv1.1 with STM32F405RG
Type "help()" for more information.
>>> from make import do2
>>> do2()
>>> dir()
['__name__', 'do2']
>>> a.b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'a' isn't defined
maybe try this name space because of the way you have imported it

Code: Select all

do2.a.b

User avatar
sebi
Posts: 48
Joined: Tue Mar 29, 2016 9:36 pm
Location: France

Re: Noob has questions about exec() and import()

Post by sebi » Wed Mar 27, 2019 2:07 am

Thanks for your reply OutoftheBOTS_

Code: Select all

MicroPython v1.10-131-g952139904 on 2019-02-26; PYBv1.1 with STM32F405RG
Type "help()" for more information.
>>> from make import do2
>>>
__class__       __name__        do2
>>> do2.__
__class__       __name__
>>> do2.a.b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute 'a'
>>> import code
>>> code.a.b
-1
>>> code.toggle()
>>> code.a.b
1
>>> from code import *
>>> a.b
-1
>>> toggle()
>>> a.b
1
do2.a.b doesn't do the job.
import does exactly what I want, but only once...
If I modifiy code.py with, let's say, a.b=11, save the file, and type:

Code: Select all

>>> from code import *
>>> a.b
1
>>> import code
>>> code.a.b
1
>>> from make import reload
>>> reload(code)
<module 'code' from 'code.py'>
>>> code.a.b
1
>>> from make import run
>>> run("code")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "make.py", line 13, in run
  File "<string>", line 10, in <module>
  File "<string>", line 8, in toggle
NameError: name 'a' isn't defined
>>> exec(open("code.py").read())
>>> a.b
-11
...import doesn't take into accounts the changes I have made to code.py.
Neither reload.
and run yields the error I have described in my original post.
I added the following code to make.py:

Code: Select all

def reload(mod):
    import sys
    mod_name = mod.__name__
    del sys.modules[mod_name]
    return __import__(mod_name)

def run(file):
    """ Run given file (without extension) """
    exec(open(file + ".py").read())
where reload is taken from one of your posts.
Only exec(open("code.py").read()) does the job correctly!
My goal would be to have run("code") doing the very same as exec(open("code.py").read()).
Do you know how to do that?

OutoftheBOTS_
Posts: 847
Joined: Mon Nov 20, 2017 10:18 am

Re: Noob has questions about exec() and import()

Post by OutoftheBOTS_ » Wed Mar 27, 2019 9:30 pm

I add the the reload function to my boot.py so that it sets up during boot up
here is my boot.py

Code: Select all

# This file is executed on every boot (including wake-boot from deepsleep)
import sys, gc

sys.path[1] = '/flash/lib'

def reload(mod):
  mod_name = mod.__name__
  del sys.modules[mod_name]
  gc.collect()
  return __import__(mod_name)
This has the effect of removing the module from RAM then reloading it.

Now
I am in process of moving house so can't run anythiong on a MP board to test atm but try this.

create your code.py

Code: Select all

class A:
    pass

a=A()
a.b=1

def toggle():
    a.b*=-1

toggle()
load the code.py into ram

Code: Select all

import code
try your tests

Code: Select all

dir()
a.b
toggle()
a.b

User avatar
sebi
Posts: 48
Joined: Tue Mar 29, 2016 9:36 pm
Location: France

Re: Noob has questions about exec() and import()

Post by sebi » Wed Mar 27, 2019 10:39 pm

Dear OutoftheBOTS_,

Thanks for your assistance.
I followed your instructions:
- modified boot.py with your code
- rebooted the MicroPython device
- loaded the content of code.py into RAM using import
- printed the value of code.a.b
- modified the value of a.b in code.py
- exectued reload(code)
- printed the value of code.a.b

--> I still got its original value (the one before modification).

I don't understand it but it seems reload doesn't do its job properly.

Can you confirm once you have some more free time?

Details below:

Code: Select all

MicroPython v1.10-8-g8b7039d7d on 2019-01-26; ESP module with ESP8266
Type "help()" for more information.
>>> import code
>>> code.a.b
-1
>>>
Uploading project (main folder)...
Not safe booting, disabled in settings

Reading file status
[1/1] Writing file code.py (0kb)
Upload done

>>> reload(code)
<module 'code'>
>>> code.a.b
-1
>>>

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Noob has questions about exec() and import()

Post by dhylands » Wed Mar 27, 2019 11:05 pm

The problem is that there is a local variable called code which is holding onto the old copy of code.

When you reload(code) this can't change the local variable.

After you call reload(code) you can then do:

Code: Select all

del code
import code
and you'll now have a reference to the new code.

The proper way use reload is to to this:

Code: Select all

code = reload(code)
this will replace the local variable code with the newly imported version of code.

User avatar
sebi
Posts: 48
Joined: Tue Mar 29, 2016 9:36 pm
Location: France

Re: Noob has questions about exec() and import()

Post by sebi » Thu Mar 28, 2019 3:36 am

Dear Dave,
Thanks a lot for your explanations!
It really gave me a lot to think of, especially that notion of local variable holding the code.
I searched for a way to get reload(mod) modify that variable directly, and eventually I modified its last line such a way:

Code: Select all

def reload(mod):
    import sys
    mod_name = mod.__name__
    del sys.modules[mod_name]
    exec("import {}".format(mod_name))
...and it seems to work.
Typing reload(code) only does the job:
When updating code.py, modifications are reflected.
I will still investigate exec() which is still mysterious to me, but I thank you a lot for your help!

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Noob has questions about exec() and import()

Post by dhylands » Thu Mar 28, 2019 4:09 am

Fundamentally, doing the exec won't work for all situations.

You really need to do code = reload(code). That's the only way to cover all of the edge cases. The exec case only works if the original code variable is a global. If it isn't a global, then it won't do what you want it to.

User avatar
sebi
Posts: 48
Joined: Tue Mar 29, 2016 9:36 pm
Location: France

Re: Noob has questions about exec() and import()

Post by sebi » Sun Mar 31, 2019 8:45 pm

Indeed the only good way is code = reload(code).

Is there a similar way to reload a module after a from code import *?

Actually, currently I prefer to use exec(open("code.py").read()) instead as I don't understand the program behavior after a from code import *.
Indeed, for code.py being:

Code: Select all

a = 0
def func():
    global a
    a = 1
if I type from code import *; func(); print(a) I get 0 instead of 1 as I would have expected. Why?

OutoftheBOTS_
Posts: 847
Joined: Mon Nov 20, 2017 10:18 am

Re: Noob has questions about exec() and import()

Post by OutoftheBOTS_ » Sun Mar 31, 2019 9:28 pm

First of all how about you tell us what your trying to achieve?

If it is simply writing code into a py file (i.e myscript_file.py) then loading it into RAM instead of having to type the script into the REPL then you can simply just use

Code: Select all

import myscript_file
This works perfectly for me :)

Post Reply