Strange behaviour class variables [SOLVED]

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
User avatar
wimpie
Posts: 12
Joined: Sun Jul 05, 2020 11:14 am
Location: The Netherlands

Strange behaviour class variables [SOLVED]

Post by wimpie » Thu Jul 16, 2020 12:22 pm

The problem described below was found while using the currently latest build of MicroPython on an ESP8266. A (test) class with 3 class variables is created, as well as two instances of the test class. Two class variables are modified, but only one of the two modifications is visible to both instances.

The test class, saved in file showProblem.py, is:

Code: Select all

class showProblem():
  cvsp_lst_task = 0
  cvsp_max_task = 8
  cvsp_wait_time= [None] * cvsp_max_task

  def __init__( self ):
    self.TimePeriod= 0

  def showcv( self ):
    print( self.cvsp_lst_task, self.cvsp_max_task, self.cvsp_wait_time )

  def setcv( self, value ):
    self.cvsp_wait_time[self.cvsp_lst_task]= value
    self.cvsp_lst_task+= 1
while the main program is:

Code: Select all

import showProblem

obj0= showProblem.showProblem()
obj0.showcv()
obj1= showProblem.showProblem()
obj1.showcv()

for j in range(3):
  obj0.setcv( j )
  obj0.showcv()
  obj1.showcv()
When this script is run, the following output is generated:

Code: Select all

>>> %Run -c $EDITOR_CONTENT
0 8 [None, None, None, None, None, None, None, None]
0 8 [None, None, None, None, None, None, None, None]
1 8 [0, None, None, None, None, None, None, None]
0 8 [0, None, None, None, None, None, None, None]
2 8 [0, 1, None, None, None, None, None, None]
0 8 [0, 1, None, None, None, None, None, None]
3 8 [0, 1, 2, None, None, None, None, None]
0 8 [0, 1, 2, None, None, None, None, None]
>>> 
Class variable cvsp_wait_time is modified using object obj0 and the modifications are visible in object obj1. However, the modification of class variable cvsp_lst_task is only visible in object obj0, not in object obj1.

My expectation is that variable cvsp_lst_task is shared between both objects, and thus will show the same value if accessed via both objects. Clearly, it does not. What is wrong in the code fragments above?

Explanatory note: This test class is an excerpt of a class in which list cvsp_wait_time will be modified quite often. Therefore, I decided not to use a list with a dynamic length in the expectation that a static list will minimise heap fragmentation. Thus methods like list.insert() are not used. Consequently, the current length of the list is maintained in cvsp_lst_task. As this variable seems to have a per-instance value rather than a per-class value, it is not possible to maintain list cvsp_wait_time properly.

User avatar
pythoncoder
Posts: 4646
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

That is an interesting one

Post by pythoncoder » Thu Jul 16, 2020 3:48 pm

This has me foxed too, but CPython behaves exactly the same way so it isn't a bug. If you explicitly name the class rather than using self, the problem goes away. Perhaps someone else can shed some light on this - I did think I had some grasp of this language, but I'm starting to wonder...
The code can be reduced to

Code: Select all

class Foo():
  x = 0

  def showcv(self):
    print(self.x)

  def setcv(self):
    self.x += 1

obj0 = Foo()
obj0.showcv()
obj1 = Foo()
obj1.showcv()

for j in range(3):
  obj0.setcv()
  obj0.showcv()
  obj1.showcv()
which produces this unexpected (to me) result under CPython 3.6.9 and MicroPython

Code: Select all

$ python3 -m rats54
0
0
1
0
2
0
3
0
The following works as expected under both:

Code: Select all

class Bar():
  x = 0

  def showcv(self):
    print(Bar.x)

  def setcv(self):
    Bar.x+= 1

obj0 = Bar()
obj0.showcv()
obj1 = Bar()
obj1.showcv()

for j in range(3):
  obj0.setcv()
  obj0.showcv()
  obj1.showcv()
producing

Code: Select all

0
0
1
1
2
2
3
3
The workround is simple, but I'd love to know why this happens. Are we missing something obvious?
Peter Hinch

User avatar
wimpie
Posts: 12
Joined: Sun Jul 05, 2020 11:14 am
Location: The Netherlands

Re: Strange behaviour class variables

Post by wimpie » Thu Jul 16, 2020 5:56 pm

Remarkable, this (what seems to be) inconsistent behaviour is thus reproducible in CPython.

The workaround mentioned is now used, so I can continue with development and tests. I probably will go for a shorter module and class name.

User avatar
Roberthh
Posts: 2228
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Strange behaviour class variables

Post by Roberthh » Thu Jul 16, 2020 6:17 pm

That seems like using local and global variables in a function. You can access global variables in a function, but writing to it creates a local variable unless you have declared it as global.

User avatar
wimpie
Posts: 12
Joined: Sun Jul 05, 2020 11:14 am
Location: The Netherlands

Re: Strange behaviour class variables

Post by wimpie » Thu Jul 16, 2020 8:29 pm

It might look like local versus global variables, but the tests don't support that idea. In the method in which this phenomena was discovered the first line is an assert command containing both cvsp_lst_task and cvsp_max_task in the condition part. Thus the first action on those variables is a read.

kevinkk525
Posts: 781
Joined: Sat Feb 03, 2018 7:02 pm

Re: Strange behaviour class variables

Post by kevinkk525 » Thu Jul 16, 2020 9:08 pm

Thanks to @pythoncoder for pointing me to this thread, I somehow just always skipped it without reading, which was wrong, because I think I can contribute to your discussion:

Some time ago I tried using class variables as a common information storage accessible to multiple class instances (in read/write) and ran into the same problem.
The usage of class and instance variables can be read here: https://www.digitalocean.com/community/ ... n-python-3
However, it doesn't say anything about class variables becoming instance variables.

My theory (without a source I can quote I'd feel uncomfortable making this a "fact"):
1. You try to read variable self.x but the instance variable self.x doesn't exist so the class variable "x" is being looked up
2. You write to self.x which creates the instance variable "x" because assigning to "self" will always create an instance variable.
3. Now reading self.x reads the instance variable and not the class variable but reading "self.x" in a different class instance will still return the class variable because in that instance no instance variable "x" exists.

Class variables can not be changed by using "self" because it is used to access instance variables. The behaviour seems very similar to global and local variables.

One oddity: In the first example "cvsp_wait_time" can be modified because it is a list and a list is passed through as a pointer and therefore every instance can directly modify it as no new object is being created while modifying it. Same goes for dictionaries and bytearrays, as long as you don't create a new dictionary or byterray by slicing.

Therefore, to modify class variables, you have to use Bar.x, which is not a workaroud but the intended way, similar to calling "global var" before assigning something to the global variable.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 4646
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Strange behaviour class variables

Post by pythoncoder » Fri Jul 17, 2020 5:41 am

This does seem to be the case. If, in setcv, we wrote

Code: Select all

self.the_answer = 42
we would all know what would happen: an instance variable would be created and assigned the value. What is counter-intuitive is that

Code: Select all

self.x += 1
will read a class variable and write an instance variable. It's not often I say this about Python, but I think this sucks. It can be demoed with this script:

Code: Select all

class Foo():
  x = 0

  def showcv( self ):
    print(self.x, Foo.x)

  def setcv(self):
    self.x += 1

obj0 = Foo()
obj0.showcv()
obj1 = Foo()
obj1.showcv()

for j in range(3):
  obj0.setcv()
  obj0.showcv()
  obj1.showcv()
which produces

Code: Select all

0 0
0 0
1 0
0 0
2 0
0 0
3 0
0 0
The class variable is being read but never written :o
Peter Hinch

User avatar
pythoncoder
Posts: 4646
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Strange behaviour class variables [SOLVED]

Post by pythoncoder » Sat Jul 18, 2020 9:42 am

I've used Python for 20 years so have encountered many of its quirks, but this was new to me.

After further thought I reckon this behaviour of the += operator is profoundly confusing.
Peter Hinch

User avatar
dhylands
Posts: 3482
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Strange behaviour class variables [SOLVED]

Post by dhylands » Sat Jul 18, 2020 10:14 pm

If you want to increment the class variable then you can use a class reference.

i.e. use Foo.x += 1 rather than self.x += 1

User avatar
pythoncoder
Posts: 4646
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Strange behaviour class variables [SOLVED]

Post by pythoncoder » Sun Jul 19, 2020 8:30 am

Indeed. I'm coming to the view that the intent is clearest if you always reference class variables this way rather than via self.
Peter Hinch

Post Reply