QSTR database isn't updated when building

[elliotwoods]

When I add a QSTR to my own module and using the command

Code: Select all

make USER_C_MODULES=../../modules CFLAGS_EXTRA=-DMODULE_QUADRATURE_ENABLED=1 all -j8

I get

Code: Select all

../../modules/quadrature/quadrature.c:133:10: error: 'MP_QSTR_encoder' undeclared here (not in a function); did you mean 'MP_QSTR_encode'?
  .name = MP_QSTR_encoder,

The problem goes away if beforehand I do a

Code: Select all

make clean

Is there a way to change the QSTR's without cleaning the whole project?
Also - it would be great to add a note about this into the docs:
https://docs.micropython.org/en/latest/ ... dules.html
Especially it was confusing to get the error at all (It took me a while to figure out that it might be because QSTR's are not being updated)

Thank you
Elliot

[jimmo]

Which port are you building this with? I tested it on the Unix and STM32 ports and it seems to detect the QSTR change, so maybe there's something port-specific going on.

Is there a way to change the QSTR's without cleaning the whole project?

It would be good to get to the bottom of why this isn't being detected. Can you post the output of the start of the build?

For example if I add a new QSTR, I see:

Code: Select all

$ blaze BOARD=PYBD_SF6 USER_C_MODULES=../../../../v923z/micropython-ulab
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
Including User C Module from ../../../../v923z/micropython-ulab/code
GEN build-PYBD_SF6/genhdr/moduledefs.h
GEN build-PYBD_SF6/genhdr/qstr.i.last
GEN build-PYBD_SF6/genhdr/compressed.split
GEN build-PYBD_SF6/genhdr/qstr.split
GEN build-PYBD_SF6/genhdr/compressed.collected
GEN build-PYBD_SF6/genhdr/qstrdefs.collected.h
QSTR updated   <-------------------------------------------------------------------------
GEN build-PYBD_SF6/genhdr/qstrdefs.generated.h
Compressed data not updated

Unfortunately though a QSTR change does require rebuilding everything (as it changes the generated header file which is included by ~everything).

Also - it would be great to add a note about this into the docs:
https://docs.micropython.org/en/latest/ ... dules.html

Once we figure out what's going on I agree that would be worth doing.

[elliotwoods]

I'm on the ESP32 port
Since performing the make clean, I haven't been able to recreate the issue.

My sequence before AFAIK was:

1. Clone latest micropython
2. Clone relevant esp-idf v4
3. make
4. make again but this time with the user module folder <--- issue came here
5. make clean
6. make with module <--- worked

Sorry I can't find it again now.

[elliotwoods]

It happened once more today.
The sequence was

- make clean
- make
- make USER_C_MODULES=../../modules

This mirrors my experience before
It fails on the 3rd step. So I presume it fails to detect a change in the QSTR here
If it's sporadic, then maybe we should just add a line to the documentation for user modules like "Occasionally the build system fails to detect that a new QSTR exists. In this case you might need to try to perform a `make clean` and try building again"

[jimmo]

It happened once more today.
The sequence was

...

Ahh, yeah, this definitely explains it.

The problem here is that Make only cares about whether the things in the dependency list are newer than the output. Adding new things to the dependency list doesn't change anything if the new inputs are themselves older than the output.

So for example:
- Create foo.txt
- Create bar.txt
- Create Makefile:

Code: Select all

output.txt: foo.txt
	@echo "inputs" $^
	cat $^ > output.txt

then run

Code: Select all

$ make output.txt
inputs foo.txt
cat foo.txt > output.txt

then modify Makefile to add "bar.txt" as a dependency

Code: Select all

output.txt: foo.txt bar.txt
	@echo "inputs" $^
	cat $^ > output.txt

then run

Code: Select all

$ make output.txt
make: 'output.txt' is up to date.

I can't see a neat way around this... we could potentially cache the set of files used, but this gets tricky with the incremental generation. Obviously forcing the QSTR generation each time (although it would only force a full rebuild if any QSTRs changed) would still be a performance hit for every build.

I think (like you mention) this might be better addressed by the documentation?

[pythoncoder]

...Obviously forcing the QSTR generation each time (although it would only force a full rebuild if any QSTRs changed) would still be a performance hit for every build...

Too right

QSTR generation is slow and, unlike the rest of the build process, doesn't seem to be helped by changes to the PC hardware - more cores, faster network connection to server, nothing seems to speed it up.

Perhaps because building code is familiar I've always regarded the need for an occasional make clean as unsurprising.

My vote is to document it.