Understanding the qstr generation process

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
User avatar
hotel
Posts: 4
Joined: Sun Jan 23, 2022 2:21 am

Understanding the qstr generation process

Post by hotel » Sun Jan 23, 2022 2:43 am

i'm trying to work out how exactly the build process works when generating the qstr headers, and i've run into an issue:

py/py.mk says

Code: Select all

# Sources that may contain qstrings
SRC_QSTR_IGNORE = py/nlr%
SRC_QSTR += $(SRC_MOD) $(filter-out $(SRC_QSTR_IGNORE),$(PY_CORE_O_BASENAME:.o=.c)) $(PY_EXTMOD_O_BASENAME:.o=.c)
and py/mkrules.mk says

Code: Select all

(HEADER_BUILD)/qstr.i.last: $(SRC_QSTR) $(QSTR_GLOBAL_DEPENDENCIES) $(HEADER_BUILD)/moduledefs.h | $(QSTR_GLOBAL_REQUIREMENTS)
    ...
so it's pretty straightforward, that qstr.i.last depends on the core objects (with some exceptions, like the nlr\* stuff). but where i'm confused is that

Code: Select all

# py/py.mk
$(HEADER_BUILD)/qstrdefs.generated.h: $(PY_QSTR_DEFS) $(QSTR_DEFS) $(QSTR_DEFS_COLLECTED) $(PY_SRC)/makeqstrdata.py mpconfigport.h $(MPCONFIGPORT_MK) $(PY_SRC)/mpconfig.h | $(HEADER_BUILD)
    ...

# py/mkrules.mk
$(QSTR_DEFS_COLLECTED): $(HEADER_BUILD)/qstr.split
    ...
    
$(HEADER_BUILD)/qstr.split: $(HEADER_BUILD)/qstr.i.last
    ...
so it looks like qstrdefs.generated.h depends on QSTR_DEFS_COLLECTED which depends on qstr.split which depends on qstr.i.last which as we saw above depends on the core objects....... which is where the problem lies, because immediately in mpstate.c, one of the core sources, we can follow the includes to a dependency on qstrdefs.generated.h, which is what i'm trying to build.

so the question is: how does building the qstrdefs work with a circular dependency like this?

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Understanding the qstr generation process

Post by stijn » Sun Jan 23, 2022 12:00 pm

so the question is: how does building the qstrdefs work with a circular dependency like this?
It's not circular during preprocessing which generates said header, because it's inclusion is guarded by

Code: Select all

#ifdef NO_QSTR
see mkrules.mk (or equivalent genhdr.targets for msvc):

Code: Select all

# QSTR generation uses the same CFLAGS, with these modifications.
QSTR_GEN_FLAGS = -DNO_QSTR

User avatar
hotel
Posts: 4
Joined: Sun Jan 23, 2022 2:21 am

Re: Understanding the qstr generation process

Post by hotel » Sun Jan 23, 2022 7:13 pm

ahh of course, that makes sense. thanks

User avatar
hotel
Posts: 4
Joined: Sun Jan 23, 2022 2:21 am

Re: Understanding the qstr generation process

Post by hotel » Sun Jan 23, 2022 7:34 pm

stijn wrote:
Sun Jan 23, 2022 12:00 pm
so the question is: how does building the qstrdefs work with a circular dependency like this?
It's not circular during preprocessing which generates said header, because it's inclusion is guarded by

Code: Select all

#ifdef NO_QSTR
see mkrules.mk (or equivalent genhdr.targets for msvc):

Code: Select all

# QSTR generation uses the same CFLAGS, with these modifications.
QSTR_GEN_FLAGS = -DNO_QSTR
ok i take it back i'm still confused. as far as i can tell, despite the comment, the only reference to QSTR_GEN_C\[XX\]FLAGS is as an argument to makeqstrdefs.py which doesn't explain how it gets passed down to the compilation rules make tries to invoke for the prerequisites of the rule (qstr.i.last) that calls the python script

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Understanding the qstr generation process

Post by stijn » Mon Jan 24, 2022 8:05 am

makeqstrdefs.py is called with arguments including the compiler and flags (i.e. QSTR_GEN_CFLAGS), which makeqstrdefs.py uses to do the preprocessing by running the compiler with those flags (you can see that when running make V=1 btw)

User avatar
hotel
Posts: 4
Joined: Sun Jan 23, 2022 2:21 am

Re: Understanding the qstr generation process

Post by hotel » Mon Jan 24, 2022 7:45 pm

stijn wrote:
Mon Jan 24, 2022 8:05 am
makeqstrdefs.py is called with arguments including the compiler and flags (i.e. QSTR_GEN_CFLAGS), which makeqstrdefs.py uses to do the preprocessing by running the compiler with those flags
but my point is that the way Make works, whatever implicit rules exist for generating object files will run before the body of the rule runs, so i don't understand how makeqstrdefs.py could possibly run the preprocessing commands with the right flags set first.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Understanding the qstr generation process

Post by stijn » Tue Jan 25, 2022 7:41 am

I don't understand what you mean (I also don't know a lot about Make). The rules are such that OBJ depends on qstrdefs.generated.h, which depends on QSTR_DEFS_COLLECTED (among other things) which depends on qstr.split which depends on qstr.i.last. So the body of the qstr.i.last rule gets ran first, right? And that calls makeqstrdata.py to run the compiler with QSTR_GEN_FLAGS. And those flags aren't in rules, but global/statis whatever that is called in Make lingo. Btw there are a bunch of comments in mkrules.mk regarding qstr generation, order-only dependencies etc, did yuo check all of those?

Post Reply