qstrdefs generator

[stick]

split-off conversation from extmod generator thread:

The issue I see with generating qstrdefs automtically is figuring out what #ifdefs they belong in. We don't want to include unnecessary qstrs in the image.

I see two approaches for achieving this:

a) collect all qstrdefs and remember set of #if conditionals that need to be met in order to emit this definition. Obviously this needs a lot of postprocessing in order to generate a proper qstrdefs.h file.

b) just process the source files, throw away everything else except #if .. #endif blocks and convert MP_QSTR_foo usage to definition of Q(foo). Throw away unnecessary blocks (not containing any Q() lines). We also leverage the fact that Q() can be called multiple times (if this produces unwanted results, we should fix them, but I haven't found any).

I created a proof-of-concepts for both approaches here:

a) https://gist.github.com/prusnak/44aec7557e0ffc9d505b

b) https://gist.github.com/prusnak/8390fe090f0f1a43f357

Output of script a) for py/objstr.c:

Code: Select all

('Q(keepends)', ['(MICROPY_PY_BUILTINS_STR_SPLITLINES)'])
('Q(_backslash_n)', ['(MICROPY_PY_BUILTINS_STR_SPLITLINES)'])
('Q(utf_hyphen_8)', ['(MICROPY_CPYTHON_COMPAT)'])
('Q(utf_hyphen_8)', ['(MICROPY_CPYTHON_COMPAT)'])
('Q(decode)', ['(MICROPY_CPYTHON_COMPAT)'])
('Q(encode)', ['(MICROPY_CPYTHON_COMPAT)', '(!MICROPY_PY_BUILTINS_STR_UNICODE)'])
('Q(find)', [])
('Q(rfind)', [])
('Q(index)', [])
('Q(rindex)', [])
('Q(join)', [])
('Q(split)', [])
('Q(splitlines)', ['(MICROPY_PY_BUILTINS_STR_SPLITLINES)'])
('Q(rsplit)', [])
('Q(startswith)', [])
('Q(endswith)', [])
('Q(strip)', [])
('Q(lstrip)', [])
('Q(rstrip)', [])
('Q(format)', [])
('Q(replace)', [])
('Q(count)', [])
('Q(partition)', [])
('Q(rpartition)', [])
('Q(lower)', [])
('Q(upper)', [])
('Q(isspace)', [])
('Q(isalpha)', [])
('Q(isdigit)', [])
('Q(isupper)', [])
('Q(islower)', [])
('Q(str)', ['(!MICROPY_PY_BUILTINS_STR_UNICODE)'])
('Q(bytes)', [])
('Q(NULL)', [])
('Q(NULL)', [])

Output of script b) for py/objstr.c:

Code: Select all

#if MICROPY_PY_BUILTINS_STR_SPLITLINES
Q(keepends)
Q(_backslash_n)
#endif
#if MICROPY_CPYTHON_COMPAT
Q(utf_hyphen_8)
Q(utf_hyphen_8)
#endif
#if MICROPY_CPYTHON_COMPAT
Q(decode)
#if !MICROPY_PY_BUILTINS_STR_UNICODE
Q(encode)
#endif
Q(find)
Q(rfind)
Q(index)
Q(rindex)
Q(join)
Q(split)
#if MICROPY_PY_BUILTINS_STR_SPLITLINES
Q(splitlines)
#endif
Q(rsplit)
Q(startswith)
Q(endswith)
Q(strip)
Q(lstrip)
Q(rstrip)
Q(format)
Q(replace)
Q(count)
Q(partition)
Q(rpartition)
Q(lower)
Q(upper)
Q(isspace)
Q(isalpha)
Q(isdigit)
Q(isupper)
Q(islower)
#if !MICROPY_PY_BUILTINS_STR_UNICODE
Q(str)
#endif
Q(bytes)
Q(NULL)
Q(NULL)

What do you think? Which approach do you think is better? (I like the latter as it is simpler and we should not care how complex is the generated file, because it is created automagically during the build and the developer does not need to edit it at all).

[pfalcon]

Adhoc preprocessor to generate qstr defs from raw source code is the idea which I had when only started participating in uPy development (see early tickets in github tracker, some of them may still be open). I never went to argue it far enough/implement it, as there were always better things to do. (Not implying it shouldn't be done eventually, maybe even soon.)

[dhylands]

What do you think? Which approach do you think is better? (I like the latter as it is simpler and we should not care how complex is the generated file, because it is created automagically during the build and the developer does not need to edit it at all).

I kind of like the second approach as well (it was one of the ways I was envisioning that this could be implemented).

[braiins]

The b) approach is straightfully usable including the ifdef hell. I like it as it could be possible to integrate it into the build process.
I have looked into the build system a bit and it seems that it is not as easy since each .o compilation step has dependency generation as a side effect. A trivial implementation could obviously perform full scan of the sources but that is not too nice and would slow down the build process.

[stick]

So it seems like we all like approach b). Good.

I think we need to perform the full scan anyway, because the way it is handled now is that qdefs from multiple sources are being output to one file and this would break on parallel build.

I updated the script, so it performs a subdir scan for *.c if argv[1] is a directory: https://gist.github.com/prusnak/8390fe090f0f1a43f357

To integrate this in the build process I would need a help from someone who has deeper insight into it than I do. But I am willing to make further changes to the generate script if needed.

[braiins]

I am drafting a prototype - patch into the build system

[dhylands]

Feel free to ask questions about the build system. I was one of the major contributors to that.

[dhylands]

I think we need to perform the full scan anyway, because the way it is handled now is that qdefs from multiple sources are being output to one file and this would break on parallel build.

I think that it should be upto a port to specify which trees to scan. We probably shouldn't be scanning the cc3200 tree while building stmhal for example. This would be inefficient and I'm pretty sure you would get alot more qstrs than desired.

[stick]

I think we need to perform the full scan anyway, because the way it is handled now is that qdefs from multiple sources are being output to one file and this would break on parallel build.

I think that it should be upto a port to specify which trees to scan. We probably shouldn't be scanning the cc3200 tree while building stmhal for example. This would be inefficient and I'm pretty sure you would get alot more qstrs than desired.

Of course. That's why the tool only scans subdir of the dir you gave it to. We are trying to figure out the details with braiins over the IM and I think it will go.

[braiins]

I have fixed the script to handle nested ifdefs properly: https://gist.github.com/honzik666/23def819ee6285eebc97 Let me know if anything is still not in line...

The build system changes are minimal, I will make some tests tomorrow and post it to github tomorrow. I am curious if Dave likes it

Jan