qstrdefs generator

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
User avatar
stick
Posts: 13
Joined: Fri Jan 08, 2016 9:17 pm

qstrdefs generator

Post by stick » Mon Mar 07, 2016 12:35 pm

split-off conversation from extmod generator thread:
dhylands wrote:The issue I see with generating qstrdefs automtically is figuring out what #ifdefs they belong in. We don't want to include unnecessary qstrs in the image.
I see two approaches for achieving this:

a) collect all qstrdefs and remember set of #if conditionals that need to be met in order to emit this definition. Obviously this needs a lot of postprocessing in order to generate a proper qstrdefs.h file.

b) just process the source files, throw away everything else except #if .. #endif blocks and convert MP_QSTR_foo usage to definition of Q(foo). Throw away unnecessary blocks (not containing any Q() lines). We also leverage the fact that Q() can be called multiple times (if this produces unwanted results, we should fix them, but I haven't found any).

I created a proof-of-concepts for both approaches here:

a) https://gist.github.com/prusnak/44aec7557e0ffc9d505b

b) https://gist.github.com/prusnak/8390fe090f0f1a43f357

Output of script a) for py/objstr.c:

Code: Select all

('Q(keepends)', ['(MICROPY_PY_BUILTINS_STR_SPLITLINES)'])
('Q(_backslash_n)', ['(MICROPY_PY_BUILTINS_STR_SPLITLINES)'])
('Q(utf_hyphen_8)', ['(MICROPY_CPYTHON_COMPAT)'])
('Q(utf_hyphen_8)', ['(MICROPY_CPYTHON_COMPAT)'])
('Q(decode)', ['(MICROPY_CPYTHON_COMPAT)'])
('Q(encode)', ['(MICROPY_CPYTHON_COMPAT)', '(!MICROPY_PY_BUILTINS_STR_UNICODE)'])
('Q(find)', [])
('Q(rfind)', [])
('Q(index)', [])
('Q(rindex)', [])
('Q(join)', [])
('Q(split)', [])
('Q(splitlines)', ['(MICROPY_PY_BUILTINS_STR_SPLITLINES)'])
('Q(rsplit)', [])
('Q(startswith)', [])
('Q(endswith)', [])
('Q(strip)', [])
('Q(lstrip)', [])
('Q(rstrip)', [])
('Q(format)', [])
('Q(replace)', [])
('Q(count)', [])
('Q(partition)', [])
('Q(rpartition)', [])
('Q(lower)', [])
('Q(upper)', [])
('Q(isspace)', [])
('Q(isalpha)', [])
('Q(isdigit)', [])
('Q(isupper)', [])
('Q(islower)', [])
('Q(str)', ['(!MICROPY_PY_BUILTINS_STR_UNICODE)'])
('Q(bytes)', [])
('Q(NULL)', [])
('Q(NULL)', [])
Output of script b) for py/objstr.c:

Code: Select all

#if MICROPY_PY_BUILTINS_STR_SPLITLINES
Q(keepends)
Q(_backslash_n)
#endif
#if MICROPY_CPYTHON_COMPAT
Q(utf_hyphen_8)
Q(utf_hyphen_8)
#endif
#if MICROPY_CPYTHON_COMPAT
Q(decode)
#if !MICROPY_PY_BUILTINS_STR_UNICODE
Q(encode)
#endif
Q(find)
Q(rfind)
Q(index)
Q(rindex)
Q(join)
Q(split)
#if MICROPY_PY_BUILTINS_STR_SPLITLINES
Q(splitlines)
#endif
Q(rsplit)
Q(startswith)
Q(endswith)
Q(strip)
Q(lstrip)
Q(rstrip)
Q(format)
Q(replace)
Q(count)
Q(partition)
Q(rpartition)
Q(lower)
Q(upper)
Q(isspace)
Q(isalpha)
Q(isdigit)
Q(isupper)
Q(islower)
#if !MICROPY_PY_BUILTINS_STR_UNICODE
Q(str)
#endif
Q(bytes)
Q(NULL)
Q(NULL)
What do you think? Which approach do you think is better? (I like the latter as it is simpler and we should not care how complex is the generated file, because it is created automagically during the build and the developer does not need to edit it at all).

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: qstrdefs generator

Post by pfalcon » Tue Mar 08, 2016 7:26 am

Adhoc preprocessor to generate qstr defs from raw source code is the idea which I had when only started participating in uPy development (see early tickets in github tracker, some of them may still be open). I never went to argue it far enough/implement it, as there were always better things to do. (Not implying it shouldn't be done eventually, maybe even soon.)
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
dhylands
Posts: 3417
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: qstrdefs generator

Post by dhylands » Tue Mar 08, 2016 8:40 am

stick wrote:What do you think? Which approach do you think is better? (I like the latter as it is simpler and we should not care how complex is the generated file, because it is created automagically during the build and the developer does not need to edit it at all).
I kind of like the second approach as well (it was one of the ways I was envisioning that this could be implemented).

User avatar
braiins
Posts: 19
Joined: Mon Feb 23, 2015 1:09 pm
Contact:

Re: qstrdefs generator

Post by braiins » Tue Mar 08, 2016 10:33 am

The b) approach is straightfully usable including the ifdef hell. I like it as it could be possible to integrate it into the build process.
I have looked into the build system a bit and it seems that it is not as easy since each .o compilation step has dependency generation as a side effect. A trivial implementation could obviously perform full scan of the sources but that is not too nice and would slow down the build process.
braiins - Your partner in embedded world - http://www.braiins.cz

User avatar
stick
Posts: 13
Joined: Fri Jan 08, 2016 9:17 pm

Re: qstrdefs generator

Post by stick » Tue Mar 08, 2016 1:12 pm

So it seems like we all like approach b). Good.

I think we need to perform the full scan anyway, because the way it is handled now is that qdefs from multiple sources are being output to one file and this would break on parallel build.

I updated the script, so it performs a subdir scan for *.c if argv[1] is a directory: https://gist.github.com/prusnak/8390fe090f0f1a43f357

To integrate this in the build process I would need a help from someone who has deeper insight into it than I do. But I am willing to make further changes to the generate script if needed.

User avatar
braiins
Posts: 19
Joined: Mon Feb 23, 2015 1:09 pm
Contact:

Re: qstrdefs generator

Post by braiins » Tue Mar 08, 2016 1:43 pm

I am drafting a prototype - patch into the build system
braiins - Your partner in embedded world - http://www.braiins.cz

User avatar
dhylands
Posts: 3417
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: qstrdefs generator

Post by dhylands » Tue Mar 08, 2016 4:52 pm

Feel free to ask questions about the build system. I was one of the major contributors to that.

User avatar
dhylands
Posts: 3417
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: qstrdefs generator

Post by dhylands » Tue Mar 08, 2016 4:55 pm

stick wrote:I think we need to perform the full scan anyway, because the way it is handled now is that qdefs from multiple sources are being output to one file and this would break on parallel build.
I think that it should be upto a port to specify which trees to scan. We probably shouldn't be scanning the cc3200 tree while building stmhal for example. This would be inefficient and I'm pretty sure you would get alot more qstrs than desired.

User avatar
stick
Posts: 13
Joined: Fri Jan 08, 2016 9:17 pm

Re: qstrdefs generator

Post by stick » Tue Mar 08, 2016 5:08 pm

dhylands wrote:
stick wrote:I think we need to perform the full scan anyway, because the way it is handled now is that qdefs from multiple sources are being output to one file and this would break on parallel build.
I think that it should be upto a port to specify which trees to scan. We probably shouldn't be scanning the cc3200 tree while building stmhal for example. This would be inefficient and I'm pretty sure you would get alot more qstrs than desired.
Of course. That's why the tool only scans subdir of the dir you gave it to. We are trying to figure out the details with braiins over the IM and I think it will go. :-)

User avatar
braiins
Posts: 19
Joined: Mon Feb 23, 2015 1:09 pm
Contact:

Re: qstrdefs generator

Post by braiins » Tue Mar 08, 2016 8:49 pm

I have fixed the script to handle nested ifdefs properly: https://gist.github.com/honzik666/23def819ee6285eebc97 Let me know if anything is still not in line...

The build system changes are minimal, I will make some tests tomorrow and post it to github tomorrow. I am curious if Dave likes it ;-)

Jan
braiins - Your partner in embedded world - http://www.braiins.cz

Post Reply