RuntimeError from PEP479 violation - need help
Posted: Sun Feb 17, 2019 12:14 pm
Hi all,
The medea low-memory JSON parser I developed in Micropython for ESP8266 has stopped working since v1.10.
I'm struggling to know how to modify the library to satisfy the requirements of PEP479 which was brought in by 1.10. This means all cases where I use nested generators now terminate in a "RuntimeError: generator raised StopIteration" even when I'm apparently catching the StopIteration
Can anyone suggest how I can trace where the PEP479 non-compliant code is in my generator stack, and where extra 'except StopIteration' clauses might be needed as I can't solve it, and I speculate there is not enough information in the RuntimeError to do so.
REFERENCE CASE
I'm currently getting REPL console logs like the one below. It shows a successful read of an account's tweets (a single tweet was requested from Twitter to keep the packet size low), followed by a RuntimeError when the generators run out of bytes to process from the HTTP REST response provided by the Twitter API...
See https://github.com/ShrimpingIt/medea/bl ... tFields.py for this example. It's a main.py I would expect to work without raising RuntimeErrors since the raised StopIteration 'errors' should be caught. Both line 32 and line 36 of twitterTimelineExtractFields.py are guarded by an 'except StopIteration'.
The diff at https://github.com/ShrimpingIt/medea/co ... 5982c0?w=1 shows the extra 'except' clauses I added, expecting to solve the problem.
I think the actual line where a StopIteration is explicitly or implicitly raised can't be traced currently, meaning it's impossible to fix.
WORKAROUNDS
I can work around the issue by building with v1.9.4 which doesn't have PEP479, but this is not sustainable.
An invocation like the following in the REPL successfully swallows all RuntimeErrors, but heapRe doesn't seem to have knowledge of the StopIteration or the line number where it was raised if it was further down in the stack. Anyway I don't feel good about catching a RuntimeError as normal behaviour every time I use the library.
BACKGROUND
Before PEP479 StopIterations would propagate through a series of generators in the stack without being explicitly caught. Each raised StopIteration caused the calling Generators to also stop implicitly. They bubbled up from a tree of other generators like...
PEP479 was focused on knowing where the actual terminating generator was in the stack (and making this explicit) but unfortunately the way Micropython reports the RuntimeError through the stack makes it impossible for me to trace where the uncaught StopIteration originates, while also being forced to fix it!
Can anyone help?
The medea low-memory JSON parser I developed in Micropython for ESP8266 has stopped working since v1.10.
I'm struggling to know how to modify the library to satisfy the requirements of PEP479 which was brought in by 1.10. This means all cases where I use nested generators now terminate in a "RuntimeError: generator raised StopIteration" even when I'm apparently catching the StopIteration
Can anyone suggest how I can trace where the PEP479 non-compliant code is in my generator stack, and where extra 'except StopIteration' clauses might be needed as I can't solve it, and I speculate there is not enough information in the RuntimeError to do so.
REFERENCE CASE
I'm currently getting REPL console logs like the one below. It shows a successful read of an account's tweets (a single tweet was requested from Twitter to keep the packet size low), followed by a RuntimeError when the generators run out of bytes to process from the HTTP REST response provided by the Twitter API...
Code: Select all
1096982979107278850 : '....The U.S. does not want to watch as these ISIS fighters permeate Europe, which is where they are expected to go. We do so much, and spend so much - Time for others to step up and do the job that they are so capable of doing. We are pulling back after 100% Caliphate victory!'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "examples/scripts/twitterTimelinePollFields.py", line 22, in loop
File "examples/scripts/twitterTimelinePollFields.py", line 17, in loop
File "examples/scripts/twitterTimelineExtractFields.py", line 36, in generateTweets
File "examples/scripts/twitterTimelineExtractFields.py", line 32, in generateTweets
RuntimeError: generator raised StopIteration
The diff at https://github.com/ShrimpingIt/medea/co ... 5982c0?w=1 shows the extra 'except' clauses I added, expecting to solve the problem.
I think the actual line where a StopIteration is explicitly or implicitly raised can't be traced currently, meaning it's impossible to fix.
WORKAROUNDS
I can work around the issue by building with v1.9.4 which doesn't have PEP479, but this is not sustainable.
An invocation like the following in the REPL successfully swallows all RuntimeErrors, but heapRe doesn't seem to have knowledge of the StopIteration or the line number where it was raised if it was further down in the stack. Anyway I don't feel good about catching a RuntimeError as normal behaviour every time I use the library.
Code: Select all
heapRe = None
from examples.scripts.twitterTimelinePollFields import loop
try:
loop()
except RuntimeError as re:
heapRe = re
Before PEP479 StopIterations would propagate through a series of generators in the stack without being explicitly caught. Each raised StopIteration caused the calling Generators to also stop implicitly. They bubbled up from a tree of other generators like...
Code: Select all
* Tokenizer#tokenizeValue (generator for SAX-style JSON tokens)
* calls next() on a byteGenerator which yields bytes
* yields tokens for matching literals like true, false, null
* calls 'yield from' to hand over byte processing to e.g...
* Tokenizer#tokenizeObject()
* Tokenizer#tokenizeKey()
* Tokenizer#tokenizeValue()
* Tokenizer#tokenizeArray()
* Tokenizer#tokenizeValue()
* Tokenizer#tokenizeString()
* Tokenizer#tokenizeNumber()
Can anyone help?