The medea low-memory JSON parser I developed in Micropython for ESP8266 has stopped working since v1.10.
I'm struggling to know how to modify the library to satisfy the requirements of PEP479 which was brought in by 1.10. This means all cases where I use nested generators now terminate in a "RuntimeError: generator raised StopIteration" even when I'm apparently catching the StopIteration
Can anyone suggest how I can trace where the PEP479 non-compliant code is in my generator stack, and where extra 'except StopIteration' clauses might be needed as I can't solve it, and I speculate there is not enough information in the RuntimeError to do so.
REFERENCE CASE
I'm currently getting REPL console logs like the one below. It shows a successful read of an account's tweets (a single tweet was requested from Twitter to keep the packet size low), followed by a RuntimeError when the generators run out of bytes to process from the HTTP REST response provided by the Twitter API...
Code: Select all
1096982979107278850 : '....The U.S. does not want to watch as these ISIS fighters permeate Europe, which is where they are expected to go. We do so much, and spend so much - Time for others to step up and do the job that they are so capable of doing. We are pulling back after 100% Caliphate victory!'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "examples/scripts/twitterTimelinePollFields.py", line 22, in loop
File "examples/scripts/twitterTimelinePollFields.py", line 17, in loop
File "examples/scripts/twitterTimelineExtractFields.py", line 36, in generateTweets
File "examples/scripts/twitterTimelineExtractFields.py", line 32, in generateTweets
RuntimeError: generator raised StopIteration
The diff at https://github.com/ShrimpingIt/medea/co ... 5982c0?w=1 shows the extra 'except' clauses I added, expecting to solve the problem.
I think the actual line where a StopIteration is explicitly or implicitly raised can't be traced currently, meaning it's impossible to fix.
WORKAROUNDS
I can work around the issue by building with v1.9.4 which doesn't have PEP479, but this is not sustainable.
An invocation like the following in the REPL successfully swallows all RuntimeErrors, but heapRe doesn't seem to have knowledge of the StopIteration or the line number where it was raised if it was further down in the stack. Anyway I don't feel good about catching a RuntimeError as normal behaviour every time I use the library.
Code: Select all
heapRe = None
from examples.scripts.twitterTimelinePollFields import loop
try:
loop()
except RuntimeError as re:
heapRe = re
Before PEP479 StopIterations would propagate through a series of generators in the stack without being explicitly caught. Each raised StopIteration caused the calling Generators to also stop implicitly. They bubbled up from a tree of other generators like...
Code: Select all
* Tokenizer#tokenizeValue (generator for SAX-style JSON tokens)
* calls next() on a byteGenerator which yields bytes
* yields tokens for matching literals like true, false, null
* calls 'yield from' to hand over byte processing to e.g...
* Tokenizer#tokenizeObject()
* Tokenizer#tokenizeKey()
* Tokenizer#tokenizeValue()
* Tokenizer#tokenizeArray()
* Tokenizer#tokenizeValue()
* Tokenizer#tokenizeString()
* Tokenizer#tokenizeNumber()
Can anyone help?