micropyGPS: a GPS parser for Micropython

Showroom for MicroPython related hardware projects.
Target audience: Users wanting to show off their project!
inmcm
Posts: 7
Joined: Tue Dec 16, 2014 12:27 am

micropyGPS: a GPS parser for Micropython

Post by inmcm » Tue Dec 16, 2014 12:41 am

For the past couple weeks I've been working on a GPS NMEA sentence parser that will work well with Micropython and the pyboard. The code and some explanation are available at Github:

https://github.com/inmcm/micropyGPS

I'm most definitely still working on it, but I feel the code is far enough along now to share. I'm currently supporting parsing the most common NMEA sentences: RMC, GGA, GSA, VTG, and GSV. I'm planning to add most of the ones detailed here: http://aprs.gids.nl/nmea/#rmc at some point soon

I've tested the code with my own pyboard and Adafruit Ulitmate GPS breakout and it works well at 9600 baud at a 1Hz update rate. I still need to try faster baudrates and updates rates to see just how fast the pyboard can parse sentences.

I'm also looking at adding several more helper functions related to pretty printing of GPS info, tracking fix time, and tracking time/distance/course to a target location.

Questions and comments are welcome :D

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: micropyGPS: a GPS parser for Micropython

Post by Turbinenreiter » Tue Dec 16, 2014 3:19 pm

Awesome! This will help with the navigation skin and the quacopter.

blmorris
Posts: 348
Joined: Fri May 02, 2014 3:43 pm
Location: Massachusetts, USA

Re: micropyGPS: a GPS parser for Micropython

Post by blmorris » Tue Dec 16, 2014 3:48 pm

Turbinenreiter wrote:Awesome! This will help with the navigation skin and the quacopter.
Absolutely, nice work! I just had a chance to play around a little bit with it and run your example code; I have the same GPS as you, the Adafruit Ultimate GPS module. It is also the device I used for the IMU+GPS pyskin (BTW, boards and parts are on order, but probably won't arrive until after I leave on vacation. Something to look forward to for the new year!)

I started to write a simple GPS parser a month or so ago, but for me it didn't get much past an exercise in Python string formatting. Your code will be much more useful.

I see that the module requires an explicit command to receive and parse new GPS data from the UART. (Not a criticism, it is clear that the code is intended as a GPS data parser, and it is good to keep the code that manages the incoming data stream separate.) I wonder if it would be possible to use the asyncio stuff that has been developed to allow the GPS data to be updated continuously while keeping the REPL open. Alternately, maybe we could use the PPS signal from the GPS module to trigger a callback to listen for new GPS strings; this has the benefit of providing a high-precision timestamp as well.

Finally - I don't know how much of a project it would be, but it would be pretty cool if we could use the Extended Prediction Orbit (EPO) data (available here) to implement Assisted GPS for faster fix times.
-Bryan

inmcm
Posts: 7
Joined: Tue Dec 16, 2014 12:27 am

Re: micropyGPS: a GPS parser for Micropython

Post by inmcm » Wed Dec 17, 2014 12:33 am

Thank you for the kind words.

blmorris, I've also given some thought to how to 'feed' the object efficiently. I haven't looked into the asyncio yet, so I can't comment on that. In general though, given this is GPS, you're going to get data from the receiver regularly. As long as you schedule in some time to poll the UART, most projects will be fine. Now on something complicated or time sensitive like controlling a drone, using some form of interrupt is the ideal solution. Using a GPIO interrupt fed from the GPS sounds good initially, but I'm afraid you would have to setup the UART with a large buffer (300-500 bytes) in order to catch all the sentences cleanly. The better and not possible at the moment solution would be for Micropython to have 'New Data Available' callbacks on the async communication ports: UART, CAN and USB_VCP. In that case, micropyGPS could have a wrapper function you can use in a UART callback to automatically feed new characters as they are received. I don't know if that's something the devs would be open to having, but it may be worth looking into for more than just this. Right now though, I'm more focused on the actual parser and adding features.

That Extended Prediction Orbit data format looks like a whole other project. I'll have to sit down and read about it more thoroughly.

Damien
Site Admin
Posts: 647
Joined: Mon Dec 09, 2013 5:02 pm

Re: micropyGPS: a GPS parser for Micropython

Post by Damien » Wed Dec 17, 2014 4:27 pm

Re callbacks and UART buffers. Note that you can set the UART receive buffer size when you init the UART object. You can use much ram as you like for the buffer, eg 1k bytes is easily accommodated.

I think we should add a callback mechanism to UART receive, but would you use it in this case for more than just buffering data at the Python level? If you had a callback, how exactly would you user it? Would it be better to have a callback for "receive buffer half full" rather than "one char received"?

If your data rate is very large and bursty (eg 115200 baud but only for a short time once per second) then it makes sense to just buffer the input and process it all at once.

blmorris
Posts: 348
Joined: Fri May 02, 2014 3:43 pm
Location: Massachusetts, USA

Re: micropyGPS: a GPS parser for Micropython

Post by blmorris » Wed Dec 17, 2014 8:28 pm

Damien wrote:Re callbacks and UART buffers. Note that you can set the UART receive buffer size when you init the UART object. You can use much ram as you like for the buffer, eg 1k bytes is easily accommodated.
Good to know, I had overlooked this.
I think we should add a callback mechanism to UART receive, but would you use it in this case for more than just buffering data at the Python level? If you had a callback, how exactly would you use it? Would it be better to have a callback for "receive buffer half full" rather than "one char received"?
Since I still do most of my tinkering from the REPL, I thought that if the gps parser object could receive updated information in the background, (i.e. as a callback) then I could poll the object from the REPL prompt to get current GPS location data. However, I don't see a simple way to do this, even with a UART callback, as the callback ISR can't allocate memory. (I know that allocating memory in the ISR is on the agenda, but there are better reasons to do that than catering to my lazy programming habits. I should really study the asyncio stuff to see if there is a solution there.)

In a real application we probably aren't worried about the REPL. We could already implement polling with a timer, although having the UART callback would allow for cleaner program logic.
inmcm wrote:That Extended Prediction Orbit data format looks like a whole other project. I'll have to sit down and read about it more thoroughly.
Agreed, it's not really within the scope of the GPS parser. I've just been reading a lot about GPS lately and thinking that this would be nice to have.

-Bryan

inmcm
Posts: 7
Joined: Tue Dec 16, 2014 12:27 am

Re: micropyGPS: a GPS parser for Micropython

Post by inmcm » Thu Dec 18, 2014 1:13 am

Damien wrote:I think we should add a callback mechanism to UART receive, but would you use it in this case for more than just buffering data at the Python level? If you had a callback, how exactly would you user it? Would it be better to have a callback for "receive buffer half full" rather than "one char received"? If your data rate is very large and bursty (eg 115200 baud but only for a short time once per second) then it makes sense to just buffer the input and process it all at once.
My original thought was to trigger the callback on every character, but your question made me wonder how fast can I can actually parse. I whipped up a little code to run on the pyboard: https://gist.github.com/inmcm/bf22a292a7609314937f This code returns the number of microseconds it takes to run each character of a sentence through the update() function.

The results were pretty cool. The time for various pieces of data were:
Most Normal Characters: 113-127us
',' which end segments: ~110us
'*' which signifies the end of GPS data: ~90us

When the test gets to the last valid character of the sentence, it also calls the sentence function to update the object. These times were a bit more noisy, so I rigged up another test to get the average time to parse the final character and update the GPS object with fix data: https://gist.github.com/inmcm/179e31c3890349c4aad9

The times for each sentence type were pretty close:
RMC: 583us
GGA: 534us
GSA: 555us
GSV: 596us

Thinking about baud rates; 9600 baud is 104us per bit, so receiving a single NMEA sentence character takes 1041us (assuming 8N1 encoding). That's plenty slow enough to do the call back on every new character. On the other hand, 115200 baud only takes 86us to send a character. Based on the times above, we'd have to buffer before processing.

So to answer your question, I would vote for a UART callback system based off of the "fullness" of the UART buffer. The UART would collect characters in its buffer till a certain percentage was reached at which point, the callback would be triggered. With that system, you can use the size of the buffer to control how often you service the callback. Want the callback serviced a lot? Set the buffer to 5 characters. You have a lot of fast incoming data, set the buffer to 1K. The fullness of FIFO wouldn't need to configurable; just low enough that some characters could be read before it overruns. With that sort of system you could have something like the code below

Code: Select all

my_uart = (UART3,9600,512)  # Create UART with 512 byte buffer
my_gps = MicropyGPS() 

# Create the callback with a special GPS update method
my_uart.callback(my_gps.auto_update(UART3))

# Do other stuff.......

### Within micropyGPS class
# This method gets called once the X% of 512 bytes is filled
def auto_update(self,some_uart):
    while some_uart.any():
          self.update(chr(some_uart.readchr()))

Damien
Site Admin
Posts: 647
Joined: Mon Dec 09, 2013 5:02 pm

Re: micropyGPS: a GPS parser for Micropython

Post by Damien » Thu Dec 18, 2014 12:29 pm

I would vote for a UART callback system based off of the "fullness" of the UART buffer. The UART would collect characters in its buffer till a certain percentage was reached at which point, the callback would be triggered. With that system, you can use the size of the buffer to control how often you service the callback. Want the callback serviced a lot? Set the buffer to 5 characters. You have a lot of fast incoming data, set the buffer to 1K. The fullness of FIFO wouldn't need to configurable; just low enough that some characters could be read before it overruns.
That's a neat idea: to specify how full the buffer is before the callback is called. Eg: uart.callback(func, percent_full=50). Having the percentage configurable might be useful. If not, I would choose a fixed value of 50%.

But there's still the issue that you can't allocate memory in the callback (since it's called on an interrupt). The best you can do (apart from writing your code to not allocate memory) is set a flag for your main code, and then the main code must poll that flag to see if there is anything to do. But then why not just poll the size of the uart receive buffer in the main loop instead of having a callback?

The ultimate solution is to be able to allocate memory in an interrupt, because then your main loop can be busy doing whatever it likes, and your callback will run only when necessary.

inmcm
Posts: 7
Joined: Tue Dec 16, 2014 12:27 am

Re: micropyGPS: a GPS parser for Micropython

Post by inmcm » Tue Feb 03, 2015 2:15 am

I've updated the library with some new features and bug fixes.

I also made one quick try to make a script that uses an external interrupt to trigger the updater, but it crashed with a MemoryError. I thought I might get away with it since virtually every variable in the object is pre-allocated. Sadly, though, it looks like I create two temporary lists as part of the parsing operation that would be very kludgy to fix. So I had to settle for the global flag model where the update happens in the main execution loop.

I eagerly await the day we can allocate memory in callbacks :D

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: micropyGPS: a GPS parser for Micropython

Post by dhylands » Tue Feb 03, 2015 2:24 am

All interrupt errors (even syntax errors) are reported as MemoryErrors unless you do something like this:

Code: Select all

import micropython
micropython.alloc_emergency_exception_buf(100)

Post Reply