GPS parsing using UART

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
Antoine
Posts: 6
Joined: Sat Aug 19, 2017 6:11 am

GPS parsing using UART

Post by Antoine » Sat Aug 19, 2017 6:24 pm

I am trying to read the NMEA serial stream of a NEO-7M GPS module on an ESP32 dev board. I monitor the output lines and check their validity (NMEA lines come with a checksum). Most of the time, reading the flow of serial stream fails: only a few lines come with valid checksum, because there are missing letters. Sometimes, maybe once in 50, I get a perfect (infinite) serial stream of well formed NMEA lines. By this, I mean that once in ~50 times I start my logging program, all goes perfectly until I ctrl+C the infinite loop. In the over cases, the failure has 1s period: for instance, the 3rd and 5th line inside a 1s packet will systematically be bad.

Since I sometimes get a perfect stream (no dropped characters), I think there is some timing issue: the perfect result occur when I sync with the 1s period of data stream. I was wondering if anybody here has any idea what is going on. Is there a solution?

- cheers

some details:
[code]
>>> os.uname()
(sysname='esp32', nodename='esp32', release='1.9.1', version='v1.9.1-436-gd3ad3fdb on 2017-08-19', machine='ESP32 module with ESP32')
[/code]

I have set the serial according to the documentation of the NEO-7M GPS module:
[code]
u = machine.UART(2, 9600)
u.init(bits=8, stop=1, parity=None)
[/code]

I use a simple code inside an infinite loop:
[code]
buf=''
while True:
if u.any():
buf += u.read().decode()
if '\r\n' in buf:
line, buf = buf.split('\r\n',1)
print(line)
[/code]

I have tried many other ways to read the serial stream: “select.poll()", "u.readline()", “u.readinto()”, none of them make a difference. Additionally, I can change the device's baud rate to any value in [4800, 9600, 19200, 38400, 57600, 115200]: It does not improved the situation of the dropped data...

User avatar
pythoncoder
Posts: 4577
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: GPS parsing using UART

Post by pythoncoder » Mon Aug 21, 2017 5:52 am

Are the missing characters at the start of the line or in the middle? If they are at the start the problem is one of achieving line synchronisation. You need to read and discard characters until you've read the newline character(s), then issue readline(). That way you should receive an entire line. So I would structure the code as follows:

Code: Select all

started = False
while True:
	if not started:
		started = True
		u.readline()  # discard 1st (possibly partial) line
	buf = u.readline() # wait for a complete line
	print(buf)
If the missing characters are in the middle of a line, especially at baudrates as low as 9600, then something more serious is going on. If you have access to a Pyboard it would be interesting to try your code on that. The ESP32 is new and under development so it is possible that there is an issue with the UART implementation. Although from your description of the symptoms I doubt this.
Peter Hinch

Antoine
Posts: 6
Joined: Sat Aug 19, 2017 6:11 am

Re: GPS parsing using UART

Post by Antoine » Mon Aug 21, 2017 3:32 pm

Thanks Peter for the suggestions, but unfortunately it does not help... The result is basically the same: some lines are well formed, but most of them are missing characters, not only at the end of the line, and sometimes most of the line is missing.

I do not have a pyboard, unfortunately.

In the meantime, I got an idea to test the ESP32 UART: I used an esp8266 dev board and talk to it using UART rather than the USB-to-Serial interface of the dev board. The result is very interesting! I can see from the "screen" console running in the background that the esp8266 receives the simple commands I sent in the REPL, but the answer is scrambled:

ESP32:
[code]
>>> import machine, utime
>>> u = machine.UART(2,115200)
>>> u.init(parity=None, stop=1, bits=8)
>>> u.read(); u.write('print(1+2)\r\n'); utime.sleep(0.2); u.read()
b''
12
b'1+2)\r\n> prin (1+2)\r'
[/code]

on the ESP8266 "screen" console, running in my computer's terminal, I get what I expect:
[code]
>>> print(1+2)
3
[/code]

Obviously, the ESP32 is reading scrambled text, even if the writing command is working correctly. Note that what the ESP32 is reading will change every time I re-send the command...

User avatar
pythoncoder
Posts: 4577
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: GPS parsing using UART

Post by pythoncoder » Tue Aug 22, 2017 12:09 pm

It does sound as if there is an issue with the ESP32 UART read implementation. It is a new port, still labelled "experimental", so this is unsurprising. I'll try to find time to do some tests myself with a view to raising an issue - unless you want to raise one?
Peter Hinch

User avatar
pythoncoder
Posts: 4577
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Test results

Post by pythoncoder » Thu Aug 24, 2017 9:19 am

tl;dr UART 2 seems to work fine here.

The following script tests concurrent sending and receiving and it works on my Espressif ESP32-DevKit. To run it link UART2 Txd and Rxd.

Code: Select all

import uasyncio as asyncio
from machine import UART
uart = UART(2, 9600)

async def sender():
    swriter = asyncio.StreamWriter(uart, {})
    while True:
        await swriter.awrite('Hello uart. The quick brown fox jumps over the lazy dog.\n')
        print('Wrote')
        await asyncio.sleep(2)

async def receiver():
    sreader = asyncio.StreamReader(uart)
    while True:
        res = await sreader.readline()
        print('Recieved', res)

loop = asyncio.get_event_loop()
loop.create_task(sender())
loop.create_task(receiver())
loop.run_forever()
[EDIT]
I've also measured the baudrate and it is spot-on.
Peter Hinch

Antoine
Posts: 6
Joined: Sat Aug 19, 2017 6:11 am

Re: GPS parsing using UART

Post by Antoine » Sun Sep 24, 2017 10:23 am

Hi Peter,

Finally got back to this... to find out the current esp32 version of micropython (MicroPython v1.9.2-272-g0d183d7f) does not have asyncio.

I tried to figure out how to install it but I failed. The only thing I could find is viewtopic.php?f=18&t=3691&hilit=uasyncio+esp32#p21407 but the detailed procedure is not given: I used the micropython unix port to install uasyncio but copying the resulting files failed:

Code: Select all

>>> import uasyncio
>>> dir(uasyncio)
['__name__', '__path__']
My second try was to use upip on the board, but it did not work either, but I should have expected it based on the thread I just mentioned.

So the bottom line is: how do I install uasyncio on the ESP32? I understand that the ESP32 port is experimental and I might have missed the procedure in my search, but if it not too much to ask, could you please point me to a page describing how to install the uasyncio module on the ESP32?

cheers,

Antoine

User avatar
pythoncoder
Posts: 4577
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: GPS parsing using UART

Post by pythoncoder » Tue Sep 26, 2017 5:04 pm

Sorry for the slow response - am on a short break away from home. My approach is to use the Unix build. Install uasyncio using upip and check it can be imported. The files should be in /home/username/.micropython where the directories can be copied to the target.
Peter Hinch

Antoine
Posts: 6
Joined: Sat Aug 19, 2017 6:11 am

Re: GPS parsing using UART

Post by Antoine » Sun Jan 07, 2018 6:42 pm

After efforts spread over months, I could not make uasyncio work on the ESP32, so I have switched to a Raspberry pi Zero-W running full-fledge linux and moved forward on my project (and confirmed my GPS unit is not defective).

Today, I gave a try to Arduino to convince myself that my ESP32 board is not defective and found out that, until a few month ago, there was exactly the same problem I reported here with reading the GPS from the second UART of an ESP32! It is not clear to me how the issue was fixed, though: https://github.com/espressif/arduino-esp32/issues/665

I do not want to give up on micropython! after coding in Arduino, I feel the pain because I only managed to read and parse the NMEA lines, which would have taken a tenth of the code and a tenth of the debug time in micropython :cry:

Antoine

User avatar
pythoncoder
Posts: 4577
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: GPS parsing using UART

Post by pythoncoder » Mon Jan 08, 2018 7:01 am

Antoine wrote:
Sun Jan 07, 2018 6:42 pm
After efforts spread over months, I could not make uasyncio work on the ESP32...
uasyncio does work on the ESP32. My asynchronous MQTT library runs on the ESP32 and makes heavy use of uasyncio. It suffers from certain issues with the ESP32 vendor firmware, but over the course of its development I experienced no problems whatsoever with uasyncio.

I did find it necessart to issue sleep_ms(20) at certain points in the code in order to ensure that the vendor WiFi stack ran - you might like to browse my mqtt_as.py to see the places it was needed. These are faults in the vendor code, not in uasyncio. You might like to look at this to see some of the problems I encountered with the vendor code.
Peter Hinch

Antoine
Posts: 6
Joined: Sat Aug 19, 2017 6:11 am

Re: GPS parsing using UART

Post by Antoine » Sun Jan 14, 2018 8:34 am

After playing with Arduino, I flashed again the ESP32 board with the latest micropython, and re-install uasyncio: it works now. The example code provided by Peter works perfectly, the one connecting Rx and Tx together.

However, when I adapted it for reading the GPS stream, it fails just like the code I wrote before: some lines are good, some are bad (corrupted)...

I wish I could help with micropython development at this point, but this is beyond my skill sets... Thanks Peter for sticking with me and my issue, the help from this forum is invaluable!

Post Reply