ntpclient - uasyncio based NTP synchronization

[MostlyHarmless]

Inspired by another thread about ntptime and its limitations I have created a new module, ntpclient.

This is work in progress. At this point it works reasonably well on ESP32 and to some degree on ESP8266.

Both implementations very much benefit from a local NTP server, which is rather simple to set up. Just add one "restrict" line to the /etc/ntp.conf file of any Linux machine in your network and open port 123/udp. Well, that is if you have a local Linux machine that is normally on 24/7. Due to the resource limitations, using public NTP servers (like pool.ntp.org) will add inaccuracy. One just cannot afford all the fancy statistical analysis, a real ntpd can do.

At the time of this writing each implementation needs a commit to be cherry-picked into a custom firmware build. See the README for reference. Each commit is adding functionality to MicroPython that allows to adjust/slew the RTC, which can only be implemented in C.

The provided test scrips produce a 100ms square wave pulse on a specified pin. I use this in combination with the PPS from a GPS receiver to visualize the behavior on an oscilloscope.


Regards, Jan

[jedie]

adjust/slew the RTC would be really nice. So maybe a sync every hour can be expand to one sync per day.

But one question to this: why are the RTC so inacurate? Is the difference equally on every device, or is this a kind of randomness? Does it always slew constantly ?

[pythoncoder]

...why are the RTC so inacurate?...

It's hardware dependent. I believe the one on ESP8266 does not use a crystal oscillator, so it's wildly inaccurate. That on a Pyboard 1.x can be calibrated to within a couple of minutes a year.

[MostlyHarmless]

...why are the RTC so inacurate?...

It's hardware dependent. I believe the one on ESP8266 does not use a crystal oscillator, so it's wildly inaccurate. That on a Pyboard 1.x can be calibrated to within a couple of minutes a year.

This is my understanding too. On the ESP8266 the RTC is completely driven by an internal R/C circuit that jitters +/- 5%. Adjusting that once a day is not going to make you happy. Recalibrating it based on the main XTAL every two seconds barely keeps it from wandering off by more than 50ms between NTP server polls (every 17 minutes).

I don't have a Pyboard, so I can't comment on that.

On the ESP32 the clock uses the main XTAL while powered, but will switch to a similar 150kHz R/C based oscillator for deep sleep. Which is why I think that this isn't going to work well if you need to sleep.

And yes, the drift is different from device to device. I have one ESP32 that has almost no drift at room temperature. Another consistently averages in at -30us per second.


Regards, Jan

[tve]

@jedie:
The accuracy depends on the oscillator. The pyboard has a 32.768khz crystal. Most esp boards don't and rely on the high speed xtal or RC oscillator which are less accurate. You can add a 32.768khz crystal to an esp32 (and I believe also esp8266) and get higher accuracy at the expense of higher HW cost and fewer I/O pins.

@MostlyHarmless: nice work! Now good luck getting a PR reviewed

[MostlyHarmless]

@MostlyHarmless: nice work! Now good luck getting a PR reviewed

That is the part that I haven't figured out in this community yet.

For example I have a PR open for the async-friendly improvements to the DHT driver, but so far it is more or less ignored. Do I have to open an issue in parallel to the PR on github or what?

[pythoncoder]

The response to PR's varies. Bear in mind that Christmas/New Year are at the height of summer in Australia. Damien may be on holiday.

In my experience PR's against MicroPython do get reviewed when other commitments permit. Those against micropython-lib have historically tended to languish.

[tve]

I didn't realize Damien is in AU. Thanks for the info.

WRT clock sync, does the 8266 RTC always run off the RC? Can't be changed to run off the xtal when not in sleep mode?

[MostlyHarmless]

WRT clock sync, does the 8266 RTC always run off the RC? Can't be changed to run off the xtal when not in sleep mode?

I think it can run on a dedicated 32kHz XTAL while not in sleep, but not on the hight speed CPU XTAL, like the esp32 does. Like you said most boards lack that 32kHz source, including the boards I have.

Running off the 150kHz R/C oscillator the best I was able to do so far is the RTC.calibrate() function, for which there is a PR here. Calling that function every 2 seconds alone stabilizes the clock quite a bit. I don't know what happens on a board that has a dedicated 32kHz.

Note that the function doesn't actually affect the oscillator frequency at all. It just changes the multiplier, that RTC.datetime() is using, to determine us since 2000-01-01 based on the RTC ticks. It not only changes that multiplier but also recalculates the delta for the RTC ticks, so that RTC.datetime() stays monotonic.


Regards, Jan

[tve]

I'm puzzled by the need to recalibrate every few seconds... Isn't the variation primarily due to temperature fluctuations? Most systems have a relatively stable temperature environment on the scale of minutes. Now, due to changes in workload (e.g. Wifi TX) the chip temperature can vary a lot. But for most IoT stuff I'd expect high periodicity and again, at the scale of a few minutes, quite some stability. So I would guess that if you measured drift over, say, a one minute period you'd find quite some consistency minute-by-minute. If you compare RC vs XTAL over a few milliseconds then that will depend highly on the momentary power consumption. Just guesses...