ESP32, Gmail's SMTP server and uMail woes

[davef]

For the last 6 months I have tried sending
probably several hundred logs and a failure
pattern has started to emerge.

For some reason I latched on to 465 with SSL
as the most successful but decided to give 587
without SSL. That didn't work so tried going back
to 465 and now that wouldn't work. I erased,
re-flashed thecESP32 and 465 started working again.
Appears that doing a hard-reset on the ESP32 didn't
clear out all previous information or ??

One observation is that even when I get 465 with SSL
working sending test logs every minute or every hour
always seems to be successful. If I go to a 12
hour or 24 hour email send period then things stop working.
Most common error is 220 server not found.

I read through this information about Gmail's session
limits
https://developers.google.com/gmail/ima ... gth limits

Gmail POP sessions are limited about to 7 days.
Gmail IMAP sessions are limited to about 24 hours.
If the session was authenticated using OAuth credentials,
it's limited to about the validity period of the access
token used (usually 1 hour). A session in this context
is one continuous TCP connection.

When the time elapses and the session expires,
Gmail closes the connection with a message saying that
the session expired. After that, the client can reconnect,
authenticate again, and continue. If using OAuth, make
sure that the access token used is valid (if you try to
use an access token older than 1 hour, it might be invalid).

Sounds to me that the ESP32 would need go through the
authentication process every time if sending emails after
an hour has passed. But,surely it can cope with that?

I think I must be missing something important is setting
up Gmail on a ESP32 using uMail. Any suggestions?

Would I be more successful using a Pyboard-D for this quite
demanding task?

[CmdrDeLiver]

Most common error is 220 server not found.

Socket error 220 is "Host not Found". It means that the DNS lookup failed. This could be an error at your upstream DNS server or perhaps (and more likely) poor connectivity from your device to the Internet. What other errors are you getting?

gMail more or less requires you to enable "less" secure mode to push mail to the server. I'm assuming you have this enabled, but it's worth checking. It may be the reason you can only connect to port 465.

In order to to use smtp on gmail, you have to turn on the 'less secure app' option on your google account first.
https://support.google.com/accounts/answer/6010255

Port 465, while commonly used, is not actually part of a published Internet RFC (Standard). Port 587 is where SMTP connections with security (TLS/SSL/etc) should connect by default although many servers support this on port 25 as well. You can find an example of a port 587 connection to gMail here https://github.com/shawwwn/uMail as well as documentation on uMail. I'd highly recommend using that instead of connecting to port 465.

Gmail POP sessions are limited about to 7 days.
Gmail IMAP sessions are limited to about 24 hours.

You're not using POP to send mail. IMAP isn't supported by uMail.

If the session was authenticated using OAuth credentials,
it's limited to about the validity period of the access
token used (usually 1 hour). A session in this context
is one continuous TCP connection.

When the time elapses and the session expires,
Gmail closes the connection with a message saying that
the session expired.

This applies to sessions using an oAuth token which uMail doesn't appear to support.

Hope this helps.

[davef]

Hi CmdrDeLiver,

Thank you the comprehensive response. The main problem was that I followed an example that I ended up using my hotspot as the DNS server. Changing to 8.8.8.8 sorted that issue.

With regard to the 465 vs 587 issue I read a lot about the history and status. 465 with SSL on uMail is now reliable with uMail. For a week or so I have been getting reliable logs twice a day while on holiday. 587 without SSL as per his example doesn't seem to work for me. I will give it another try next week. One issue there was that changing the port rendered the ESP32 or ESP8277 "dead". Perhaps there was some other programming issue.

Thanks again for setting me straight on several issues.

[CmdrDeLiver]

using my hotspot as the DNS server

That will cause all kinds of problems for certain. Lots of hotspots will idle out connections if you're not pushing data. The internal DNS servers are often not configured to wake up the connection.

587 without SSL as per his example doesn't seem to work for me.

It won't on gMail. They are trying to enforce good practices an SSL is one of those.

changing the port rendered the ESP32 or ESP8277 "dead"

It was likely hung for some reason. Might be worth checking on the issues on the uMail git and possibly filing one. The Author is pretty responsive and very helpful.

[davef]

I filed an issue a week or so ago and got a quick response, which helped me to sort the DNS issue.

I have had to re-flash the firmware so many times, but didn't feel confident that it was an issue with uMail. I will try to repeat what I did to cause that problem and then maybe file an issue.

Thanks again.

[davef]

Hi CmdrDeLiver,

Still having grief with this.

You said:

It won't on gMail. They are trying to enforce good practices an SSL is one of those.

To my statement ... 587 without SSL as per his example doesn't seem to work for me.

Actually, over the last few months I have been using 587 without SSL and seem to get a similar number of fails as I did with 465 with SSL. Is TLS not more desirable?

At https://support.google.com/mail/answer/7126229?hl=en I found this:

smtp.gmail.com

Requires SSL: Yes

Requires TLS: Yes (if available)

Requires Authentication: Yes

Port for SSL: 465

Port for TLS/STARTTLS: 587

I read this as ... if you use 465 you need SSL and if you use 587 you don't need SSL but need TLS.

For a month or so I have been checking that smtp.gmail.com is available:

Code: Select all

s.connect(socket.getaddrinfo('smtp.gmail.com', 587, 0, SOCK_STREAM)[0][-1])

a few seconds before calling uMail which then sometimes fails at:

Code: Select all

addr = usocket.getaddrinfo(host, port)[0][-1]

Not being able to catch a ETIMEOUT error on usocket is making it difficult to find a path forward.

Any further suggestions gratefully received.

[davef]

Probably about 95% of my daily logs go through without a problem. However, there appear to two issues which still persist. So, using 587 and TLS:

1) sometimes it fails at the line:

Code: Select all

code, resp= self.cmd(CMD_EHLO + '_' + LOCAL_DOMAIN
where LOCAL_DOMAIN = '127.0.0.1'

so it doesn't even start the SMTP session.

2) sometimes the data (a .csv file) is sent but I don't get a 221 response. I have been catering for this error by re-sending the .csv file at the next logging period. If the send proceeds correctly I delete the .csv file.

I have done some reading about final line endings. uMail uses:

Code: Select all

self._sock.write('\r\n.\r\n') # the five letter sequence marked for ending

I also have another example that uses:

Code: Select all

write('...\n')

Also, I have seen:

Code: Select all

 ' . ' 

I am not convinced it is what is used for a line ending as the data is normally sent without the 221 response.

Any ideas about these two issues would be appreciated.