[SOLVED] [Unix port] examples/network/http_client_ssl.py only works for google.com?

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
Post Reply
HermannSW
Posts: 181
Joined: Wed Nov 01, 2017 7:46 am
Contact:

[SOLVED] [Unix port] examples/network/http_client_ssl.py only works for google.com?

Post by HermannSW » Sat Apr 17, 2021 11:23 am

Both, Unix port MicroPython as well as python3, only work for example going against google.com -- any other https target does not:

OSError -40 for "en.wikipedia.org".

Domain not found for "nytimes.com":

Code: Select all

...
<_SSLSocket 11f9cf0>
b'HTTP/1.1 500 Domain Not Found\r\nConnection: close\r\nContent-Length: 221\r\nServer: Varnish\r\nRetry-After: 0\r\ncontent-type: text/html\r\nCache-Control: private, no-cache\r\nX-Served-By: cache-fra19157-FRA\r\nAccept-Ranges: bytes\r\nDate: Sat, 17 Apr 2021 11:21:18 GMT\r\nVia: 1.1 varnish\r\n\r\n\n<html>\n<head>\n<title>Fastly error: unknown domain </title>\n</head>\n<body>\n<p>Fastly error: unknown domain: . Please check that this domain has been added to a service.</p>\n<p>Details: cache-fra19157-FRA</p></body></html>'
$

Bad request for "stamm-wilbrandt.de":

Code: Select all

...
<_SSLSocket 5c65f8>
b'HTTP/1.1 400 Bad Request\r\nServer: nginx\r\nDate: Sat, 17 Apr 2021 11:24:55 GMT\r\nContent-Type: text/html; charset=iso-8859-1\r\nContent-Length: 226\r\nConnection: close\r\n\r\n<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br />\n</p>\n</body></html>\n'
$
What is wrong with the example or with MicroPython?
Last edited by HermannSW on Thu Apr 22, 2021 8:54 am, edited 1 time in total.
Pico (USB) reboot/reflash tool
https://gist.github.com/Hermann-SW/ca07 ... 6d81de01a7

Pico HTTPS GET through ESP01 serial2wifi
viewtopic.php?f=21&t=10122

Wireless Pico Micropython REPL
viewtopic.php?f=21&t=10109

HermannSW
Posts: 181
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: [Unix port] examples/network/http_client_ssl.py only works for google.com?

Post by HermannSW » Sat Apr 17, 2021 7:34 pm

I think the example is wrong (works only for google.com, needs other code for other websites).

Does not work for http without ussl either:

Code: Select all

pi@raspberrypi4B:~/pico/micropython/ports/unix $ cat nssl.py 
import usocket as _socket

s=_socket.socket()
ai=_socket.getaddrinfo('neverssl.com',80)
s.connect(ai[0][-1])
s.write(b"GET / HTTP/1.0\r\n\r\n")
print(s.read(4096))
pi@raspberrypi4B:~/pico/micropython/ports/unix $

Something is missing in that short script for not getting HTTP 403 forbidden response:

Code: Select all

pi@raspberrypi4B:~/pico/micropython/ports/unix $ ./micropython nssl.py 
b'HTTP/1.1 403 Forbidden\r\nServer: CloudFront\r\nDate: Sat, 17 Apr 2021 19:27:15 GMT\r\nContent-Type: text/html\r\nContent-Length: 915\r\nX-Cache: Error from cloudfront\r\nX-Amz-Cf-Pop: HAM50-C1\r\nX-Amz-Cf-Id: Q0E5gegX2lk3nb_TA6adbFNmnWJ68LVkaUihFkI7dr_vHzE79GSUgQ==\r\nX-Cache: MISS from uni-heidelberg.eSienet.de\r\nX-Cache-Lookup: MISS from uni-heidelberg.eSienet.de:8080\r\nVia: 1.1 bf65a83733ea7a81d9100310d3bbbfb8.cloudfront.net (CloudFront), 1.1 uni-heidelberg.eSienet.de (squid/4.4)\r\nConnection: close\r\n\r\n<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">\n<TITLE>ERROR: The request could not be satisfied</TITLE>\n</HEAD><BODY>\n<H1>403 ERROR</H1>\n<H2>The request could not be satisfied.</H2>\n<HR noshade size="1px">\nBad request.\nWe can\'t connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.\n<BR clear="all">\nIf you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.\n<BR clear="all">\n<HR noshade size="1px">\n<PRE>\nGenerated by cloudfront (CloudFront)\nRequest ID: Q0E5gegX2lk3nb_TA6adbFNmnWJ68LVkaUihFkI7dr_vHzE79GSUgQ==\n</PRE>\n<ADDRESS>\n</ADDRESS>\n</BODY></HTML>'
pi@raspberrypi4B:~/pico/micropython/ports/unix $

"curl" does not have any issue to access neverssl.com ...

Code: Select all

$ curl http://neverssl.com/
<html>
    <head>
        <title>NeverSSL - helping you get online</title>

        <style>
        body {
             font-family: Montserrat, helvetica, arial, sans-serif; 
             font-size: 16x;
             color: #444444;
             margin: 0;
        }
        h2 {
            font-weight: 700;
            font-size: 1.6em;
            margin-top: 30px;
        }
        p {
            line-height: 1.6em;
        }
        .container {
            max-width: 650px;
            margin: 20px auto 20px auto;
            padding-left: 15px;
            padding-right: 15px
        }
        .header {
            background-color: #42C0FD;
            color: #FFFFFF;
            padding: 10px 0 10px 0;
            font-size: 2.2em;
        }
        <!-- CSS from Mark Webster https://gist.github.com/markcwebster/9bdf30655cdd5279bad13993ac87c85d -->
        </style>
    </head>
    <body>

    <div class="header">
        <div class="container">
        <h1>NeverSSL</h1>
        </div>
    </div>
    
    <div class="content">
    <div class="container">

    <h2>What?</h2>
    <p>This website is for when you try to open Facebook, Google, Amazon, etc
    on a wifi network, and nothing happens. Type "http://neverssl.com"
    into your browser's url bar, and you'll be able to log on.</p>

    <h2>How?</h2>
    <p>neverssl.com will never use SSL (also known as TLS). No
    encryption, no strong authentication, no <a
    href="https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security">HSTS</a>,
    no HTTP/2.0, just plain old unencrypted HTTP and forever stuck in the dark
    ages of internet security.</p> 

    <h2>Why?</h2>
    <p>Normally, that's a bad idea. You should always use SSL and secure
    encryption when possible. In fact, it's such a bad idea that most websites
    are now using https by default.</p>
    
    <p>And that's great, but it also means that if you're relying on
    poorly-behaved wifi networks, it can be hard to get online.  Secure
    browsers and websites using https make it impossible for those wifi
    networks to send you to a login or payment page. Basically, those networks
    can't tap into your connection just like attackers can't. Modern browsers
    are so good that they can remember when a website supports encryption and
    even if you type in the website name, they'll use https.</p> 

    <p>And if the network never redirects you to this page, well as you can
    see, you're not missing much.</p>

    </div>
    </div>
    
    </body>
</html>
$
Pico (USB) reboot/reflash tool
https://gist.github.com/Hermann-SW/ca07 ... 6d81de01a7

Pico HTTPS GET through ESP01 serial2wifi
viewtopic.php?f=21&t=10122

Wireless Pico Micropython REPL
viewtopic.php?f=21&t=10109

HermannSW
Posts: 181
Joined: Wed Nov 01, 2017 7:46 am
Contact:

Re: [Unix port] examples/network/http_client_ssl.py only works for google.com?

Post by HermannSW » Sat Apr 17, 2021 9:50 pm

google.com domain is really different.

Doing HTTP request with slightly modified script ...

Code: Select all

pi@raspberrypi4B:~/pico/micropython/ports/unix $ diff nssl.py s.py 
4c4
< ai=_socket.getaddrinfo('neverssl.com',80)
---
> ai=_socket.getaddrinfo('google.com',80)
pi@raspberrypi4B:~/pico/micropython/ports/unix $ 

... shows HTTP 301 response ...

Code: Select all

pi@raspberrypi4B:~/pico/micropython/ports/unix $ ./micropython s.py
b'HTTP/1.1 301 Moved Permanently\r\nLocation: http://www.google.com/\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Sat, 17 Apr 2021 21:44:58 GMT\r\nExpires: Mon, 17 May 2021 21:44:58 GMT\r\nCache-Control: public, max-age=2592000\r\nServer: gws\r\nContent-Length: 219\r\nX-XSS-Protection: 0\r\nX-Frame-Options: SAMEORIGIN\r\nX-Cache: MISS from uni-heidelberg.eSienet.de\r\nX-Cache-Lookup: MISS from uni-heidelberg.eSienet.de:8080\r\nVia: 1.1 uni-heidelberg.eSienet.de (squid/4.4)\r\nConnection: close\r\n\r\n<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\n<TITLE>301 Moved</TITLE></HEAD><BODY>\n<H1>301 Moved</H1>\nThe document has moved\n<A HREF="http://www.google.com/">here</A>.\r\n</BODY></HTML>\r\n'
pi@raspberrypi4B:~/pico/micropython/ports/unix $



... as "curl" does as well:

Code: Select all

$ curl http://google.com/
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
$
Pico (USB) reboot/reflash tool
https://gist.github.com/Hermann-SW/ca07 ... 6d81de01a7

Pico HTTPS GET through ESP01 serial2wifi
viewtopic.php?f=21&t=10122

Wireless Pico Micropython REPL
viewtopic.php?f=21&t=10109

HermannSW
Posts: 181
Joined: Wed Nov 01, 2017 7:46 am
Contact:

[SOLVED] Re: [Unix port] examples/network/http_client_ssl.py only works for google.com?

Post by HermannSW » Thu Apr 22, 2021 8:53 am

I identified root cause:
google.com is a bit sloppy, other http[s] servers need at least "Host" header presented!

I fixed both network http[s] client examples, see this commit on my micropython fork for details:
https://github.com/Hermann-SW/micropyth ... cf85fbb86d

Now both work with unix micropython port (eg. http example)

Code: Select all

pi@raspberrypi4B:~/pico/micropython/ports/unix $ ./micropython ../../examples/network/http_client.py 
Address infos: [(2, 1, 6, None, bytearray(b'\x02\x00\x00PA\tE\xb1\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 2, 17, None, bytearray(b'\x02\x00\x00PA\tE\xb1\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 3, 0, None, bytearray(b'\x02\x00\x00PA\tE\xb1\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 1, 6, None, bytearray(b'\x02\x00\x00PA\tE\xb8\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 2, 17, None, bytearray(b'\x02\x00\x00PA\tE\xb8\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 3, 0, None, bytearray(b'\x02\x00\x00PA\tE\xb8\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 1, 6, None, bytearray(b'\x02\x00\x00PA\tEx\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 2, 17, None, bytearray(b'\x02\x00\x00PA\tEx\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 3, 0, None, bytearray(b'\x02\x00\x00PA\tEx\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 1, 6, None, bytearray(b'\x02\x00\x00PA\tE\x05\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 2, 17, None, bytearray(b'\x02\x00\x00PA\tE\x05\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 3, 0, None, bytearray(b'\x02\x00\x00PA\tE\x05\x00\x00\x00\x00\x00\x00\x00\x00'))]
Connect address: bytearray(b'\x02\x00\x00PA\tE\xb1\x00\x00\x00\x00\x00\x00\x00\x00')
b'HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 2536\r\nLast-Modified: Thu, 19 Nov 2020 22:29:21 GMT\r\nAccept-Ranges: bytes\r\nServer: AmazonS3\r\nDate: Thu, 22 Apr 2021 06:35:14 GMT\r\nETag: "e8bb9152091d61caa9d69fed8c4aebc6"\r\nVary: Accept-Encoding\r\nX-Cache: Hit from cloudfront\r\nX-Amz-Cf-Pop: FRA56-C1\r\nX-Amz-Cf-Id: KjXhCDjU-DpeNgjhqq-75fSvu2uyc3C5-FZWYsJWpw5Jh20YMuG7TA==\r\nAge: 8063\r\nX-Cache: HIT from uni-heidelberg.eSienet.de\r\nX-Cache-Lookup: HIT from uni-heidelberg.eSienet.de:8080\r\nVia: 1.1 d947c3ab534102b2c9a7f0a4541d2ed9.cloudfront.net (CloudFront), 1.1 uni-heidelberg.eSienet.de (squid/4.4)\r\nConnection: close\r\n\r\n<html>\n    <head>\n        <title>NeverSSL - helping you get online</title>\n\n        <style>\n        body {\n             font-family: Montserrat, helvetica, arial, sans-serif; \n             font-size: 16x;\n             color: #444444;\n             margin: 0;\n        }\n        h2 {\n            font-weight: 700;\n            font-size: 1.6em;\n            margin-top: 30px;\n        }\n        p {\n            line-height: 1.6em;\n        }\n        .container {\n            max-width: 650px;\n            margin: 20px auto 20px auto;\n            padding-left: 15px;\n            padding-right: 15px\n        }\n        .header {\n            background-color: #42C0FD;\n            color: #FFFFFF;\n            padding: 10px 0 10px 0;\n            font-size: 2.2em;\n        }\n        <!-- CSS from Mark Webster https://gist.github.com/markcwebster/9'
pi@raspberrypi4B:~/pico/micropython/ports/unix $ 

as well as with python3 (eg. https example):

Code: Select all

pi@raspberrypi4B:~/pico/micropython/ports/unix $ python3 ../../examples/network/http_client_ssl.py 
Address infos: [(2, 1, 6, '', ('93.90.177.145', 443)), (2, 2, 17, '', ('93.90.177.145', 443)), (2, 3, 0, '', ('93.90.177.145', 443))]
Connect address: ('93.90.177.145', 443)
<ssl.SSLSocket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.14.247.224', 58898), raddr=('93.90.177.145', 443)>
b'HTTP/1.1 200 OK\r\nServer: nginx\r\nDate: Thu, 22 Apr 2021 08:50:49 GMT\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Length: 290\r\nConnection: close\r\nVary: Accept-Encoding\r\nAccess-Control-Allow-Origin: *\r\nVary: Accept-Encoding\r\n\r\n<pre>60000\n\n.........\n...   ...\n...   ...\n.       .\n.    oo .\n.       .\n...   ...\n...   ...\n.........\n\n.........\n...   ...\n...   ...\n.       .\n.    oX .\n.       .\n...   ...\n...   ...\n.........\n\n.........\n...   ...\n...   ...\n.       .\n.   o   .\n.       .\n...   ...\n...   ...\n.........\n</pre>'
pi@raspberrypi4B:~/pico/micropython/ports/unix $ 
Pico (USB) reboot/reflash tool
https://gist.github.com/Hermann-SW/ca07 ... 6d81de01a7

Pico HTTPS GET through ESP01 serial2wifi
viewtopic.php?f=21&t=10122

Wireless Pico Micropython REPL
viewtopic.php?f=21&t=10109

Post Reply