urequests and SSL on WiPy

Questions and discussion about The WiPy 1.0 board and CC3200 boards.
Target audience: Users with a WiPy 1.0 or CC3200 board.
jgmdavies
Posts: 57
Joined: Tue Aug 09, 2016 2:39 pm

Re: urequests and SSL on WiPy

Post by jgmdavies » Mon Sep 05, 2016 5:31 pm

@danielm

No, sorry, I just wanted to get it working with HTTPS. Are you sure you need the certificate stuff?

I tried a GET from https://www.google.com and got:

File "urequests.py", line 116, in get
File "urequests.py", line 104, in request
NotImplementedError: Redirects not yet supported

(N.B. my line numbers may be different to yours as I've modified urequests.py a little.)

Code was:

Code: Select all

import urequests

url = "https://www.google.com"
resp = urequests.get(url)

print(resp.content[:100])
Jim

danielm
Posts: 167
Joined: Mon Oct 05, 2015 12:24 pm

Re: urequests and SSL on WiPy

Post by danielm » Mon Sep 05, 2016 7:44 pm

Yes, server certificate validation is a counter-measure against Man-in-the-middle attacks and DNS spoofing.

Try to call GET on your local google domain. Check in the browser where you will be redirected from google.com.

jgmdavies
Posts: 57
Joined: Tue Aug 09, 2016 2:39 pm

Re: urequests and SSL on WiPy

Post by jgmdavies » Mon Sep 05, 2016 8:25 pm

@danielm

I tried it on http://www.google.co.uk and got:

Code: Select all

  File "google1.py", line 6, in <module>
  File "urequests.py", line 25, in content
MemoryError: memory allocation failed, allocating 39168 bytes
Need a bigger WiPy! ;)

danielm
Posts: 167
Joined: Mon Oct 05, 2015 12:24 pm

Re: urequests and SSL on WiPy

Post by danielm » Tue Sep 06, 2016 7:41 am

That means it worked - the client was able to get content of that url :)

Wait for ESP32(LoPy) - it should be equipped with approx. 512kB of RAM - not sure what portion of that will be available for users of LoPy MP port. Hope they will share some info soon because they should ship in the end of the month: https://www.kickstarter.com/projects/17 ... ts/1652522

jgmdavies
Posts: 57
Joined: Tue Aug 09, 2016 2:39 pm

Re: urequests and SSL on WiPy

Post by jgmdavies » Tue Sep 06, 2016 9:14 am

Thanks danielm, I now know a bit more about 'MITM' attacks!

I was going to try the certificate business on the WiPy - how would I know if it's working correctly, apart from trying it with and without the cert file?

Jim

danielm
Posts: 167
Joined: Mon Oct 05, 2015 12:24 pm

Re: urequests and SSL on WiPy

Post by danielm » Tue Sep 06, 2016 10:40 am

You can flash incorrect/different CA certificate as a 'negative' test. I believe that if working correctly, you should get some error code or at least you should not receive any http response.

jgmdavies
Posts: 57
Joined: Tue Aug 09, 2016 2:39 pm

Re: urequests and SSL on WiPy

Post by jgmdavies » Tue Sep 06, 2016 12:25 pm

@danielm

OK, I may have got it to work - there's conflicting info out there in Google land... This was a helpful post: viewtopic.php?t=1089

The cert file has to be in DER format, but must be named ca.pem. If you FTP it to /flash/cert it will 'disappear', which is expected as it's automatically moved to internal storage.

Here's my test code:

Code: Select all

import machine
from machine import RTC
import urequests

rtc = RTC(datetime=(2016, 9, 6, 9, 0, 0, 0, None)) # init with a specific time and date
print(rtc.now())

url = "https://www.google.co.uk"
resp = urequests.get(url)

print("Response from GET to: " + url + ", status code " + str(resp.status_code))
print(resp.content[:100])
Note that the RTC must be set for server certificate validation to work.

And here's my version of urequests.py:

Code: Select all

# urequests.py

# From:  https://github.com/micropython/micropython-lib/blob/master/urequests/urequests.py
# with changes by J.G.Davies, Jacobus Systems Ltd.


import usocket

class Response:

	def __init__(self, f):
		self.raw = f
		self.encoding = "utf-8"
		self._cached = None

	def close(self):
		if self.raw:
			self.raw.close()
			self.raw = None
		self._cached = None

	@property
	def content(self):
		if self._cached is None:
			self._cached = self.raw.read()
			self.raw.close()
			self.raw = None
		return self._cached

	@property
	def text(self):
		return str(self.content, self.encoding)

	def json(self):
		import ujson
		return ujson.loads(self.content)


def request(method, url, data=None, json=None, headers={}, stream=None):
	try:
		proto, dummy, host, path = url.split("/", 3)
	except ValueError:
		proto, dummy, host = url.split("/", 2)
		path = ""

	if proto == "http:":
		port = 80
	elif proto == "https:":
		import ussl
		port = 443
	else:
		raise ValueError("Unsupported protocol: " + proto)

	if ":" in host:
		host, port = host.split(":", 1)
		port = int(port)

	ai = usocket.getaddrinfo(host, port)
	addr = ai[0][4]

	# JGD 5 Sep 2016
	if proto == "https:":
		s = usocket.socket(usocket.AF_INET, usocket.SOCK_STREAM, usocket.IPPROTO_SEC)
	else:
		s = usocket.socket()

	if proto == "https:":
		# JGD 6 Sep 2016
		#s = ussl.wrap_socket(s)
		s = ussl.wrap_socket(s, keyfile=None, certfile=None, server_side=False, \
			cert_reqs=ussl.CERT_REQUIRED, ca_certs='/flash/cert/ca.pem')

	# JGD 6 Sep 2016 - Connect *after* the wrap.
	s.connect(addr)

	s.write(b"%s /%s HTTP/1.0\r\n" % (method, path))
	if not "Host" in headers:
		s.write(b"Host: %s\r\n" % host)

	# Iterate over keys to avoid tuple alloc
	for k in headers:
		s.write(k)
		s.write(b": ")
		s.write(headers[k])
		s.write(b"\r\n")
	if json is not None:
		assert data is None
		import ujson
		data = ujson.dumps(json)
	if data:
		s.write(b"Content-Length: %d\r\n" % len(data))
	s.write(b"\r\n")
	if data:
		s.write(data)

	l = s.readline()
	protover, status, msg = l.split(None, 2)
	status = int(status)

	while True:
		l = s.readline()
		if not l or l == b"\r\n":
			break
		#print(line)
		if l.startswith(b"Transfer-Encoding:"):
			if b"chunked" in line:
				raise ValueError("Unsupported " + l)
		elif l.startswith(b"Location:"):
			raise NotImplementedError("Redirects not yet supported")

	resp = Response(s)
	resp.status_code = status
	resp.reason = msg.rstrip()
	return resp


def head(url, **kw):
	return request("HEAD", url, **kw)

def get(url, **kw):
	return request("GET", url, **kw)

def post(url, **kw):
	return request("POST", url, **kw)

def put(url, **kw):
	return request("PUT", url, **kw)

def patch(url, **kw):
	return request("PATCH", url, **kw)

def delete(url, **kw):
	return request("DELETE", url, **kw)
Note the changes to socket creation and wrapping, and the connect() after the wrapping.

This works with status 200 on google.co.uk (then the previous memory allocation error if I try to get the content). With a junk file substituted for ca.pem, I get OSError 456.

I'm happy to try other test methods if anyone wants.

HTH
Jim

danielm
Posts: 167
Joined: Mon Oct 05, 2015 12:24 pm

Re: urequests and SSL on WiPy

Post by danielm » Tue Sep 06, 2016 2:22 pm

Thank you Jim for extensive post. I will perform next test based on your recommendations.

OSError 456 means following:
SL_ESECBADCAFILE (-456) /* error secure level bad CA file */

By junk file you mean CA certificate with DER format which is not valid for testing server or just any file?

jgmdavies
Posts: 57
Joined: Tue Aug 09, 2016 2:39 pm

Re: urequests and SSL on WiPy

Post by jgmdavies » Tue Sep 06, 2016 2:51 pm

Sorry for vagueness - 'junk file' was a clone of the good ca.pem (which I created using Firefox), with the binary goodness replaced by a couple of lines of junk I typed.

So error 456 makes sense!

danielm
Posts: 167
Joined: Mon Oct 05, 2015 12:24 pm

Re: urequests and SSL on WiPy

Post by danielm » Tue Sep 06, 2016 2:53 pm

Just thinking - maybe it means that format of the file is not valid and not that server certificate was not validated with this CA certificate?

Post Reply