The optimization thread... ;)

jedie · Post by **jedie** » Fri Nov 29, 2019 2:39 pm

import uos as os
import ubinascii as binascii
import uhashlib as hashlib

CHUNK_SIZE = const(1024)
BUFFER = bytearray(CHUNK_SIZE)
ENCODING = 'utf-8'

FILE_TYPE=const(0x8000)

for name, type, inode, size in os.ilistdir():
    if type != FILE_TYPE:
        print(' *** Skip: %s' % name)
        continue

    print('%25s %5i Bytes SHA256:' % (name, size), end='')

    sha256 = hashlib.sha256()
    with open(name, 'rb') as f:
        while True:
            count = f.readinto(BUFFER, CHUNK_SIZE)
            if count < CHUNK_SIZE:
                sha256.update(BUFFER[:count])
                break
            else:
                sha256.update(BUFFER)

    print(binascii.hexlify(sha256.digest()).decode(ENCODING))

There are no print instructions in the productive code and the result goes: socket.sendall(binascii.hexlify(sha256.digest()))
The current product code is here: https://github.com/jedie/micropython-so ... #L104-L124

jimmo · Post by **jimmo** » Sun Dec 01, 2019 11:05 am

Did you have a question about the code snippet?

jedie · Post by **jedie** » Sun Dec 01, 2019 11:50 am

Is there a way to optimize this?

Maybe use memoryview for BUFFER[:count] ??

jimmo · Post by **jimmo** » Sun Dec 01, 2019 12:02 pm

Yes I would imagine a memoryview would help here, but unless your use case is mostly a lot of small files, rather than a small number of large files, then it's not going to be a big factor.

The way I'd approach optimizing this is
- Measure it
- Figure out what part is actually slow

So the first part is easy -- use time.ticks_us() and time.ticks_diff to measure how long it takes.
The second part is to stub out various parts and see how much they impact the total time.
i.e. if most of the time is taken by reading data from the files (i.e. if you don't actually generate the sha), then there isn't going to be much you can do to improve it. (Although maybe playing with the buffer size will help).

jedie · Post by **jedie** » Sun Dec 01, 2019 4:26 pm

Foremost I am worried about RAM consumption. It's fairly fast enough

jimmo · Post by **jimmo** » Sun Dec 01, 2019 11:53 pm

Well the memoryview will definitely help there. But also making the BUFFER only exist for the duration of the function means that the memory can be freed after the function exits.

i.e.

Code: Select all

ENCODING = 'utf-8'
FILE_TYPE=const(0x8000)

def foo():
  for name, type, inode, size in os.ilistdir():
      if type != FILE_TYPE:
          print(' *** Skip: %s' % name)
          continue

      print('%25s %5i Bytes SHA256:' % (name, size), end='')
    
      BUFFER = bytearray(CHUNK_SIZE)

      sha256 = hashlib.sha256()
      with open(name, 'rb') as f:
          while True:
              count = f.readinto(BUFFER, CHUNK_SIZE)
              if count < CHUNK_SIZE:
                  sha256.update(BUFFER[:count])
                  break
              else:
                  sha256.update(BUFFER)

      print(binascii.hexlify(sha256.digest()).decode(ENCODING))

jedie · Post by **jedie** » Mon Dec 02, 2019 6:08 am

On the other side: can increase the fragmentation, isn't it?

jedie · Post by **jedie** » Mon Dec 09, 2019 8:45 am

I distilled my web server code from my existing code to debug. Looks like this:

: Screenshot_2019-12-09 Minimal MicroPython Webserver.png (37.2 KiB) Viewed 3139 times

It used asyncio.start_server():

Code: Select all

import gc
import sys

import uasyncio as asyncio

_HEX = '0123456789ABCDEF'

_HTML_PREFIX = b"""
<html>
<head><title>Minimal MicroPython Webserver</title></head>
<body>
    <h1>Minimal MicroPython Webserver</h1>
    <pre>
"""
_HTML_SUFFIX = b"""
    </pre>
    <hr>
    <h2>POST test form:</h2>
    <form action="/test/post/" method="post">
        <textarea name="text" rows="4" cols="20">POST text
from textarea!</textarea>
        <p>
            <input type="checkbox" id="c1" name="c1" checked><label for="c1">c1</label>
            <input type="checkbox" id="c2" name="c2"><label for="c2">c2</label>
        </p>
        <input type="submit">
    </form>
    <hr>
    <h2>GET test form:</h2>
    <form action="/test/get/" method="get">
        <textarea name="text" rows="4" cols="20">GET text
from textarea!</textarea>
        <p>
            <input type="checkbox" id="c1" name="c1"><label for="c1">c1</label>
            <input type="checkbox" id="c2" name="c2" checked><label for="c2">c2</label>
        </p>
        <input type="submit">
    </form>
    <hr>
    <p>
"""
_HTML_FOOTER = """
    </p>
</body>"""


def unquote(string):
    string = string.replace('+', ' ')
    if '%' not in string:
        return string

    bits = string.split('%')
    if len(bits) == 1:
        return string

    res = [bits[0]]
    for item in bits[1:]:
        if len(item) >= 2:
            a, b = item[:2].upper()
            if a in _HEX and b in _HEX:
                res.append(chr(int(a + b, 16)))
                res.append(item[2:])
                continue

        res.append('%')
        res.append(item)

    return ''.join(res)


def parse_qsl(qs):
    if qs is None:
        return ()
    qs = str(qs)
    pairs = [s2 for s1 in qs.split('&') for s2 in s1.split(';')]
    res = []
    for name_value in pairs:
        try:
            name, value = name_value.split('=', 1)
        except ValueError:
            res.append((unquote(name_value), ''))
        else:
            res.append((unquote(name), unquote(value)))
    return res


def request_query2dict(qs):
    return dict(parse_qsl(qs))


class WebServer:
    async def parse_request(self, reader):
        method, url, http_version = (await reader.readline()).decode().strip().split()
        print(http_version)

        if '?' in url:
            url, querystring = url.split('?', 1)
        else:
            querystring = None

        # Consume all headers but use only content-length
        content_length = None
        while True:
            line = await reader.readline()
            if line == b'\r\n':
                break  # header ends

            try:
                header, value = line.split(b':', 1)
            except ValueError:
                break

            value = value.strip()

            if header == b'Content-Length':
                content_length = int(value.decode())

            print(header, value)

        print('content length:', content_length)

        # get body
        if content_length:
            body = await reader.read(content_length)
        else:
            body = None

        return method, url, querystring, body

    async def send_response(self, reader, writer):
        peername = writer.get_extra_info('peername')
        print('\nRequest from:', peername)
        await writer.awrite(b'HTTP/1.0 200 OK\r\n')
        await writer.awrite(b'Content-type: text/html; charset=utf-8\r\n\r\n')

        await writer.awrite(_HTML_PREFIX)

        await writer.awrite(b'Your IP: %s port:%s\n' % peername)

        await writer.awrite(b'\n')

        method, url, querystring, body = await self.parse_request(reader)

        await writer.awrite(b'Method: %s\n' % method)
        await writer.awrite(b'URL: %s\n' % url)
        await writer.awrite(b'querystring: %s\n' % querystring)
        await writer.awrite(b'parsed querystring: %s\n' % request_query2dict(querystring))
        await writer.awrite(b'body: %s\n' % body)
        await writer.awrite(b'parsed body: %s\n' % request_query2dict(body))

        await writer.awrite(_HTML_SUFFIX)

        alloc = gc.mem_alloc() / 1024
        free = gc.mem_free() / 1024

        await writer.awrite(
            b'RAM total: {total:.2f} KB, used: {alloc:.2f} KB, free: {free:.2f} KB'.format(
                total=alloc + free,
                alloc=alloc,
                free=free
            )
        )

        await writer.awrite(_HTML_FOOTER)
        await writer.aclose()

    async def request_handler(self, reader, writer):
        await self.send_response(reader, writer)
        gc.collect()

    def run(self):
        loop = asyncio.get_event_loop()
        loop.create_task(asyncio.start_server(self.request_handler, '0.0.0.0', 80))
        print('\nWeb server started...')
        loop.run_forever()


def main():
    from wifi import WiFi
    wifi = WiFi()
    if not wifi.is_connected:
        wifi.ensure_connection()
    del wifi
    del WiFi
    del sys.modules['wifi']
    gc.collect()

    server = WebServer()
    server.run()


if __name__ == '__main__':
    try:
        main()
    except Exception as e:
        sys.print_exception(e)

    print('Hard reset !')

    import machine
    machine.reset()

    import utime
    utime.sleep(1)

    print('sys.exit()')
    import sys
    sys.exit()

In my real code, I don't call request_query2dict() directly. I pass the raw data and only when the view really needs the data it parses it by calling request_query2dict().

Does anyone have any improvement suggestions for me?

MicroPython Forum (Archive)

The optimization thread... ;)

The optimization thread... ;)

Re: The optimization thread... ;)

Re: The optimization thread... ;)

Re: The optimization thread... ;)

Re: The optimization thread... ;)

Re: The optimization thread... ;)

Re: The optimization thread... ;)

Re: The optimization thread... ;)