JSON loading and dumping efficiently with sockets

[manseekingknowledge]

I use the code below to send/receive JSON data via sockets. I'm looking for ways to improve memory efficiency where the functions json.loads() and json.dumps() are called. In the send function json.dumps() is called and then the return string is encoded. I'm pretty sure the encode() function is making a copy of the string. I would like to avoid that copy if possible. The only way I can think to do this is to use json.dump() and stream the output to a file on the ESP, but I don't want to do that. I'm trying to make my application as bulletproof as possible, and that includes accounting for power interruptions. If power is lost on a write to the ESP's file system my experience has been that the unit must be re-flashed.

So with that said, is there any way to perform the encoding operations in my send() and recv() functions in a way that is more efficient with memory? Can I somehow use the socket.makefile() function as a stream maybe?

Code: Select all

import json
import struct
import sys


if sys.implementation.name != 'micropython':
    import socket

    # Remap CPython's recv_into function to MicroPython's readinto function
    socket.socket.readinto = socket.socket.recv_into

else:
    import micropython


def send(s, data):
    # if sys.implementation.name == 'micropython':
    #     print('\n\npre jsonsocket send\n\n')
    #     micropython.mem_info(1)

    try:
        if sys.implementation.name == 'micropython':
            data = bytearray(json.dumps(data).encode())

        else:
            data = bytearray(json.dumps(data, separators=(',', ':')).encode())

    except (TypeError, ValueError):
        raise Exception('Only JSON formatted data can be sent')

    # Prefix each message with the 4 byte size of the serialized data (network byte order)
    data[:0] = struct.pack('>I', len(data))
    s.sendall(data)

    # if sys.implementation.name == 'micropython':
    #     print('\n\npost jsonsocket send\n\n')
    #     micropython.mem_info(1)


def recv(s):
    # if sys.implementation.name == 'micropython':
    #     print('\n\npre jsonsocket recv\n\n')
    #     micropython.mem_info(1)

    # Get the packed size of the serialized data by reading the first 4 bytes of the message (network byte order)
    data = s.recv(struct.calcsize('>I'))
    if not data:
        return None

    # Unpack the size of the serialized data
    data_size = struct.unpack('>I', data)[0]

    # Use a memoryview to receive the data chunk by chunk efficiently
    data = memoryview(bytearray(data_size))
    next_offset = 0
    while data_size - next_offset > 0:
        next_offset += s.readinto(data[next_offset:], data_size - next_offset)

    try:
        data = json.loads(str(data, 'utf-8'))

    except (TypeError, ValueError):
        raise Exception('Only JSON formatted data can be received')

    # if sys.implementation.name == 'micropython':
    #     print('\n\npost jsonsocket recv\n\n')
    #     micropython.mem_info(1)

    return data