No webrepl with deactivated SSL/TLS (Hashlib missing sha1)

[kevinkk525]

I was building a firmware without ssl, tls, btree und UCRYPTOLIB to lose some weight and gain some heap by setting:
MICROPY_PY_USSL = 0
MICROPY_SSL_AXTLS = 0
MICROPY_PY_BTREE ?= 0
and in mpconfigport.h:
#define MICROPY_PY_UCRYPTOLIB (0)

This works rather well, however when I tried to use webrepl I realized that it uses uhashlib.sha1 but that got removed from uhashlib.
So I checked out the code of moduhashlib.c and it was a bummer:
Having set MICROPY_PY_UHASHLIB_SHA1 is completely useless if you want to build a firmware without SSL and TLS support because all sha1 functions are excluded because of MICROPY_SSL_AXTLS.
So no sha1 without either SSL_AXTLS and MICROPY_PY_USSL and since the implementation of course depends on those ssl/tls features.

So any idea how to solve this problem?

However activating MICROPY_PY_USSL and SSL_AXTLS doesn't seem to cause any trouble with the increased heap size of 44kB so I'm still happy.

[jimmo]

The linker is very clever about dropping code that isn't used. So the way I'd approach this is to disable functionality that pulls in code from axtls, rather than disabling axtls itself. In other words, disabling USSL should be sufficient to realise any savings.

I'm kind of surprised that you gain much (if any) heap by removing code though? Does ussl / axtls have any big static buffers? What were the before/after BSS measurements (without changing the micropython heap).

[kevinkk525]

Disabling USSL also removes sha1 and therefore no webrepl.

I have no idea which C modules use big buffers, I was just following another thread disabling (presumably) big modules that I won't need anyway in an attempt to regain more heap as I was running out of heap again..
Since reenabling ussl and axtls didn't give me any trouble with my increased heap of 44kB (from 38kB), I assume that axtls and ussl don't use any big buffers.
However I still have ucryptolib and btree disabled, maybe on of those modules uses big buffers, else I don't know why the heap is not bigger by default.

What's "BSS measurements"?

[jimmo]

I think the fact that uhashlib depends on ussl (I can see it in esp8266/mpconfigport.h) is possibly a mistake? Or at least needs something cleverer... Can you try just changing it to

Code: Select all

#define MICROPY_PY_UHASHLIB_SHA1    (MICROPY_SSL_AXTLS)

See this thread viewtopic.php?f=16&t=6761&p=38901 for more info about heap size (and BSS, etc).

[kevinkk525]

Thanks for the link, somehow I missed that thread in my research..
So apparently my changes do nothing and I'm just luky my changes to the heap size work without crashing the device. Good to know..
Have to wait for littleFS on esp8266 to get 4kB back safely.


I did try compiling firmware with your changes and it builds correctly but no sha1 method was available. So I changed it to #define MICROPY_PY_UHASHLIB_SHA1 (1) and then I got an error building:

Code: Select all

CC ../../extmod/moducryptolib.c
In file included from ../../py/mpstate.h:35:0,
                 from ../../py/runtime.h:29,
                 from ../../extmod/moduhashlib.c:30:
../../extmod/moduhashlib.c:226:60: error: 'uhashlib_sha1_digest' undeclared here (not in a function)
 STATIC MP_DEFINE_CONST_FUN_OBJ_1(uhashlib_sha1_digest_obj, uhashlib_sha1_digest);
                                                            ^
../../py/obj.h:277:46: note: in definition of macro 'MP_DEFINE_CONST_FUN_OBJ_1'
         {{&mp_type_fun_builtin_1}, .fun._1 = fun_name}
                                              ^
../../extmod/moduhashlib.c:237:17: error: 'uhashlib_sha1_make_new' undeclared here (not in a function)
     .make_new = uhashlib_sha1_make_new,
                 ^
../../extmod/moduhashlib.c:156:17: error: 'uhashlib_sha1_update' used but never defined [-Werror]
 STATIC mp_obj_t uhashlib_sha1_update(mp_obj_t self_in, mp_obj_t arg);
                 ^
CC ../../extmod/modubinascii.c
cc1: all warnings being treated as errors
../../py/mkrules.mk:47: recipe for target 'build-GENERIC/extmod/moduhashlib.o' failed

Apparently MICROPY_PY_UHASHLIB_SHA1 still gets disabled if ussl is disabled.
If I activate ussl again with MICROPY_PY_UHASHLIB_SHA1 still set to (MICROPY_SSL_AXTLS) then sha1 is available.

So somehow you do actually need both AXTLS and USSL. Not sure if it is worth investigating why. Probably easier to just know that webrepl won't work if you disable ssl.

[jimmo]

In extmod.mk, it also has the same "only do axtls/mbedss if ussl is enabled".

Here's a diff showing the changes I needed to build with sha1 but without ssl:

Code: Select all

diff --git a/extmod/extmod.mk b/extmod/extmod.mk
index e714b6028..a92c0bff2 100644
--- a/extmod/extmod.mk
+++ b/extmod/extmod.mk
@@ -31,6 +31,7 @@ endif
 
 ifeq ($(MICROPY_PY_USSL),1)
 CFLAGS_MOD += -DMICROPY_PY_USSL=1
+endif
 ifeq ($(MICROPY_SSL_AXTLS),1)
 CFLAGS_MOD += -DMICROPY_SSL_AXTLS=1 -I$(TOP)/lib/axtls/ssl -I$(TOP)/lib/axtls/crypto -I$(TOP)/extmod/axtls-include
 AXTLS_DIR = lib/axtls
@@ -127,7 +128,6 @@ SRC_MOD += $(addprefix $(MBEDTLS_DIR)/library/,\
 	xtea.c \
 	)
 endif
-endif
 
 ################################################################################
 # lwip
diff --git a/ports/esp8266/Makefile b/ports/esp8266/Makefile
index f1b718c78..c47bfe4ae 100644
--- a/ports/esp8266/Makefile
+++ b/ports/esp8266/Makefile
@@ -19,7 +19,7 @@ include ../../py/mkenv.mk
 QSTR_DEFS = qstrdefsport.h #$(BUILD)/pins_qstr.h
 QSTR_GLOBAL_DEPENDENCIES = $(BOARD_DIR)/mpconfigboard.h
 
-MICROPY_PY_USSL = 1
+MICROPY_PY_USSL = 0
 MICROPY_SSL_AXTLS = 1
 AXTLS_DEFS_EXTRA = -Dabort=abort_ -DRT_MAX_PLAIN_LENGTH=1024 -DRT_EXTRA=4096
 MICROPY_FATFS ?= 1
diff --git a/ports/esp8266/boards/GENERIC/mpconfigboard.h b/ports/esp8266/boards/GENERIC/mpconfigboard.h
index a7cacb815..5ac44e40c 100644
--- a/ports/esp8266/boards/GENERIC/mpconfigboard.h
+++ b/ports/esp8266/boards/GENERIC/mpconfigboard.h
@@ -17,5 +17,5 @@
 #define MICROPY_PY_IO_FILEIO            (1)
 #define MICROPY_PY_SYS_STDIO_BUFFER     (1)
 #define MICROPY_PY_URE_SUB              (1)
-#define MICROPY_PY_UCRYPTOLIB           (1)
+#define MICROPY_PY_UCRYPTOLIB           (0)
 #define MICROPY_PY_FRAMEBUF             (1)
diff --git a/ports/esp8266/mpconfigport.h b/ports/esp8266/mpconfigport.h
index 726319392..89de5f673 100644
--- a/ports/esp8266/mpconfigport.h
+++ b/ports/esp8266/mpconfigport.h
@@ -64,7 +64,7 @@
 #define MICROPY_PY_UBINASCII        (1)
 #define MICROPY_PY_UCTYPES          (1)
 #define MICROPY_PY_UHASHLIB         (1)
-#define MICROPY_PY_UHASHLIB_SHA1    (MICROPY_PY_USSL && MICROPY_SSL_AXTLS)
+#define MICROPY_PY_UHASHLIB_SHA1    (MICROPY_SSL_AXTLS)
 #define MICROPY_PY_UHEAPQ           (1)
 #define MICROPY_PY_UTIMEQ           (1)
 #define MICROPY_PY_UJSON            (1)

[kevinkk525]

Thanks a lot jimmo, that does indeed work!