problem with hashlib on ARM Linux

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
willie
Posts: 14
Joined: Mon Dec 14, 2015 12:05 am

problem with hashlib on ARM Linux

Post by willie » Sun Aug 14, 2016 8:45 pm

hashlib/test_hashlib.py fails on Scaleway C1 ARM Linux system (comparable to Raspberry Pi 2). Running Debian 8, gcc 4.9.2

Micropython built from current master, commit info:

commit: 21967990519b3d0778770ddb0994216d012101c3
Author: Damien George <damien.p.george@gmail.com>
Date: Sun Aug 14 16:51:54 2016 +1000

I've debugged the python code a little. The error happens in the sha_final function, where in the last block (count= 19 chars) the function tries to zero out the last 45 chars of the block using sha_info['data'][:count] and is getting an empty array as this slice (sha_info['data'] is a list of 64 ints). So it looks like the slice is being made incorrectly.

The test works fine under micropython on x64 linux and under cpython 3.4.2 on the Scaleway ARM. I haven't tried any other systems.

I haven't tried debugging the C code yet. Thought I'd ask here first if anyone knew what might be happening.

I can supply ssh access to the Scaleway server if any of the devs want to try it. Email through my registered forum address is the quickest way to reach me if I haven't checked the forum lately.

willie
Posts: 14
Joined: Mon Dec 14, 2015 12:05 am

Re: problem with hashlib on ARM Linux

Post by willie » Mon Aug 15, 2016 9:57 pm

Update: I installed Ubuntu Precise (12.04) on another Scaleway instance, and rebuilt micropython with gcc 4.6.3 from the Ubtuntu repo. The test runs ok on that system. So there is either a compiler bug or some kind of UB in micropython. Has micropython been tested with UB sanitizers?

I also tried building with clang under jessie, but that failed because of some gcc-specific asm syntax in a file with inline asm. I also downloaded and tried to build gcc 6.1, but that hit snags of its own.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: problem with hashlib on ARM Linux

Post by stijn » Tue Aug 16, 2016 7:43 am

If you have the proper steps to reproduce this it's better if you create an issue at https://github.com/micropython/micropython/issues/new : forum is more for discussion, issue tracker is for issues aka bugs

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: problem with hashlib on ARM Linux

Post by pfalcon » Tue Aug 16, 2016 2:29 pm

Has micropython been tested with UB sanitizers?
If you test it, please share the fixes for the issues you find!
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

willie
Posts: 14
Joined: Mon Dec 14, 2015 12:05 am

Re: problem with hashlib on ARM Linux

Post by willie » Tue Aug 16, 2016 4:52 pm

Deshipu kindly opened https://github.com/micropython/micropython/issues/2323 and Damien responded that this appeared to be the same issue as #2264, fixed at almost the same time. I pulled new master and the test passes now. Thanks!

willie
Posts: 14
Joined: Mon Dec 14, 2015 12:05 am

Re: problem with hashlib on ARM Linux

Post by willie » Tue Aug 16, 2016 6:21 pm

I ran "make test" with -fsanitize=undefined and got a bunch of error reports about invalid left shifts. Output is attached as "ubsan.gif". It's actually a text file but I had to use the .gif extension to get the forum software to accept the upload.

Edit: attachment didn't work. Ok, pasting the output here (200 lines):

Code: Select all

../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/mpz.c:802:16: runtime error: negation of -9223372036854775808 cannot be represented in type 'long long int'; cast to an unsigned type to negate this value to itself
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -5
../py/mpz.c:1558:20: runtime error: left shift of 262144 by 16 places cannot be represented in type 'int'
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/runtime.c:463:17: runtime error: left shift of negative value -8
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/runtime.c:463:17: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -3
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -100
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/parse.c:232:46: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -3
../py/parse.c:232:46: runtime error: left shift of negative value -123
../py/runtime.c:463:17: runtime error: left shift of negative value -18
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/objint_mpz.c:327:9: runtime error: left shift of 1073741824 by 1 places cannot be represented in type 'int'
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/runtime.c:463:17: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1073741823
../py/parsenum.c:106:14: runtime error: left shift of 1073741824 by 1 places cannot be represented in type 'int'
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741823
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/runtime.c:463:17: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -100
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -3
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -3
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -128
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -100
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -123
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1234
../py/parse.c:232:46: runtime error: left shift of negative value -10
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/parse.c:232:46: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -128
../py/binary.c:172:13: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -128
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/../extmod/crypto-algorithms/sha256.c:49:19: runtime error: left shift of 128 by 24 places cannot be represented in type 'int'
modtime.c:90:30: runtime error: signed integer overflow: 1471371216 * 1000 cannot be represented in type 'long int'
modtime.c:82:30: runtime error: signed integer overflow: 1471371216 * 1000000 cannot be represented in type 'long int'
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1234136848
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233944736
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233592368
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1234439936
../py/parse.c:232:46: runtime error: left shift of negative value -268435456
../py/runtime.c:463:17: runtime error: left shift of negative value -268435456
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233985584
../py/parse.c:232:46: runtime error: left shift of negative value -268435456
../py/runtime.c:463:17: runtime error: left shift of negative value -268435456
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233981504
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233834144
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1234108416
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1234157568
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233510224
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -15
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -123
../py/runtime.c:463:17: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/objint.c:102:16: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
../py/objint.c:102:16: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
../py/objint.c:102:16: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -100
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/objint.c:102:16: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
../py/parse.c:232:46: runtime error: left shift of negative value -101
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -100
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -6
../py/asmthumb.c:296:96: runtime error: left shift of negative value -1234067072
../py/asmthumb.c:296:96: runtime error: left shift of negative value -1233707488
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -45
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/asmthumb.c:296:96: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/asmthumb.c:296:96: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/asmthumb.c:296:96: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of 1179010630 by 1 places cannot be represented in type 'int'
../py/runtime.c:463:17: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -6
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233972943
../py/runtime.c:463:17: runtime error: left shift of negative value -9
make: *** [test] Error 1

Post Reply