Page 1 of 1

problem with hashlib on ARM Linux

Posted: Sun Aug 14, 2016 8:45 pm
by willie
hashlib/test_hashlib.py fails on Scaleway C1 ARM Linux system (comparable to Raspberry Pi 2). Running Debian 8, gcc 4.9.2

Micropython built from current master, commit info:

commit: 21967990519b3d0778770ddb0994216d012101c3
Author: Damien George <damien.p.george@gmail.com>
Date: Sun Aug 14 16:51:54 2016 +1000

I've debugged the python code a little. The error happens in the sha_final function, where in the last block (count= 19 chars) the function tries to zero out the last 45 chars of the block using sha_info['data'][:count] and is getting an empty array as this slice (sha_info['data'] is a list of 64 ints). So it looks like the slice is being made incorrectly.

The test works fine under micropython on x64 linux and under cpython 3.4.2 on the Scaleway ARM. I haven't tried any other systems.

I haven't tried debugging the C code yet. Thought I'd ask here first if anyone knew what might be happening.

I can supply ssh access to the Scaleway server if any of the devs want to try it. Email through my registered forum address is the quickest way to reach me if I haven't checked the forum lately.

Re: problem with hashlib on ARM Linux

Posted: Mon Aug 15, 2016 9:57 pm
by willie
Update: I installed Ubuntu Precise (12.04) on another Scaleway instance, and rebuilt micropython with gcc 4.6.3 from the Ubtuntu repo. The test runs ok on that system. So there is either a compiler bug or some kind of UB in micropython. Has micropython been tested with UB sanitizers?

I also tried building with clang under jessie, but that failed because of some gcc-specific asm syntax in a file with inline asm. I also downloaded and tried to build gcc 6.1, but that hit snags of its own.

Re: problem with hashlib on ARM Linux

Posted: Tue Aug 16, 2016 7:43 am
by stijn
If you have the proper steps to reproduce this it's better if you create an issue at https://github.com/micropython/micropython/issues/new : forum is more for discussion, issue tracker is for issues aka bugs

Re: problem with hashlib on ARM Linux

Posted: Tue Aug 16, 2016 2:29 pm
by pfalcon
Has micropython been tested with UB sanitizers?
If you test it, please share the fixes for the issues you find!

Re: problem with hashlib on ARM Linux

Posted: Tue Aug 16, 2016 4:52 pm
by willie
Deshipu kindly opened https://github.com/micropython/micropython/issues/2323 and Damien responded that this appeared to be the same issue as #2264, fixed at almost the same time. I pulled new master and the test passes now. Thanks!

Re: problem with hashlib on ARM Linux

Posted: Tue Aug 16, 2016 6:21 pm
by willie
I ran "make test" with -fsanitize=undefined and got a bunch of error reports about invalid left shifts. Output is attached as "ubsan.gif". It's actually a text file but I had to use the .gif extension to get the forum software to accept the upload.

Edit: attachment didn't work. Ok, pasting the output here (200 lines):

Code: Select all

../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/mpz.c:802:16: runtime error: negation of -9223372036854775808 cannot be represented in type 'long long int'; cast to an unsigned type to negate this value to itself
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -5
../py/mpz.c:1558:20: runtime error: left shift of 262144 by 16 places cannot be represented in type 'int'
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/runtime.c:463:17: runtime error: left shift of negative value -8
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/runtime.c:463:17: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -3
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -100
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/parse.c:232:46: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -3
../py/parse.c:232:46: runtime error: left shift of negative value -123
../py/runtime.c:463:17: runtime error: left shift of negative value -18
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -283894311
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/objint_mpz.c:327:9: runtime error: left shift of 1073741824 by 1 places cannot be represented in type 'int'
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/runtime.c:463:17: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1073741823
../py/parsenum.c:106:14: runtime error: left shift of 1073741824 by 1 places cannot be represented in type 'int'
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741823
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/runtime.c:463:17: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -100
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -3
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -3
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -128
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -100
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -123
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1234
../py/parse.c:232:46: runtime error: left shift of negative value -10
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/parse.c:232:46: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -128
../py/binary.c:172:13: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -128
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -4
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/../extmod/crypto-algorithms/sha256.c:49:19: runtime error: left shift of 128 by 24 places cannot be represented in type 'int'
modtime.c:90:30: runtime error: signed integer overflow: 1471371216 * 1000 cannot be represented in type 'long int'
modtime.c:82:30: runtime error: signed integer overflow: 1471371216 * 1000000 cannot be represented in type 'long int'
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1234136848
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233944736
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233592368
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1234439936
../py/parse.c:232:46: runtime error: left shift of negative value -268435456
../py/runtime.c:463:17: runtime error: left shift of negative value -268435456
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233985584
../py/parse.c:232:46: runtime error: left shift of negative value -268435456
../py/runtime.c:463:17: runtime error: left shift of negative value -268435456
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233981504
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233834144
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1234108416
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1234157568
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233510224
../py/parse.c:232:46: runtime error: left shift of negative value -1073741824
../py/runtime.c:463:17: runtime error: left shift of negative value -1073741824
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -15
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -123
../py/runtime.c:463:17: runtime error: left shift of negative value -5
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/objint.c:102:16: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
../py/objint.c:102:16: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
../py/objint.c:102:16: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -100
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/objint.c:102:16: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
../py/parse.c:232:46: runtime error: left shift of negative value -101
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -100
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -6
../py/asmthumb.c:296:96: runtime error: left shift of negative value -1234067072
../py/asmthumb.c:296:96: runtime error: left shift of negative value -1233707488
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/vm.c:226:36: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -45
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/asmthumb.c:296:96: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -2
../py/asmthumb.c:296:96: runtime error: left shift of negative value -2
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/asmthumb.c:296:96: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of 1179010630 by 1 places cannot be represented in type 'int'
../py/runtime.c:463:17: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/runtime.c:463:17: runtime error: left shift of negative value -6
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/parse.c:232:46: runtime error: left shift of negative value -1
../py/objint_mpz.c:327:9: runtime error: left shift of negative value -1233972943
../py/runtime.c:463:17: runtime error: left shift of negative value -9
make: *** [test] Error 1