os.urandom... isn't very random?

Questions and discussion about The WiPy 1.0 board and CC3200 boards.
Target audience: Users with a WiPy 1.0 or CC3200 board.
Post Reply
gpshead
Posts: 6
Joined: Tue Nov 03, 2015 5:12 am
Contact:

os.urandom... isn't very random?

Post by gpshead » Sun Dec 20, 2015 1:22 am

Plotting the output of os.urandom(512) to a bitmap display is disturbingly non-random:

Image

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: os.urandom... isn't very random?

Post by dhylands » Sun Dec 20, 2015 1:37 am

What did you plot (i.e. what is the X axis and what is the Y axis)?

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: os.urandom... isn't very random?

Post by Roberthh » Sun Dec 20, 2015 8:55 pm

I made two tests. For PyBoard, I collected sufficient data to run the NIST800-22 test. It passed all tests.
For WiPy, I just collected 16k Bytes and just looked at the histogram, the simple plot (value vs. index) , a scatterplot( first half vs. second half) and a spectrogram. None of that looks obviously wrong. For the NIST test, I need about 120 MByte of data. To get these from Wipy requires a little bit more of programming than 5 lines of code. So it'll take a while.

User avatar
danicampora
Posts: 342
Joined: Tue Sep 30, 2014 7:20 am
Contact:

Re: os.urandom... isn't very random?

Post by danicampora » Sun Dec 20, 2015 9:19 pm

Hi,

The pyboard has a true harward random number generator, while the WiPy on the other hand as a software pseudo random one. The WiPy one is good enough in most cases, but I doubt it'll pass the NIST tests...

Cheers,
Daniel

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: os.urandom... isn't very random?

Post by Roberthh » Mon Dec 21, 2015 1:39 pm

A good pseudo-random number generator with a sufficiently long period should pass the NIST test, because this test looks for randomness, not for predictability. And if you are using an algorithm like AES or DEA in feedback mode for the generator, that should work. Looking at the code I see that WiPy uses a 24 bit LFSR. I extracted that into a little C program. generated data and did the NIST test on it. It did not fail every test. Bruce Schneier has published two other examples, which perform better, if that is of any matter.
https://www.schneier.com/cryptography/a ... equen.html

willie
Posts: 14
Joined: Mon Dec 14, 2015 12:05 am

Re: os.urandom... isn't very random?

Post by willie » Thu Dec 31, 2015 5:21 am

It's really not ok for os.urandom to use an LSFR generator. It's supposed to be cryptographically secure. If there is a way to initialize it through user input timings or something, then running a cryptographic PRNG is hopefully ok. But it would be good if there was some NVRAM or something to maintain the PRNG state between powerups.

cswiger
Posts: 13
Joined: Fri Oct 16, 2015 9:51 pm

Re: os.urandom... isn't very random?

Post by cswiger » Tue Feb 09, 2016 8:31 pm

Collected about 4MB of os.urandom(1024) concatenated and ran this test (where ocassional low or high p values will occasionally occur in a good trng, that is, in another collection that one weak will go away and maybe another will appear elsewhere, it happens):

Code: Select all

chuck@pe1950b:~$ dieharder -a -f bigrand
#=============================================================================#
#            dieharder version 3.31.1 Copyright 2003 Robert G. Brown          #
#=============================================================================#
   rng_name    |           filename             |rands/second|
        mt19937|                         bigrand|  1.25e+08  |
#=============================================================================#
        test_name   |ntup| tsamples |psamples|  p-value |Assessment
#=============================================================================#
   diehard_birthdays|   0|       100|     100|0.61695785|  PASSED  
      diehard_operm5|   0|   1000000|     100|0.03918532|  PASSED  
  diehard_rank_32x32|   0|     40000|     100|0.00254075|   WEAK   
    diehard_rank_6x8|   0|    100000|     100|0.32426917|  PASSED  
   diehard_bitstream|   0|   2097152|     100|0.90983775|  PASSED  
        diehard_opso|   0|   2097152|     100|0.31136592|  PASSED  
        diehard_oqso|   0|   2097152|     100|0.88849299|  PASSED  
         diehard_dna|   0|   2097152|     100|0.02632160|  PASSED  
diehard_count_1s_str|   0|    256000|     100|0.53418033|  PASSED  
diehard_count_1s_byt|   0|    256000|     100|0.10833179|  PASSED  
 diehard_parking_lot|   0|     12000|     100|0.40756286|  PASSED  
    diehard_2dsphere|   2|      8000|     100|0.62485804|  PASSED  
    diehard_3dsphere|   3|      4000|     100|0.96434579|  PASSED  
     diehard_squeeze|   0|    100000|     100|0.92749868|  PASSED  
        diehard_sums|   0|       100|     100|0.31733302|  PASSED  
        diehard_runs|   0|    100000|     100|0.76171815|  PASSED  
        diehard_runs|   0|    100000|     100|0.53259081|  PASSED  
       diehard_craps|   0|    200000|     100|0.06443498|  PASSED  
       diehard_craps|   0|    200000|     100|0.95361196|  PASSED  
 marsaglia_tsang_gcd|   0|  10000000|     100|0.07229466|  PASSED  
 marsaglia_tsang_gcd|   0|  10000000|     100|0.80494213|  PASSED  
         sts_monobit|   1|    100000|     100|0.01221109|  PASSED  
            sts_runs|   2|    100000|     100|0.19410883|  PASSED  
          sts_serial|   1|    100000|     100|0.43740452|  PASSED  
          sts_serial|   2|    100000|     100|0.76149357|  PASSED  
          sts_serial|   3|    100000|     100|0.43520694|  PASSED  
          sts_serial|   3|    100000|     100|0.76020044|  PASSED  
          sts_serial|   4|    100000|     100|0.76252435|  PASSED  
          sts_serial|   4|    100000|     100|0.97424261|  PASSED  
          sts_serial|   5|    100000|     100|0.24626500|  PASSED  
          sts_serial|   5|    100000|     100|0.81633438|  PASSED  
          sts_serial|   6|    100000|     100|0.26457489|  PASSED  
          sts_serial|   6|    100000|     100|0.69164995|  PASSED  
          sts_serial|   7|    100000|     100|0.46649650|  PASSED  
          sts_serial|   7|    100000|     100|0.92070611|  PASSED  
          sts_serial|   8|    100000|     100|0.56905248|  PASSED  
          sts_serial|   8|    100000|     100|0.90358265|  PASSED  
          sts_serial|   9|    100000|     100|0.93444552|  PASSED  
          sts_serial|   9|    100000|     100|0.99458113|  PASSED  
          sts_serial|  10|    100000|     100|0.86543565|  PASSED  
          sts_serial|  10|    100000|     100|0.40262869|  PASSED  
          sts_serial|  11|    100000|     100|0.88516655|  PASSED  
          sts_serial|  11|    100000|     100|0.94037598|  PASSED  
          sts_serial|  12|    100000|     100|0.75323099|  PASSED  
          sts_serial|  12|    100000|     100|0.85525976|  PASSED  
          sts_serial|  13|    100000|     100|0.87538565|  PASSED  
          sts_serial|  13|    100000|     100|0.57705181|  PASSED  
          sts_serial|  14|    100000|     100|0.44317210|  PASSED  
          sts_serial|  14|    100000|     100|0.58562576|  PASSED  
          sts_serial|  15|    100000|     100|0.18821489|  PASSED  
          sts_serial|  15|    100000|     100|0.46040268|  PASSED  
          sts_serial|  16|    100000|     100|0.83894356|  PASSED  
          sts_serial|  16|    100000|     100|0.73328515|  PASSED  
         rgb_bitdist|   1|    100000|     100|0.53312740|  PASSED  
         rgb_bitdist|   2|    100000|     100|0.73311971|  PASSED  
         rgb_bitdist|   3|    100000|     100|0.74776632|  PASSED  
         rgb_bitdist|   4|    100000|     100|0.92487323|  PASSED  
         rgb_bitdist|   5|    100000|     100|0.06238251|  PASSED  
         rgb_bitdist|   6|    100000|     100|0.95538826|  PASSED  
         rgb_bitdist|   7|    100000|     100|0.98758602|  PASSED  
         rgb_bitdist|   8|    100000|     100|0.72461133|  PASSED  
         rgb_bitdist|   9|    100000|     100|0.72909151|  PASSED  
         rgb_bitdist|  10|    100000|     100|0.16866374|  PASSED  
         rgb_bitdist|  11|    100000|     100|0.94563096|  PASSED  
         rgb_bitdist|  12|    100000|     100|0.84388674|  PASSED  
rgb_minimum_distance|   2|     10000|    1000|0.05088220|  PASSED  
rgb_minimum_distance|   3|     10000|    1000|0.49882948|  PASSED  
rgb_minimum_distance|   4|     10000|    1000|0.84127071|  PASSED  
rgb_minimum_distance|   5|     10000|    1000|0.41340460|  PASSED  
    rgb_permutations|   2|    100000|     100|0.24236399|  PASSED  
    rgb_permutations|   3|    100000|     100|0.89899169|  PASSED  
    rgb_permutations|   4|    100000|     100|0.62104572|  PASSED  
    rgb_permutations|   5|    100000|     100|0.19781278|  PASSED  
      rgb_lagged_sum|   0|   1000000|     100|0.49335812|  PASSED  
      rgb_lagged_sum|   1|   1000000|     100|0.57442446|  PASSED  
      rgb_lagged_sum|   2|   1000000|     100|0.72623778|  PASSED  
      rgb_lagged_sum|   3|   1000000|     100|0.76799308|  PASSED  
      rgb_lagged_sum|   4|   1000000|     100|0.02997482|  PASSED  
      rgb_lagged_sum|   5|   1000000|     100|0.18870353|  PASSED  
      rgb_lagged_sum|   6|   1000000|     100|0.43936165|  PASSED  
      rgb_lagged_sum|   7|   1000000|     100|0.38212618|  PASSED  
      rgb_lagged_sum|   8|   1000000|     100|0.21711583|  PASSED  
      rgb_lagged_sum|   9|   1000000|     100|0.03174413|  PASSED  
      rgb_lagged_sum|  10|   1000000|     100|0.78621228|  PASSED  
      rgb_lagged_sum|  11|   1000000|     100|0.07425169|  PASSED  
      rgb_lagged_sum|  12|   1000000|     100|0.65158704|  PASSED  
      rgb_lagged_sum|  13|   1000000|     100|0.31547852|  PASSED  
      rgb_lagged_sum|  14|   1000000|     100|0.86615011|  PASSED  
      rgb_lagged_sum|  15|   1000000|     100|0.94580508|  PASSED  
      rgb_lagged_sum|  16|   1000000|     100|0.90628965|  PASSED  
      rgb_lagged_sum|  17|   1000000|     100|0.42661627|  PASSED  
      rgb_lagged_sum|  18|   1000000|     100|0.92157677|  PASSED  
      rgb_lagged_sum|  19|   1000000|     100|0.60133352|  PASSED  
      rgb_lagged_sum|  20|   1000000|     100|0.52296843|  PASSED  
      rgb_lagged_sum|  21|   1000000|     100|0.37980474|  PASSED  
      rgb_lagged_sum|  22|   1000000|     100|0.98505308|  PASSED  
      rgb_lagged_sum|  23|   1000000|     100|0.86384453|  PASSED  
      rgb_lagged_sum|  24|   1000000|     100|0.91244148|  PASSED  
      rgb_lagged_sum|  25|   1000000|     100|0.41447060|  PASSED  
      rgb_lagged_sum|  26|   1000000|     100|0.68715712|  PASSED  
      rgb_lagged_sum|  27|   1000000|     100|0.06450373|  PASSED  
      rgb_lagged_sum|  28|   1000000|     100|0.76268183|  PASSED  
      rgb_lagged_sum|  29|   1000000|     100|0.86828993|  PASSED  
      rgb_lagged_sum|  30|   1000000|     100|0.65213018|  PASSED  
      rgb_lagged_sum|  31|   1000000|     100|0.22795356|  PASSED  
      rgb_lagged_sum|  32|   1000000|     100|0.04675176|  PASSED  
     rgb_kstest_test|   0|     10000|    1000|0.02459689|  PASSED  
     dab_bytedistrib|   0|  51200000|       1|0.51907397|  PASSED  
             dab_dct| 256|     50000|       1|0.98987543|  PASSED  
Preparing to run test 207.  ntuple = 0
        dab_filltree|  32|  15000000|       1|0.52251896|  PASSED  
        dab_filltree|  32|  15000000|       1|0.25957927|  PASSED  
Preparing to run test 208.  ntuple = 0
       dab_filltree2|   0|   5000000|       1|0.29477193|  PASSED  
       dab_filltree2|   1|   5000000|       1|0.68797791|  PASSED  
Preparing to run test 209.  ntuple = 0
        dab_monobit2|  12|  65000000|       1|0.16430290|  PASSED  

Post Reply