RAM Hex viewer

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

RAM Hex viewer

Post by jedie » Fri Dec 27, 2019 2:40 pm

I hacked a Web RAM Hex viewer with simple search function... Here was searched for "micropython":
Screenshot_2019-12-27 MicroPython RAM hex viewer Webserver.png
Screenshot_2019-12-27 MicroPython RAM hex viewer Webserver.png (225.02 KiB) Viewed 808 times
The finding place is in my opinion in the firmware section. Because i compare it with a hex editor:
screenshot.png
screenshot.png (77.63 KiB) Viewed 808 times
Source are here: https://github.com/jedie/micropython-so ... _viewer.py

User avatar
jimmo
Posts: 1678
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: RAM Hex viewer

Post by jimmo » Fri Jan 03, 2020 6:22 am

That's neat!

What you're looking at FWIW is the QSTR table, which is indeed part of the firmware image -- it's a compact list of the string bytes, hash, and length (which is why the entries are separated by four bytes).

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: RAM Hex viewer

Post by jedie » Sun Jan 05, 2020 2:09 pm

I "access" the RAM via esp.flash_read() into a bytearray()

That works, but it duplicated the chunk, so it's costs RAM. Is there a way read the RAM without duplicate it?

User avatar
jimmo
Posts: 1678
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: RAM Hex viewer

Post by jimmo » Sun Jan 05, 2020 11:52 pm



jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: RAM Hex viewer

Post by jedie » Mon Jan 06, 2020 8:14 am

Short question: Can i access uctypes.bytearray_at() the same memory region as with esp.flash_read() ?

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: RAM Hex viewer

Post by jedie » Mon Jan 06, 2020 8:42 am

jedie wrote:
Mon Jan 06, 2020 8:14 am
Short question: Can i access uctypes.bytearray_at() the same memory region as with esp.flash_read() ?
Seems i can't replace esp.flash_read() with uctypes.bytearray_at() because it crashed, e.g.:

Code: Select all

import esp
import uctypes

offset = const(0)
CHUNK_SIZE = const(16)

print('esp.flash_read():', end='')
BUFFER = bytearray(CHUNK_SIZE)
esp.flash_read(offset, BUFFER)
print(repr(BUFFER))

print('uctypes.bytearray_at():', end='')
memory = uctypes.bytearray_at(offset, CHUNK_SIZE)
print(repr(memory))
Output is:

Code: Select all

esp.flash_read():bytearray(b'\xe9\x03\x00@\xa0\x04\x10@\x00\x00\x10@Pz\x00\x00')
uctypes.bytearray_at():Fatal exception 28(LoadProhibitedCause):
�pc1=0x40252662, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000
 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 31312, room 16 
...

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: RAM Hex viewer

Post by jedie » Mon Jan 06, 2020 9:21 am

My "plain text" hex viewer looks now like this:

Code: Select all

import gc

import esp
import utime
from micropython import const

LINE_COUNT = const(6)
CHUNK_SIZE = const(32)
BUFFER = bytearray(CHUNK_SIZE)


def hex_dump(offset=0):
    for line_num in range(LINE_COUNT):
        current_offset = offset + (line_num * CHUNK_SIZE)
        esp.flash_read(current_offset, BUFFER)

        print('%08x - %08x' % (current_offset, current_offset + CHUNK_SIZE - 1), end=' ')
        print(' '.join('%02x' % char for char in BUFFER), end=' ')
        print(
            ''.join(
                chr(char)
                if 32 <= char <= 126 and char not in (60, 62) else '.'
                for char in BUFFER
            )
        )

    gc.collect()


def search(offset, text):
    print('search for %r start with offset 0x%x' % (text, offset))

    offset_step = CHUNK_SIZE - len(text)

    if offset_step <= 0:
        raise AssertionError('Search text too large: increase CHUNK_SIZE!')

    def print_throughput():
        duration = utime.time() - start_time
        if duration == 0:
            throughput = -1
        else:
            throughput = (offset - start_offset) / duration / 1024
        print(
            '(Search duration: %i sec. - %i KBytes/sec' % (duration, throughput)
        )

    flash_size = esp.flash_size()
    start_offset = offset
    end_researched = False
    start_time = utime.time()
    next_update = start_time + 1
    while True:
        if offset + CHUNK_SIZE > flash_size:
            # Search to esp.flash_size(), but don't go beyond.
            offset = flash_size - CHUNK_SIZE
            end_researched = True

        try:
            esp.flash_read(offset, BUFFER)
        except OSError as e:
            print('Read flash error: %s at 0x%x - 0x%x' % (e, offset, offset + CHUNK_SIZE))
            return -1

        if text in BUFFER:
            print('Found in 0x%x !' % offset, end=' ')
            print_throughput()
            return offset

        if utime.time() >= next_update:
            print('Search: 0x%x ...' % offset, end=' ')
            print_throughput()
            next_update = utime.time() + 1

        if end_researched:
            print('Memory end researched, searched up to 0x%x' % (offset + CHUNK_SIZE))
            print_throughput()
            return -1

        offset += offset_step


if __name__ == '__main__':
    print('esp.flash_user_start(): 0x%x' % (esp.flash_user_start()))
    print('esp.flash_size()......: 0x%x' % (esp.flash_size()))

    offset = 0x0
    while True:
        offset = search(offset=offset, text='micropython')
        if offset == -1:
            # not found or end researched
            break
        hex_dump(offset=offset)
        offset += (CHUNK_SIZE * LINE_COUNT) - 1
Output:

Code: Select all

esp.flash_user_start(): 0x98000
esp.flash_size()......: 0x400000
search for 'micropython' start with offset 0x0
Search: 0xae90 ... (Search duration: 1 sec. - 43 KBytes/sec
Search: 0x2af96 ... (Search duration: 2 sec. - 85 KBytes/sec
Search: 0x4b0b1 ... (Search duration: 3 sec. - 100 KBytes/sec
Search: 0x6b1e1 ... (Search duration: 4 sec. - 107 KBytes/sec
Found in 0x7bfed ! (Search duration: 4 sec. - 123 KBytes/sec
0007bfed - 0007c00c 0b 7c 0b 6d 69 63 72 6f 70 79 74 68 6f 6e 00 42 88 04 6e 65 78 74 00 90 8d 06 6f 62 6a 65 63 74 .|.micropython.B..next....object
0007c00d - 0007c02c 00 d1 3a 04 6f 70 65 6e 00 1c 5e 03 6f 72 64 00 2a 73 03 70 6f 70 00 bf 2c 07 70 6f 70 69 74 65 ..:.open..^.ord.*s.pop..,.popite
0007c02d - 0007c04c 6d 00 2d 73 03 70 6f 77 00 54 c6 05 70 72 69 6e 74 00 1a 5e 05 72 61 6e 67 65 00 b7 f9 04 72 65 m.-s.pow.T..print..^.range....re
0007c04d - 0007c06c 61 64 00 4b bf 08 72 65 61 64 69 6e 74 6f 00 f9 19 08 72 65 61 64 6c 69 6e 65 00 63 8a 06 72 65 ad.K..readinto....readline.c..re
0007c06d - 0007c08c 6d 6f 76 65 00 49 25 07 72 65 70 6c 61 63 65 00 d0 f7 04 72 65 70 72 00 25 2a 07 72 65 76 65 72 move.I%.replace....repr.%*.rever
0007c08d - 0007c0ac 73 65 00 d2 9c 05 72 66 69 6e 64 00 e9 2b 06 72 69 6e 64 65 78 00 e7 25 05 72 6f 75 6e 64 00 3b se....rfind..+.rindex..%.round.;
search for 'micropython' start with offset 0x7c0ac
Found in 0x7f39d ! (Search duration: 0 sec. - -1 KBytes/sec
0007f39d - 0007f3bc 64 20 6d 69 63 72 6f 70 79 74 68 6f 6e 20 64 65 63 6f 72 61 74 6f 72 00 73 75 70 65 72 28 29 20 d micropython decorator.super() 
0007f3bd - 0007f3dc 63 61 6e 27 74 20 66 69 6e 64 20 73 65 6c 66 00 27 61 77 61 69 74 27 20 6f 75 74 73 69 64 65 20 can't find self.'await' outside 
0007f3dd - 0007f3fc 66 75 6e 63 74 69 6f 6e 00 27 72 65 74 75 72 6e 27 20 6f 75 74 73 69 64 65 20 66 75 6e 63 74 69 function.'return' outside functi
0007f3fd - 0007f41c 6f 6e 00 63 61 6e 27 74 20 64 65 63 6c 61 72 65 20 6e 6f 6e 6c 6f 63 61 6c 20 69 6e 20 6f 75 74 on.can't declare nonlocal in out
0007f41d - 0007f43c 65 72 20 63 6f 64 65 00 69 64 65 6e 74 69 66 69 65 72 20 72 65 64 65 66 69 6e 65 64 20 61 73 20 er code.identifier redefined as 
0007f43d - 0007f45c 67 6c 6f 62 61 6c 00 6e 6f 20 62 69 6e 64 69 6e 67 20 66 6f 72 20 6e 6f 6e 6c 6f 63 61 6c 20 66 global.no binding for nonlocal f
search for 'micropython' start with offset 0x7f45c
Found in 0x85c8a ! (Search duration: 0 sec. - -1 KBytes/sec
00085c8a - 00085ca9 00 66 72 6f 6d 20 6d 69 63 72 6f 70 79 74 68 6f 6e 20 69 6d 70 6f 72 74 20 63 6f 6e 73 74 0a 00 .from micropython import const..
00085caa - 00085cc9 53 74 6f 72 65 20 69 6e 20 25 72 3a 20 25 72 00 73 61 76 65 20 70 79 20 63 6f 6e 66 69 67 00 53 Store in %r: %r.save py config.S
00085cca - 00085ce9 6b 69 70 20 73 61 76 65 20 70 79 20 63 6f 6e 66 69 67 3a 20 41 6c 72 65 61 64 79 20 65 78 69 73 kip save py config: Already exis
00085cea - 00085d09 74 73 20 77 69 74 68 20 74 68 69 73 20 76 61 6c 75 65 00 53 79 6e 74 61 78 20 65 72 72 6f 72 20 ts with this value.Syntax error 
00085d0a - 00085d29 69 6e 3a 00 49 6d 70 6f 72 74 20 65 72 72 6f 72 3a 00 72 65 73 74 6f 72 65 20 70 79 20 63 6f 6e in:.Import error:.restore py con
00085d2a - 00085d49 66 69 67 00 53 6b 69 70 20 73 61 76 65 20 6a 73 6f 6e 20 63 6f 6e 66 69 67 3a 20 41 6c 72 65 61 fig.Skip save json config: Alrea
search for 'micropython' start with offset 0x85d49
Search: 0x89a51 ... (Search duration: 1 sec. - 15 KBytes/sec
Found in 0x90f21 ! (Search duration: 1 sec. - 44 KBytes/sec
00090f21 - 00090f40 73 69 74 20 68 74 74 70 3a 2f 2f 64 6f 63 73 2e 6d 69 63 72 6f 70 79 74 68 6f 6e 2e 6f 72 67 2f sit http://docs.micropython.org/
00090f41 - 00090f60 65 6e 2f 6c 61 74 65 73 74 2f 65 73 70 38 32 36 36 2f 20 2e 0a 46 6f 72 20 64 69 61 67 6e 6f 73 en/latest/esp8266/ ..For diagnos
00090f61 - 00090f80 74 69 63 20 69 6e 66 6f 72 6d 61 74 69 6f 6e 20 74 6f 20 69 6e 63 6c 75 64 65 20 69 6e 20 62 75 tic information to include in bu
00090f81 - 00090fa0 67 20 72 65 70 6f 72 74 73 20 65 78 65 63 75 74 65 20 27 69 6d 70 6f 72 74 20 70 6f 72 74 5f 64 g reports execute 'import port_d
00090fa1 - 00090fc0 69 61 67 27 2e 0a 0a 42 61 73 69 63 20 57 69 46 69 20 63 6f 6e 66 69 67 75 72 61 74 69 6f 6e 3a iag'...Basic WiFi configuration:
00090fc1 - 00090fe0 0a 0a 69 6d 70 6f 72 74 20 6e 65 74 77 6f 72 6b 0a 73 74 61 5f 69 66 20 3d 20 6e 65 74 77 6f 72 ..import network.sta_if = networ
search for 'micropython' start with offset 0x90fe0
Search: 0x9e957 ... (Search duration: 1 sec. - 54 KBytes/sec
Search: 0xbea1e ... (Search duration: 2 sec. - 91 KBytes/sec
Search: 0xdeb63 ... (Search duration: 3 sec. - 103 KBytes/sec
Found in 0xe1a25 ! (Search duration: 3 sec. - 107 KBytes/sec
000e1a25 - 000e1a44 69 6e 00 b9 43 03 6d 61 70 00 0b 7c 0b 6d 69 63 72 6f 70 79 74 68 6f 6e 00 42 88 04 6e 00 00 00 in..C.map..|.micropython.B..n...
000e1a45 - 000e1a64 00 00 00 00 00 82 d6 88 91 07 c7 ee 29 f2 9c d3 94 35 c4 91 ce e6 b4 3b c1 02 e3 38 19 3a 34 30 ............)....5.....;...8.:40
000e1a65 - 000e1a84 3b 07 74 2e 33 b7 d1 7e 72 4d b3 9b a1 78 e2 bc ac 0d 06 40 08 d7 7f 0b 32 4c a3 d9 99 15 b1 6c ;.t.3..~rM...x.....@....2L.....l
000e1a85 - 000e1aa4 d7 30 84 e7 b3 54 01 c1 07 00 00 00 00 00 01 00 c0 07 00 65 78 74 00 90 8d 06 6f 62 6a 65 63 74 .0...T.............ext....object
000e1aa5 - 000e1ac4 00 d1 3a 04 6f 70 65 6e 00 1c 5e 03 6f 72 64 00 2a 73 03 70 6f 70 00 bf 2c 07 70 6f 70 69 74 65 ..:.open..^.ord.*s.pop..,.popite
000e1ac5 - 000e1ae4 6d 00 2d 73 03 70 6f 77 00 54 c6 05 70 72 69 6e 74 00 1a 5e 05 72 61 6e 67 65 00 b7 f9 04 72 65 m.-s.pow.T..print..^.range....re
search for 'micropython' start with offset 0xe1ae4
Found in 0xe5f4e ! (Search duration: 0 sec. - -1 KBytes/sec
000e5f4e - 000e5f6d 6f 6e 00 69 6e 76 61 6c 69 64 20 6d 69 63 72 6f 70 79 74 68 6f 6e 20 64 65 63 6f 72 61 74 6f 72 on.invalid micropython decorator
000e5f6e - 000e5f8d 00 73 75 70 65 72 28 29 20 63 61 6e 27 74 20 66 69 6e 64 20 73 65 6c 66 00 27 61 77 61 69 74 27 .super() can't find self.'await'
000e5f8e - 000e5fad 20 6f 75 74 73 69 64 65 20 66 75 6e 63 74 69 6f 6e 00 27 72 65 74 75 72 6e 27 20 6f 75 74 73 69  outside function.'return' outsi
000e5fae - 000e5fcd 64 65 20 66 75 6e 63 74 69 6f 6e 00 00 00 00 00 00 00 00 00 b8 b5 f0 52 2a 60 fb df 11 bb 25 c0 de function............R*`....%.
000e5fce - 000e5fed 53 8e fc 32 97 30 5c 59 df 8f 56 32 f3 99 e1 e7 9c a0 41 34 5f 52 2f 87 1c 9b 52 aa fb 4a 5b 91 S..2.0\Y..V2......A4_R/...R..J[.
000e5fee - 000e600d 79 b6 ba b1 28 5e 05 c1 d8 14 51 4c 36 89 ed 8d d1 19 ed 94 54 01 f5 07 00 00 00 00 00 01 00 f4 y...(^....QL6.......T...........
search for 'micropython' start with offset 0xe600d
Search: 0xe9a36 ... (Search duration: 1 sec. - 14 KBytes/sec
Found in 0xeea07 ! (Search duration: 1 sec. - 34 KBytes/sec
000eea07 - 000eea26 20 6d 69 63 72 6f 70 79 74 68 6f 6e 20 69 6d 70 6f 72 74 20 63 6f 6e 73 74 0a 00 53 74 6f 72 65  micropython import const..Store
000eea27 - 000eea46 20 69 6e 20 25 72 3a 20 25 72 00 73 61 76 65 20 70 79 20 63 6f 6e 66 69 67 00 53 6b 69 70 20 73  in %r: %r.save py config.Skip s
000eea47 - 000eea66 61 76 65 20 70 79 20 63 6f 6e 66 69 67 3a 20 41 6c 72 65 61 64 79 20 65 78 69 73 74 73 20 77 69 ave py config: Already exists wi
000eea67 - 000eea86 74 68 20 74 68 69 73 20 76 61 6c 75 65 00 53 79 6e 74 61 78 20 65 72 72 6f 72 20 69 6e 3a 00 49 th this value.Syntax error in:.I
000eea87 - 000eeaa6 6d 70 6f 72 74 20 65 72 72 6f 72 3a 00 72 65 73 74 6f 72 65 20 70 79 20 63 6f 6e 66 69 67 00 53 mport error:.restore py config.S
000eeaa7 - 000eeac6 6b 69 70 00 00 00 00 00 00 00 00 30 01 47 5c 18 57 2d 52 0c 61 0e f3 5b 30 57 c7 db d5 a1 47 e9 kip........0.G\.W-R.a..[0W....G.
search for 'micropython' start with offset 0xeeac6
Found in 0xfdab7 ! (Search duration: 0 sec. - -1 KBytes/sec
000fdab7 - 000fdad6 3a 2f 2f 64 6f 63 73 2e 6d 69 63 72 6f 70 79 74 68 6f 6e 2e 6f 72 67 2f 65 6e 2f 6c 61 74 65 73 ://docs.micropython.org/en/lates
000fdad7 - 000fdaf6 74 2f 65 73 70 38 32 36 36 2f 20 2e 0a 46 6f 72 20 64 69 61 67 6e 6f 73 74 69 63 20 69 6e 66 6f t/esp8266/ ..For diagnostic info
000fdaf7 - 000fdb16 72 6d 61 74 69 6f 6e 20 74 6f 20 69 6e 63 6c 75 64 65 20 69 6e 20 62 75 67 20 72 65 70 6f 72 74 rmation to include in bug report
000fdb17 - 000fdb36 73 20 65 78 65 63 75 74 65 20 27 00 00 00 00 00 00 00 00 2d 97 e6 f4 11 e5 9a 89 60 a9 04 6f a7 s execute '........-.......`..o.
000fdb37 - 000fdb56 e4 4e b9 12 84 46 8b c8 e0 b5 17 94 e2 00 9b 6c 36 9d 31 a5 1d 41 a5 de ab 45 58 92 bd b3 9f 90 .N...F.........l6.1..A...EX.....
000fdb57 - 000fdb76 98 7d ca 75 91 db d2 bc f9 66 bb 19 f9 44 53 dc f8 7b b9 54 01 11 09 00 00 00 00 00 01 00 10 09 .}.u.....f...DS..{.T............
search for 'micropython' start with offset 0xfdb76
Search: 0x114523 ... (Search duration: 1 sec. - 90 KBytes/sec
Search: 0x1343b3 ... (Search duration: 2 sec. - 109 KBytes/sec
Search: 0x1541da ... (Search duration: 3 sec. - 115 KBytes/sec
Search: 0x174055 ... (Search duration: 4 sec. - 118 KBytes/sec
Search: 0x193ebb ... (Search duration: 5 sec. - 120 KBytes/sec
Search: 0x1b3d75 ... (Search duration: 6 sec. - 121 KBytes/sec
Search: 0x1d3b9c ... (Search duration: 7 sec. - 122 KBytes/sec
Search: 0x1f3a2c ... (Search duration: 8 sec. - 122 KBytes/sec
Search: 0x2138d1 ... (Search duration: 9 sec. - 123 KBytes/sec
Search: 0x233737 ... (Search duration: 10 sec. - 123 KBytes/sec
Search: 0x2535c7 ... (Search duration: 11 sec. - 124 KBytes/sec
Search: 0x273481 ... (Search duration: 12 sec. - 124 KBytes/sec
Search: 0x2932e7 ... (Search duration: 13 sec. - 124 KBytes/sec
Search: 0x2b31a1 ... (Search duration: 14 sec. - 124 KBytes/sec
Search: 0x2d3046 ... (Search duration: 15 sec. - 125 KBytes/sec
Search: 0x2f2eac ... (Search duration: 16 sec. - 125 KBytes/sec
Search: 0x312da5 ... (Search duration: 17 sec. - 125 KBytes/sec
Search: 0x332bcc ... (Search duration: 18 sec. - 125 KBytes/sec
Search: 0x352a32 ... (Search duration: 19 sec. - 125 KBytes/sec
Search: 0x3728c2 ... (Search duration: 20 sec. - 125 KBytes/sec
Search: 0x392752 ... (Search duration: 21 sec. - 125 KBytes/sec
Search: 0x3b260c ... (Search duration: 22 sec. - 125 KBytes/sec
Search: 0x3d249c ... (Search duration: 23 sec. - 126 KBytes/sec
Search: 0x3f2380 ... (Search duration: 24 sec. - 126 KBytes/sec
Memory end researched, searched up to 0x400000
(Search duration: 24 sec. - 128 KBytes/sec
Think it can be made faster by increase the CHUNK_SIZE, but that will make the hex_dump() function more complicated :lol:
Search the complete 4MB RAM in 24 sec. is okay... :mrgreen:

User avatar
jimmo
Posts: 1678
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: RAM Hex viewer

Post by jimmo » Mon Jan 06, 2020 11:33 am

jedie wrote:
Mon Jan 06, 2020 8:42 am
Seems i can't replace esp.flash_read() with uctypes.bytearray_at() because it crashed, e.g.:
Because RAM and flash are not the same thing.

Flash is mapped into some region of RAM (starting at 0x3F400000 I think).

But you probably can't just go accessing any random address of RAM (for example, zero).

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: RAM Hex viewer

Post by jedie » Mon Jan 06, 2020 11:45 am

Hm. It's not so easy as on my old home computer that maps everything in RAM: https://github.com/jedie/DragonPy :ugeek:

But with esp.flash_read() i can read 0x0 - 0x400000 it's the complete 4MB RAM that this ESP8266 has and i can access this information via esp.flash_size()

Is there a way to get information about the areas i can access via uctypes.bytearray_at() ?

But maybe i should simply stay with the curent esp.flash_read() solution, because of:
Screenshot_2020-01-06 uctypes – access binary data in a structured way — MicroPython 1 12 documentation.png
Screenshot_2020-01-06 uctypes – access binary data in a structured way — MicroPython 1 12 documentation.png (13.84 KiB) Viewed 651 times

Post Reply