How do I make a port of MicroPython for Casio calculators?

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Zezombye
Posts: 33
Joined: Mon Jul 30, 2018 8:29 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by Zezombye » Mon Sep 03, 2018 10:59 am

There you go: https://youtu.be/X9MSxI2BVK0

If you want to try it yourself, you can always use the emulator:
- Download the emulator at https://edu.casio.com/freetrial/fr/free ... LANGUAGE=1
- Download the program (.g1a) at https://www.planet-casio.com/Fr/program ... howid=3603
- Install the emulator, open the shortcut on desktop
- Go to memory menu, press F3 (SD)
- Select the g1a you downloaded, and when prompted press 2 for storage memory
- Go at the bottom of the main menu and select the python icon.

Zezombye
Posts: 33
Joined: Mon Jul 30, 2018 8:29 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by Zezombye » Mon Sep 03, 2018 5:41 pm

I managed to make the import statement work how I want to ("import test" imports from the file test.py), however I found that it doesn't close the file if an error happens (syntax error, or out of memory error for example).

I just overrode the close, read and open functions in reader.c with the Casio ones (which are very similar in function, if not the exact same).

How do I make sure MPy closes the file? I can't put the file handle in global and close it myself, because multiple file handles might be opened.

jickster
Posts: 546
Joined: Thu Sep 07, 2017 8:57 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by jickster » Mon Sep 03, 2018 7:53 pm

Zezombye wrote:
Mon Sep 03, 2018 5:41 pm
I managed to make the import statement work how I want to ("import test" imports from the file test.py), however I found that it doesn't close the file if an error happens (syntax error, or out of memory error for example).

I just overrode the close, read and open functions in reader.c with the Casio ones (which are very similar in function, if not the exact same).

How do I make sure MPy closes the file? I can't put the file handle in global and close it myself, because multiple file handles might be opened.
When you call the sequence of functions to compile and execute

Code: Select all

                mp_lexer_t * lex = mp_lexer_new_from_str_len(0, ptr_code, (size_t)code_length, (size_t)0);
		mp_parse_tree_t parse_tree = mp_parse(lex, parse_type);
		mp_obj_t module_func = mp_compile(&parse_tree, lex->source_name, MP_EMIT_OPT_NONE, repl);

		mp_call_function_0(module_func);
mp_parse calls mp_lexer_free()

Code: Select all

    // we also free the lexer on behalf of the caller
    mp_lexer_free(lex);
Which calls a function pointer to close the file

Code: Select all

void mp_lexer_free(mp_lexer_t *lex) {
    if (lex) {
        lex->reader.close(lex->reader.data);
        vstr_clear(&lex->vstr);
        m_del(uint16_t, lex->indent_level, lex->alloc_indent_level);
        m_del_obj(mp_lexer_t, lex);
    }
}
Which should've been set in

Code: Select all

void mp_reader_new_file_from_fd(mp_reader_t *reader, int fd, bool close_fd) {
    mp_reader_posix_t *rp = m_new_obj(mp_reader_posix_t);
    rp->close_fd = close_fd;
    rp->fd = fd;
    int n = read(rp->fd, rp->buf, sizeof(rp->buf));
    if (n == -1) {
        if (close_fd) {
            close(fd);
        }
        mp_raise_OSError(errno);
    }
    rp->len = n;
    rp->pos = 0;
    reader->data = rp;
    reader->readbyte = mp_reader_posix_readbyte;
    reader->close = mp_reader_posix_close;
}
Called in

Code: Select all

mp_lexer_t *mp_lexer_new_from_fd(qstr filename, int fd, bool close_fd) {
    mp_reader_t reader;
    mp_reader_new_file_from_fd(&reader, fd, close_fd);
    return mp_lexer_new(filename, reader);
}
Conceptually this should make sense to you that the file is closed at the end of mp_parse().
Once the text of a .py has been digested, by the parsing stage, into a tree of data-structures for the compiler to consume, there's no need for the file to be open.

Obviously replace the relevant function pointers with your own `Bfile*()` functions

jickster
Posts: 546
Joined: Thu Sep 07, 2017 8:57 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by jickster » Mon Sep 03, 2018 8:05 pm

Zezombye wrote:
Mon Sep 03, 2018 5:41 pm
I managed to make the import statement work how I want to ("import test" imports from the file test.py), however I found that it doesn't close the file if an error happens (syntax error, or out of memory error for example).

I just overrode the close, read and open functions in reader.c with the Casio ones (which are very similar in function, if not the exact same).

How do I make sure MPy closes the file? I can't put the file handle in global and close it myself, because multiple file handles might be opened.
Ok I realize I didn't answer your question with my last post.

jickster
Posts: 546
Joined: Thu Sep 07, 2017 8:57 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by jickster » Mon Sep 03, 2018 8:29 pm

Zezombye wrote:
Mon Sep 03, 2018 5:41 pm
I managed to make the import statement work how I want to ("import test" imports from the file test.py), however I found that it doesn't close the file if an error happens (syntax error, or out of memory error for example).

I just overrode the close, read and open functions in reader.c with the Casio ones (which are very similar in function, if not the exact same).

How do I make sure MPy closes the file? I can't put the file handle in global and close it myself, because multiple file handles might be opened.
I think you've found a bug.

When a syntax error happens, following lines from mp_parse() execute:

Code: Select all

   syntax_error:;
        mp_obj_t exc;
        if (lex->tok_kind == MP_TOKEN_INDENT) {
            exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
                "unexpected indent");
        } else if (lex->tok_kind == MP_TOKEN_DEDENT_MISMATCH) {
            exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
                "unindent does not match any outer indentation level");
        } else {
            exc = mp_obj_new_exception_msg(&mp_type_SyntaxError,
                "invalid syntax");
        }
        // add traceback to give info about file name and location
        // we don't have a 'block' name, so just pass the NULL qstr to indicate this
        mp_obj_exception_add_traceback(exc, lex->source_name, lex->tok_line, MP_QSTR_NULL);
        nlr_raise(exc);
The information stored in the exception does NOT contain the "fd" stored in reader->data necessary to close the file.

Code: Select all

void mp_reader_new_file_from_fd(mp_reader_t *reader, int fd, bool close_fd) {
    mp_reader_posix_t *rp = m_new_obj(mp_reader_posix_t);
    rp->close_fd = close_fd;
    rp->fd = fd;
    int n = read(rp->fd, rp->buf, sizeof(rp->buf));
    if (n == -1) {
        if (close_fd) {
            close(fd);
        }
        mp_raise_OSError(errno);
    }
    rp->len = n;
    rp->pos = 0;
    reader->data = rp;
    reader->readbyte = mp_reader_posix_readbyte;
    reader->close = mp_reader_posix_close;
}
Issue is https://github.com/micropython/micropython/issues/4095

Zezombye
Posts: 33
Joined: Mon Jul 30, 2018 8:29 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by Zezombye » Fri Sep 07, 2018 5:24 pm

I am perplexed by this bit of code:

Code: Select all

typedef struct _mp_reader_posix_t {
    bool close_fd;
    int fd;
    size_t len;
    size_t pos;
    byte buf[25];
} mp_reader_posix_t;

STATIC mp_uint_t mp_reader_posix_readbyte(void *data) {
    mp_reader_posix_t *reader = (mp_reader_posix_t*)data;
    if (reader->pos >= reader->len) {
        if (reader->len == 0) {
            return MP_READER_EOF;
        } else {
            //int n = read(reader->fd, reader->buf, sizeof(reader->buf));
			int n = Bfile_ReadFile(reader->fd, reader->buf, sizeof(reader->buf), reader->pos);
            if (n <= 0) {
                reader->len = 0;
                return MP_READER_EOF;
            }
            reader->len = n;
            reader->pos = 0;
        }
    }
    return reader->buf[reader->pos++];
}
Basically the buffer causes errors for files bigger than it (it was originally 20, but I increased it to 25 to be sure it was the buffer).

For a buffer of 25:
- Importing a file of 0-25 bytes is no problem.
- Importing a file of 26-51 bytes throws a "syntax error"
- Importing a file of 52 bytes or more throws a memory error (allocation failed).

Why is this buffer so low, and not dynamically allocated?

Also, the function Bfile_ReadFile takes the same arguments as the read() function, but with the position at the end. What I don't understand is that the reader->pos is reset to 0 at each byte read, so it should always read the first byte of the file, but it doesn't. How does it work?

jickster
Posts: 546
Joined: Thu Sep 07, 2017 8:57 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by jickster » Fri Sep 07, 2018 7:23 pm

Zezombye wrote:
Fri Sep 07, 2018 5:24 pm
I am perplexed by this bit of code:

Code: Select all

typedef struct _mp_reader_posix_t {
    bool close_fd;
    int fd;
    size_t len;
    size_t pos;
    byte buf[25];
} mp_reader_posix_t;

STATIC mp_uint_t mp_reader_posix_readbyte(void *data) {
    mp_reader_posix_t *reader = (mp_reader_posix_t*)data;
    if (reader->pos >= reader->len) {
        if (reader->len == 0) {
            return MP_READER_EOF;
        } else {
            //int n = read(reader->fd, reader->buf, sizeof(reader->buf));
			int n = Bfile_ReadFile(reader->fd, reader->buf, sizeof(reader->buf), reader->pos);
            if (n <= 0) {
                reader->len = 0;
                return MP_READER_EOF;
            }
            reader->len = n;
            reader->pos = 0;
        }
    }
    return reader->buf[reader->pos++];
}
Basically the buffer causes errors for files bigger than it (it was originally 20, but I increased it to 25 to be sure it was the buffer).

For a buffer of 25:
- Importing a file of 0-25 bytes is no problem.
- Importing a file of 26-51 bytes throws a "syntax error"
- Importing a file of 52 bytes or more throws a memory error (allocation failed).

Why is this buffer so low, and not dynamically allocated?

Also, the function Bfile_ReadFile takes the same arguments as the read() function, but with the position at the end. What I don't understand is that the reader->pos is reset to 0 at each byte read, so it should always read the first byte of the file, but it doesn't. How does it work?
The reason why this is not straightforward is because an optimization was made: instead of calling "read()" everytime you want to read ONE byte, this function reads in a chunk of 25 bytes at a time.

The naming of the struct members could be clearer.

reader->len
is the number of bytes obtained from "read()". This number will be sizeof(reader->buf) until you reach near the end of the file when you have less than sizeof(reader->buf) bytes left to read.
Ideally it would be named "reader->num_chars_in_buf"

reader->pos
is the position inside the reader->buf for the next character.
Ideally it would be named "reader->pos_next_char_in_buf"

For "Bfile_ReadFile" to be a drop-in replacement for "read", it has to behave exactly like "read()".
Does it? I don't have the description of it.
Does it return the actual number of bytes that were read?

https://gist.github.com/holubv/89df8019 ... 7659727d0e
/**
* Reads data from a file, starting at the position indicated by the file pointer. After the
* read operation has been completed, the file pointer is adjusted by the number of bytes actually read.
* @param HANDLE This is the handle of the file to read. HANDLE should be the handle opened by the Bfile_OpenFile or
* Bfile_OpenMainMemory function.
* @param buf This is the pointer to the buffer that receives the data read from the file.
* @param size This is the number of bytes to be read from the file.
* @param readpos This is the starting position to read. If the readpos parameter is -1, this function reads data from the position
* indicated by the file pointer. If the readpos parameter greater than or equal to 0, this function reads data from
* the position indicated by the readpos parameter.
* @return If the function succeeds, this function returns the number of bytes actually read. It is greater than or equal to 0.
* If the function fails, the return value is an error code. It is a negative value.
* @remarks If you read the Windows file, the multi byte data is stored in Little Endian format. To use this data, you should
* convert the multi byte data into the Big Endian format.
*/
int Bfile_ReadFile(int HANDLE, void *buf, int size, int readpos);
You have to set last argument to (-1)

Zezombye
Posts: 33
Joined: Mon Jul 30, 2018 8:29 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by Zezombye » Sat Sep 08, 2018 9:31 am

Thanks. This is very misleading, I expected reader->pos to be the position in the file, and reader->len to be the length of the file. Also the name mp_reader_posix_readbyte() implies that it reads only one byte, it should be mp_reader_posix_readbytechunk or something that implies it reads 20 bytes by 20 bytes. (mp_reader_posix_readbytes() would be misleading too, I'd expect that this function reads all the bytes of the file)

I put -1 as the position and now it works fine, thanks :D

As for the closing bug, I put the freeing lines at the end of the syntax_error label (as the nlr_raise seems to act like a return):

Code: Select all

syntax_error:;
		
        mp_obj_t exc;
        if (lex->tok_kind == MP_TOKEN_INDENT) {
            exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
                "unexpected indent");
        } else if (lex->tok_kind == MP_TOKEN_DEDENT_MISMATCH) {
            exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
                "unindent does not match any outer indentation level");
        } else {
            exc = mp_obj_new_exception_msg(&mp_type_SyntaxError,
                "invalid syntax");
        }
        // add traceback to give info about file name and location
        // we don't have a 'block' name, so just pass the NULL qstr to indicate this
        mp_obj_exception_add_traceback(exc, lex->source_name, lex->tok_line, MP_QSTR_NULL);
		
		
		// free the memory that we don't need anymore
		m_del(rule_stack_t, parser.rule_stack, parser.rule_stack_alloc);
		m_del(mp_parse_node_t, parser.result_stack, parser.result_stack_alloc);

		// we also free the lexer on behalf of the caller
		mp_lexer_free(lex);
		
        nlr_raise(exc);
It seems to work fine, however can this cause errors elsewhere? Why wasn't this done by default?

jickster
Posts: 546
Joined: Thu Sep 07, 2017 8:57 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by jickster » Sat Sep 08, 2018 11:24 am

Micropython is a work in progress.


Sent from my iPhone using Tapatalk Pro

Zezombye
Posts: 33
Joined: Mon Jul 30, 2018 8:29 pm

Re: How do I make a port of MicroPython for Casio calculators?

Post by Zezombye » Tue Nov 13, 2018 5:45 pm

As 2064 is very close to a power of 2, I think there is a hardcoded limit somewhere. I don't think there is a coincidence because 2 kb is quite low
I knew it, there is indeed a hardcoded limit (in main.c, hiding in plain sight):

Code: Select all

static char heap[2048];
Changing the size of the heap works, but in my case I have only 8 kb of static RAM, so I needed to allocate on the heap:

Code: Select all

const char* heap;

int mpy_main(char *text) {
	#if MICROPY_ENABLE_GC
	heap = malloc(16384);
	#endif
	//...
}

Post Reply