Page 4 of 7
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Mon Sep 03, 2018 10:59 am
by Zezombye
There you go:
https://youtu.be/X9MSxI2BVK0
If you want to try it yourself, you can always use the emulator:
- Download the emulator at
https://edu.casio.com/freetrial/fr/free ... LANGUAGE=1
- Download the program (.g1a) at
https://www.planet-casio.com/Fr/program ... howid=3603
- Install the emulator, open the shortcut on desktop
- Go to memory menu, press F3 (SD)
- Select the g1a you downloaded, and when prompted press 2 for storage memory
- Go at the bottom of the main menu and select the python icon.
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Mon Sep 03, 2018 5:41 pm
by Zezombye
I managed to make the import statement work how I want to ("import test" imports from the file test.py), however I found that it doesn't close the file if an error happens (syntax error, or out of memory error for example).
I just overrode the close, read and open functions in reader.c with the Casio ones (which are very similar in function, if not the exact same).
How do I make sure MPy closes the file? I can't put the file handle in global and close it myself, because multiple file handles might be opened.
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Mon Sep 03, 2018 7:53 pm
by jickster
Zezombye wrote: ↑Mon Sep 03, 2018 5:41 pm
I managed to make the import statement work how I want to ("import test" imports from the file test.py), however I found that it doesn't close the file if an error happens (syntax error, or out of memory error for example).
I just overrode the close, read and open functions in reader.c with the Casio ones (which are very similar in function, if not the exact same).
How do I make sure MPy closes the file? I can't put the file handle in global and close it myself, because multiple file handles might be opened.
When you call the sequence of functions to compile and execute
Code: Select all
mp_lexer_t * lex = mp_lexer_new_from_str_len(0, ptr_code, (size_t)code_length, (size_t)0);
mp_parse_tree_t parse_tree = mp_parse(lex, parse_type);
mp_obj_t module_func = mp_compile(&parse_tree, lex->source_name, MP_EMIT_OPT_NONE, repl);
mp_call_function_0(module_func);
mp_parse calls mp_lexer_free()
Code: Select all
// we also free the lexer on behalf of the caller
mp_lexer_free(lex);
Which calls a function pointer to close the file
Code: Select all
void mp_lexer_free(mp_lexer_t *lex) {
if (lex) {
lex->reader.close(lex->reader.data);
vstr_clear(&lex->vstr);
m_del(uint16_t, lex->indent_level, lex->alloc_indent_level);
m_del_obj(mp_lexer_t, lex);
}
}
Which should've been set in
Code: Select all
void mp_reader_new_file_from_fd(mp_reader_t *reader, int fd, bool close_fd) {
mp_reader_posix_t *rp = m_new_obj(mp_reader_posix_t);
rp->close_fd = close_fd;
rp->fd = fd;
int n = read(rp->fd, rp->buf, sizeof(rp->buf));
if (n == -1) {
if (close_fd) {
close(fd);
}
mp_raise_OSError(errno);
}
rp->len = n;
rp->pos = 0;
reader->data = rp;
reader->readbyte = mp_reader_posix_readbyte;
reader->close = mp_reader_posix_close;
}
Called in
Code: Select all
mp_lexer_t *mp_lexer_new_from_fd(qstr filename, int fd, bool close_fd) {
mp_reader_t reader;
mp_reader_new_file_from_fd(&reader, fd, close_fd);
return mp_lexer_new(filename, reader);
}
Conceptually this should make sense to you that the file is closed at the end of mp_parse().
Once the text of a .py has been digested, by the parsing stage, into a tree of data-structures for the compiler to consume, there's no need for the file to be open.
Obviously replace the relevant function pointers with your own `Bfile*()` functions
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Mon Sep 03, 2018 8:05 pm
by jickster
Zezombye wrote: ↑Mon Sep 03, 2018 5:41 pm
I managed to make the import statement work how I want to ("import test" imports from the file test.py), however I found that it doesn't close the file if an error happens (syntax error, or out of memory error for example).
I just overrode the close, read and open functions in reader.c with the Casio ones (which are very similar in function, if not the exact same).
How do I make sure MPy closes the file? I can't put the file handle in global and close it myself, because multiple file handles might be opened.
Ok I realize I didn't answer your question with my last post.
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Mon Sep 03, 2018 8:29 pm
by jickster
Zezombye wrote: ↑Mon Sep 03, 2018 5:41 pm
I managed to make the import statement work how I want to ("import test" imports from the file test.py), however I found that it doesn't close the file if an error happens (syntax error, or out of memory error for example).
I just overrode the close, read and open functions in reader.c with the Casio ones (which are very similar in function, if not the exact same).
How do I make sure MPy closes the file? I can't put the file handle in global and close it myself, because multiple file handles might be opened.
I think you've found a bug.
When a syntax error happens, following lines from mp_parse() execute:
Code: Select all
syntax_error:;
mp_obj_t exc;
if (lex->tok_kind == MP_TOKEN_INDENT) {
exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
"unexpected indent");
} else if (lex->tok_kind == MP_TOKEN_DEDENT_MISMATCH) {
exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
"unindent does not match any outer indentation level");
} else {
exc = mp_obj_new_exception_msg(&mp_type_SyntaxError,
"invalid syntax");
}
// add traceback to give info about file name and location
// we don't have a 'block' name, so just pass the NULL qstr to indicate this
mp_obj_exception_add_traceback(exc, lex->source_name, lex->tok_line, MP_QSTR_NULL);
nlr_raise(exc);
The information stored in the exception does NOT contain the "fd" stored in reader->data necessary to close the file.
Code: Select all
void mp_reader_new_file_from_fd(mp_reader_t *reader, int fd, bool close_fd) {
mp_reader_posix_t *rp = m_new_obj(mp_reader_posix_t);
rp->close_fd = close_fd;
rp->fd = fd;
int n = read(rp->fd, rp->buf, sizeof(rp->buf));
if (n == -1) {
if (close_fd) {
close(fd);
}
mp_raise_OSError(errno);
}
rp->len = n;
rp->pos = 0;
reader->data = rp;
reader->readbyte = mp_reader_posix_readbyte;
reader->close = mp_reader_posix_close;
}
Issue is
https://github.com/micropython/micropython/issues/4095
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Fri Sep 07, 2018 5:24 pm
by Zezombye
I am perplexed by this bit of code:
Code: Select all
typedef struct _mp_reader_posix_t {
bool close_fd;
int fd;
size_t len;
size_t pos;
byte buf[25];
} mp_reader_posix_t;
STATIC mp_uint_t mp_reader_posix_readbyte(void *data) {
mp_reader_posix_t *reader = (mp_reader_posix_t*)data;
if (reader->pos >= reader->len) {
if (reader->len == 0) {
return MP_READER_EOF;
} else {
//int n = read(reader->fd, reader->buf, sizeof(reader->buf));
int n = Bfile_ReadFile(reader->fd, reader->buf, sizeof(reader->buf), reader->pos);
if (n <= 0) {
reader->len = 0;
return MP_READER_EOF;
}
reader->len = n;
reader->pos = 0;
}
}
return reader->buf[reader->pos++];
}
Basically the buffer causes errors for files bigger than it (it was originally 20, but I increased it to 25 to be sure it was the buffer).
For a buffer of 25:
- Importing a file of 0-25 bytes is no problem.
- Importing a file of 26-51 bytes throws a "syntax error"
- Importing a file of 52 bytes or more throws a memory error (allocation failed).
Why is this buffer so low, and not dynamically allocated?
Also, the function Bfile_ReadFile takes the same arguments as the read() function, but with the position at the end. What I don't understand is that the reader->pos is reset to 0 at each byte read, so it should always read the first byte of the file, but it doesn't. How does it work?
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Fri Sep 07, 2018 7:23 pm
by jickster
Zezombye wrote: ↑Fri Sep 07, 2018 5:24 pm
I am perplexed by this bit of code:
Code: Select all
typedef struct _mp_reader_posix_t {
bool close_fd;
int fd;
size_t len;
size_t pos;
byte buf[25];
} mp_reader_posix_t;
STATIC mp_uint_t mp_reader_posix_readbyte(void *data) {
mp_reader_posix_t *reader = (mp_reader_posix_t*)data;
if (reader->pos >= reader->len) {
if (reader->len == 0) {
return MP_READER_EOF;
} else {
//int n = read(reader->fd, reader->buf, sizeof(reader->buf));
int n = Bfile_ReadFile(reader->fd, reader->buf, sizeof(reader->buf), reader->pos);
if (n <= 0) {
reader->len = 0;
return MP_READER_EOF;
}
reader->len = n;
reader->pos = 0;
}
}
return reader->buf[reader->pos++];
}
Basically the buffer causes errors for files bigger than it (it was originally 20, but I increased it to 25 to be sure it was the buffer).
For a buffer of 25:
- Importing a file of 0-25 bytes is no problem.
- Importing a file of 26-51 bytes throws a "syntax error"
- Importing a file of 52 bytes or more throws a memory error (allocation failed).
Why is this buffer so low, and not dynamically allocated?
Also, the function Bfile_ReadFile takes the same arguments as the read() function, but with the position at the end. What I don't understand is that the reader->pos is reset to 0 at each byte read, so it should always read the first byte of the file, but it doesn't. How does it work?
The reason why this is not straightforward is because an optimization was made: instead of calling "read()" everytime you want to read ONE byte, this function reads in a chunk of 25 bytes at a time.
The naming of the struct members could be clearer.
reader->len
is the number of bytes obtained from "read()". This number will be sizeof(reader->buf) until you reach near the end of the file when you have less than sizeof(reader->buf) bytes left to read.
Ideally it would be named "reader->num_chars_in_buf"
reader->pos
is the position inside the reader->buf for the next character.
Ideally it would be named "reader->pos_next_char_in_buf"
For "Bfile_ReadFile" to be a drop-in replacement for "read", it has to behave exactly like "read()".
Does it? I don't have the description of it.
Does it return the actual number of bytes that were read?
https://gist.github.com/holubv/89df8019 ... 7659727d0e
/**
* Reads data from a file, starting at the position indicated by the file pointer. After the
* read operation has been completed, the file pointer is adjusted by the number of bytes actually read.
* @param HANDLE This is the handle of the file to read. HANDLE should be the handle opened by the Bfile_OpenFile or
* Bfile_OpenMainMemory function.
* @param buf This is the pointer to the buffer that receives the data read from the file.
* @param size This is the number of bytes to be read from the file.
* @param readpos This is the starting position to read. If the readpos parameter is -1, this function reads data from the position
* indicated by the file pointer. If the readpos parameter greater than or equal to 0, this function reads data from
* the position indicated by the readpos parameter.
* @return If the function succeeds, this function returns the number of bytes actually read. It is greater than or equal to 0.
* If the function fails, the return value is an error code. It is a negative value.
* @remarks If you read the Windows file, the multi byte data is stored in Little Endian format. To use this data, you should
* convert the multi byte data into the Big Endian format.
*/
int Bfile_ReadFile(int HANDLE, void *buf, int size, int readpos);
You have to set last argument to (-1)
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Sat Sep 08, 2018 9:31 am
by Zezombye
Thanks. This is very misleading, I expected reader->pos to be the position in the file, and reader->len to be the length of the file. Also the name mp_reader_posix_readbyte() implies that it reads only one byte, it should be mp_reader_posix_readbytechunk or something that implies it reads 20 bytes by 20 bytes. (mp_reader_posix_readbytes() would be misleading too, I'd expect that this function reads all the bytes of the file)
I put -1 as the position and now it works fine, thanks
As for the closing bug, I put the freeing lines at the end of the syntax_error label (as the nlr_raise seems to act like a return):
Code: Select all
syntax_error:;
mp_obj_t exc;
if (lex->tok_kind == MP_TOKEN_INDENT) {
exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
"unexpected indent");
} else if (lex->tok_kind == MP_TOKEN_DEDENT_MISMATCH) {
exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
"unindent does not match any outer indentation level");
} else {
exc = mp_obj_new_exception_msg(&mp_type_SyntaxError,
"invalid syntax");
}
// add traceback to give info about file name and location
// we don't have a 'block' name, so just pass the NULL qstr to indicate this
mp_obj_exception_add_traceback(exc, lex->source_name, lex->tok_line, MP_QSTR_NULL);
// free the memory that we don't need anymore
m_del(rule_stack_t, parser.rule_stack, parser.rule_stack_alloc);
m_del(mp_parse_node_t, parser.result_stack, parser.result_stack_alloc);
// we also free the lexer on behalf of the caller
mp_lexer_free(lex);
nlr_raise(exc);
It seems to work fine, however can this cause errors elsewhere? Why wasn't this done by default?
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Sat Sep 08, 2018 11:24 am
by jickster
Micropython is a work in progress.
Sent from my iPhone using Tapatalk Pro
Re: How do I make a port of MicroPython for Casio calculators?
Posted: Tue Nov 13, 2018 5:45 pm
by Zezombye
As 2064 is very close to a power of 2, I think there is a hardcoded limit somewhere. I don't think there is a coincidence because 2 kb is quite low
I knew it, there is indeed a hardcoded limit (in main.c, hiding in plain sight):
Changing the size of the heap works, but in my case I have only 8 kb of static RAM, so I needed to allocate on the heap:
Code: Select all
const char* heap;
int mpy_main(char *text) {
#if MICROPY_ENABLE_GC
heap = malloc(16384);
#endif
//...
}