Building Micropython + urequests issue

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
doublevee
Posts: 75
Joined: Mon Jul 02, 2018 11:09 pm

Building Micropython + urequests issue

Post by doublevee » Mon Jul 12, 2021 8:17 pm

Hi,

As my painful progress into programming and Micropython continues, I have found myself stuck in another frustrating situation (due to my lack of knowledge).

I have been reading about how to build MP, and have (I think) managed to build a firmware file for my STM32F411 based product. I am using a WizNet W5500 chip as well and have the W5500 drivers installed and all that is working. As the F411 does not have any network capability, I have had to enable this in the build and I think I have managed to enable SSL support as well. I need to be able to interact with HTTPS API endpoints.

I have 2 questions:

1) When I build my code, it always says it is v1.15, but I thought the current version is 1.16. I have used git clone from the MP github to update all the files on my PC, so I was expecting these to be the latest versions of the files. Why when I build my board file does it always say 1.15? I do a make clean before each build. Apologies if this is an embarrassing question to ask, but I have to ask as I don't understand the fundamentals.

2) I am trying to use urequests, and have downloaded urequests.py, copied that to flash and can import that ok. However, when I hit the following code in urequests, i get the following:

>>> import socket
>>> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_SEC)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'IPPROTO_SEC'

So I checked the socket module, and there is no IPPROTO_SEC

>>> dir(socket)
['__class__', '__name__', 'AF_INET', 'AF_INET6', 'IPPROTO_IP', 'IP_ADD_MEMBERSHIP', 'SOCK_DGRAM', 'SOCK_RAW', 'SOCK_STREAM', 'SOL_SOCKET', 'SO_REUSEADDR', 'callback', 'getaddrinfo', 'print_pcbs', 'reset', 'socket']

I would really appreciate a steer on this please - I have tried to read and research as much as possible but I don't understand how to progress with urequests.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Building Micropython + urequests issue

Post by jimmo » Tue Jul 13, 2021 1:11 am

doublevee wrote:
Mon Jul 12, 2021 8:17 pm
Why when I build my board file does it always say 1.15?
When did you make the git clone? Are you on teh "master" branch, and is it synced with upstream?

If not

Code: Select all

git pull -r
should bring you up to date. (You may need to stash your changes first).

Can you do

Code: Select all

git rev-parse --short HEAD
git remote -v
git status
and copy the output here?
doublevee wrote:
Mon Jul 12, 2021 8:17 pm
2) I am trying to use urequests, and have downloaded urequests.py, copied that to flash and can import that ok. However, when I hit the following code in urequests, i get the following:
Where did you get your copy of urequests from? I don't see that line in the official version... looks like you have a version for the WiPy pycom board?

See https://github.com/micropython/micropyt ... equests.py

If you're building firmware yourself, the best way to add urequests is to freeze it into the firmware. (See this thread for some pointers -- viewtopic.php?f=2&t=10808 -- but basically you need to add a manifest.py to your board, set FROZEN_MANIFEST in mpconfigboard.mk, and add

Code: Select all

freeze("$(MPY_LIB_DIR)/python-ecosys/urequests", "urequests.py")
(This assumes you have micropython-lib cloned alongside the micropython repo)

doublevee
Posts: 75
Joined: Mon Jul 02, 2018 11:09 pm

Re: Building Micropython + urequests issue

Post by doublevee » Tue Jul 13, 2021 5:24 pm

Many thanks Jimmo - your help is very appreciated. 'git pull -r' seemed to fix things for me.
-----------------------------------------------------------
'git rev-parse --short HEAD' returns:
3f836c1c9

'git remote -v' returns:
origin https://github.com/badgermedia/micropython/ (fetch)
origin https://github.com/badgermedia/micropython/ (push)
upstream https://github.com/micropython/micropython-lib (fetch)
upstream https://github.com/micropython/micropython-lib (push)

'git status' returns:
On branch master
Your branch is up-to-date with 'origin/master'.

Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: lib/axtls (new commits)
modified: lib/pico-sdk (new commits)
modified: lib/tinyusb (new commits)

Untracked files:
(use "git add <file>..." to include in what will be committed)
micropython-lib/
micropython/
ports/stm32/boards/BM_BUILD_1/
ports/stm32/boards/BM_Proto_v1/
ports/stm32/boards/Black_Pill_1_16_Network/
ports/stm32/boards/stm32f411.ld.old

no changes added to commit (use "git add" and/or "git commit -a")
-----------------------------------------------------------
I have no idea what this all means in reality. I just want to have the up to date files on my laptop for building. Git seems hugely complex to use.

I have found the correct version of urequests (thank you!) and am looking into the manifest approach (this is totally new to me) regarding freezing modules.

doublevee
Posts: 75
Joined: Mon Jul 02, 2018 11:09 pm

Re: Building Micropython + urequests issue

Post by doublevee » Tue Jul 13, 2021 5:42 pm

I have also just done a new build and the firmware version is still showing as 'MicroPython v1.15-317-g3f836c1c9-dirty' so I have obviously got something else wrong.

I have however managed to freeze urequests and now understand that process - not too bad actually :)

Thanks again Jimmo - your help is invaluable.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Building Micropython + urequests issue

Post by jimmo » Wed Jul 14, 2021 12:29 pm

doublevee wrote:
Tue Jul 13, 2021 5:24 pm
'git rev-parse --short HEAD' returns:
3f836c1c9
OK, cool, so that's the commit that adds your board files, and I can see at https://github.com/badgermedia/micropython/ that this is on top of the current HEAD of https://github.com/micropython/micropython So you're definitely up to date.
This is a bit strange... you've got micropython-lib as your upstream. That's going to lead to some confusion.

I would suggest doing:

Code: Select all

git remote remove upstream
git remote add upstream https://github.com/micropython/micropython
this will set your "upstream" remote to the correct location.
doublevee wrote:
Tue Jul 13, 2021 5:24 pm
modified: lib/axtls (new commits)
modified: lib/pico-sdk (new commits)
modified: lib/tinyusb (new commits)
Depending on which board you're building for this might not be an issue, but probably worth cleaning this up.

Code: Select all

git submodule deinit -f lib/axtls lib/pico-sdk lib/tinyusb
git submodule update --init --recursive lib/axtls lib/pico-sdk lib/tinyusb
If the second command fails, then re-do the first line, then

Code: Select all

rm -rf .git/modules/lib/axtls .git/modules/lib/pico-sdk .git/modules/lib/tinyusb
before running the second line. There's this really annoying thing in git where if a submodule has a submodule that moves location, then the outer repo (i.e. micropython) gets confused.
doublevee wrote:
Tue Jul 13, 2021 5:42 pm
I have also just done a new build and the firmware version is still showing as 'MicroPython v1.15-317-g3f836c1c9-dirty' so I have obviously got something else wrong.
This message is generated during the build by finding the newest tag that is in the history of the current branch.

So what's happening is that your repo has your fork as it's origin, and somehow you've updated your fork to upstream (I'm guessing using the new "Fetch upstream" button that was recently added to the github web ui), but for some reason the tags haven't synced, so it doesn't know that the 1.16 tag exists (despite having the commits).

I'm not too sure how this happened so I don't know the best way to fix it... my guess is to do the step I suggested above to fix your "upstream" remote, then run

Code: Select all

git fetch upstream --tags
FWIW, the way I prefer to work is instead to have origin as micropython/micropython, and remote "jimmo" for jimmo/micropython. So my "master" branch in my local repo's upstream is "origin". (It's confusing that the popular advice is to name your remote "upstream", because upstream in git means multiple things, namely the upstream of a branch)

doublevee
Posts: 75
Joined: Mon Jul 02, 2018 11:09 pm

Re: Building Micropython + urequests issue

Post by doublevee » Thu Jul 15, 2021 7:50 pm

Jimmo - thank you so much. You have really helped me out here and I really appreciate the time you have taken to explain all that.

The 'tags' command has done the job as well and I am confident I have the latest all working now.

I am confused about the tree structure for my Git setup. In my home directory I have a folder called 'micropython'. I have the commands set up that you helped me with. At one stage I had another 'micropython' folder after this one somehow! When you set up the git commands, do you do that pull into the directory you are in, or does it use that as a relative position in the path, and so when you pull from upstream, it pulls that folder down as well (hence how I might have ended up with micropython/micropython).

For my micropython-lib folder (which is in micropython), do I setup separate git commands in that folder to point to my micropython-lib fork? I'm not really sure how I 'sync' different folders? Or should i have micropython-lib also in my home directory and then setup git commands etc (I did get a warning about having a git folder inside another one).

Believe it not, I have actually tried to read up on Git and watch some YouTube videos but they are so confusing - do you recommend a resource I could look at, and so stop asking you annoying (and no doubt very trivial) questions?

Post Reply