Read csv and cast str to int

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
joedog
Posts: 5
Joined: Tue Jan 18, 2022 3:37 pm

Read csv and cast str to int

Post by joedog » Tue Jan 18, 2022 4:14 pm

I have calibration data stored in a .csv file. It is structured liked:

1,2,3,4,5
6,7,8,9,10
...

I am reading the file into a Python script like:

Code: Select all

 
f = open("filename.csv", "r")
data = f.read()
f.close()
data = data.replace('\r','').split("\n")

# Now I'm trying to convert the strings to ints, but I cannot figure out the issue is:
for line in data:
     lineList = line.split(',')
     print(lineList)

     for i in lineList:
          print(repr(i))
          print(repr(int(i))) # This fails
I've tried appending things like '\r' or '\n' to the strings, but it didn't work. I am printing with repr() so I can see additional characters like '\r'. If I create a string variable like [/code]x = '7'[/code], the casting to int works fine, but for some reason there is something different about the strings from the .csv file, but I can't figure out what the difference is.

davef
Posts: 811
Joined: Thu Apr 30, 2020 1:03 am
Location: Christchurch, NZ

Re: Read csv and cast str to int

Post by davef » Tue Jan 18, 2022 6:16 pm

When I make up my .csv file for my datalogger app I have a comma at the end of each string. I must have copied an example somewhere.

joedog
Posts: 5
Joined: Tue Jan 18, 2022 3:37 pm

Re: Read csv and cast str to int

Post by joedog » Tue Jan 18, 2022 8:48 pm

The loading of the csv file works properly up until the point when I try to convert a string like '7' into an integer with int()

davef
Posts: 811
Joined: Thu Apr 30, 2020 1:03 am
Location: Christchurch, NZ

Re: Read csv and cast str to int

Post by davef » Wed Jan 19, 2022 12:29 am

I had to google repr https://www.geeksforgeeks.org/str-vs-repr-in-python/ as I didn't even know it existed.

Code: Select all

i = '45'
print(repr(int(i))) 
works for me.
v1.17 on the ESP32. Have I misunderstood?

joedog
Posts: 5
Joined: Tue Jan 18, 2022 3:37 pm

Re: Read csv and cast str to int

Post by joedog » Wed Jan 19, 2022 2:55 am

If I define and declare a string in a line like a = ‘7’, I can cast to int with int(a) and this works properly. However, if I read the string in from a csv file in the way that I described, I can get to the point where I have a string ‘7’ that prints identically to the variable a (I used repr to verify that when you define string like a = ‘7’ there isn’t some additional line return or any like that added that isn’t obvious from just doing print(a)). But even though I have some new variable (i is the variable from my example), it returns an error when converting to int. I cannot figure out how a and i are different, but my guess is there is some hidden suffix or attribute to the string object that I am missing or added from reading the csv file, but using print() and print(repr()) there was nothing obvious. The easiest way to recreate the error would be make a small csv file and try to get to the point where you have integers or floats rather than strings.

davef
Posts: 811
Joined: Thu Apr 30, 2020 1:03 am
Location: Christchurch, NZ

Re: Read csv and cast str to int

Post by davef » Wed Jan 19, 2022 3:40 am

Does print (str(string from csv )) work?

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Read csv and cast str to int

Post by Roberthh » Wed Jan 19, 2022 8:13 am

However, if I read the string in from a csv file in the way that I described, I can get to the point where I have a string ‘7’ that prints identically to the variable a
Can you make a hex.dump for the string that you get from the file and the one you try to convert?
Use:

import binascii
binascii.hexlify(str)

joedog
Posts: 5
Joined: Tue Jan 18, 2022 3:37 pm

Re: Read csv and cast str to int

Post by joedog » Wed Jan 19, 2022 9:29 pm

The conversion to str() with the input already of type str() works without error. When I tried the binascii.hexlify() with the input as a string, I got the following error:

TypeError: a bytes-like object is required

However I also get this error for any arbitrary string like '45' that I input.

davef
Posts: 811
Joined: Thu Apr 30, 2020 1:03 am
Location: Christchurch, NZ

Re: Read csv and cast str to int

Post by davef » Wed Jan 19, 2022 10:18 pm

I was going to ask earlier because I don't the reason ... why do you want to use repr?

joedog
Posts: 5
Joined: Tue Jan 18, 2022 3:37 pm

Re: Read csv and cast str to int

Post by joedog » Wed Jan 19, 2022 11:00 pm

repr() allows me to print some additional characters that are hidden by print(). For example the '\r' character for new lines will not be shown with print with would be shown with print(repr()). I think the problem is related so some kind of hidden characters that repr() also does not print. I solved the problem by creating a new csv file that from a blank file (touch newFile.csv), and then copying the contents of the csv into the new file. The old file I had created in Excel and then exported to csv on a Mac. I think there must have been some different hidden characters in that file, though I was never able to find out what the difference was between the files (since both were identical in my text editor and when using print() or print(repr()).

Post Reply