Page 1 of 1
TypeError: 'bytearray' object doesn't support item deletion
Posted: Mon Jul 06, 2020 7:25 pm
by a-ha
It appears the del on bytearray is not supported.
I am getting TypeError: 'bytearray' object doesn't support item deletion
Any idea for workaround?
-aha
Code: Select all
buffer = bytearray()
...
del buffer[0:i + 1]
Re: TypeError: 'bytearray' object doesn't support item deletion
Posted: Mon Jul 06, 2020 7:46 pm
by Roberthh
Assign a slice.
buffer = buffer[i:]
not efficient, but works.
Re: TypeError: 'bytearray' object doesn't support item deletion
Posted: Mon Jul 06, 2020 10:20 pm
by a-ha
That would work, but I am trying to avoid the object copy.
This is a data buffer which can contain huge data/bytes. The ideal is to remove in-line.
Re: TypeError: 'bytearray' object doesn't support item deletion
Posted: Tue Jul 07, 2020 6:04 am
by Roberthh
Is the data in some way structured? Then you could handle it packet-wise as a list of packets.
Re: TypeError: 'bytearray' object doesn't support item deletion
Posted: Tue Jul 07, 2020 6:10 pm
by a-ha
It is being used to read streaming tcp data from the server. At the low level data reading, it is unstructured.
Re: TypeError: 'bytearray' object doesn't support item deletion
Posted: Wed Jul 08, 2020 5:20 am
by jimmo
a-ha wrote: ↑Mon Jul 06, 2020 10:20 pm
That would work, but I am trying to avoid the object copy.
Even if "del <bytearray>" was supported, I'm not sure there's a way to avoid a copy. (Unless I'm missing something -- what did you have in mind?)
The best you can do is at least to make the copy "in-place" (i.e. shifting the rest of the bytearray down), by using a slice assignment.
This will have the effect of shifting the rest of buffer down to the start, but no new buffer will be created in RAM.
Roberthh wrote: ↑Mon Jul 06, 2020 7:46 pm
Assign a slice.
buffer = buffer[i:]
not efficient, but works.
If your goal is to minimise memory usage, then this might possibly be the more efficient option. As with the previous example, it temporarily results in two copies of the buffer, but the new copy will be sized exactly correctly, and the GC will clean up the old one, so the resulting state is overall that there are `i` total fewer bytes allocated.
Re: TypeError: 'bytearray' object doesn't support item deletion
Posted: Wed Jul 08, 2020 10:13 am
by chuckbook
What about:
Code: Select all
lba = 1000 # size of bytearray
n = 10 # number of bytes to delete
ba = bytearray(lba)
mba = memoryview(ba)
mba[0:-n] = mba[n:]
This construct leaves the bytearray intact, which might come in handy for later reuse.
Re: TypeError: 'bytearray' object doesn't support item deletion
Posted: Thu Jul 09, 2020 6:00 am
by pythoncoder
chuckbook wrote: ↑Wed Jul 08, 2020 10:13 am
...This construct leaves the bytearray intact, which might come in handy for later reuse.
I don't think it does.
Code: Select all
n = 10 # number of bytes to delete
ba = bytearray(range(20))
print(ba)
mba = memoryview(ba)
mba[0:-n] = mba[n:]
print(bytes(mba))
print(ba)
produces
Code: Select all
bytearray(b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13')
b'\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13'
bytearray(b'\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13')
As @jimmo says, the problem is actually fundamental. A bytearray stores the bytes in contiguous memory locations. Deletion of a range must involve potentially large scale data copying, the only question being how fast can you do it? CPython must be doing this under the hood to maintain the properties of a bytearray
As far as I know the only solution is to abandon contiguous storage in favour of structures like linked lists where segments can be mapped out without moving data.
Re: TypeError: 'bytearray' object doesn't support item deletion
Posted: Thu Jul 09, 2020 9:45 am
by chuckbook
Maybe I wasn't precise enough with my wording.
Intact was meant the way that storage space stays fixed not it's content.
Of course there are better ways to deal with non-contiguous data, but if you have to shift data chunks around it is pretty efficient.
Re: TypeError: 'bytearray' object doesn't support item deletion
Posted: Thu Jul 30, 2020 2:43 am
by miltmobley
You are not being completely clear as to what you want to do with this big data.
See
https://docs.python.org/3/library/stdty ... #bytearray for info as to what you can do to
a bytearray.