However, as a C programmer, you also know that strings are arrays of characters and that nothing stops you from looking at only part of this array without copying it, through the use of basic pointer arithmetic – assuming that the entire string is in a contiguous memory area. If you already have written C code, you know that using memcpy() has a significant cost, both in term of memory usage and regarding general performance: copying memory is slow. When dealing with extensive data, performing this kind of operation on large byte arrays is going to be a disaster. The slice operator is copying the entirety of content, minus the first KB, into a new string object. So what's interesting here, is to notice that the memory usage of the program increased by about 10 MB when building the variable content_to_write. The instruction on the line just after, content, copies the entire block of data minus the first KB - allocating 10 more megabytes. Python needs to allocate around 10 MB of memory to store this data as a string.
The call to source.read reads 10 MB from /dev/urandom. Running the above program using memory_profiler produces the following: $ python -m memory_profiler memoryview/copy.pyĬontent length: 10240000, content to write length 10238976
Print("Content length: %d, content to write length %d" % With open("/dev/urandom", "rb") as source: To examine out the memory usage of this program, we will use memory_profiler, an excellent Python package that allows us to see the memory usage of a program line by line. Let's consider a small program which reads a large file of binary data, andĬopies it partially into another file. However, handling such a large quantity of input in strings can be very ineffective once you start manipulating them by copying, slicing, and modifying. This data is usually represented and manipulated in the form of strings. Whatever your programs are doing, they often have to deal with vast amounts of data.
0 kommentar(er)
