NumPy Input and Output: memmap() function
numpy.memmap() function
The memmap() function is used to create a memory-map to an array stored in a binary file on disk.
Memory-mapped files are used for accessing small segments of large files on disk, without reading the entire file into memory.
NumPy's memmap's are array-like objects. This differs from Python's mmap module, which uses file-like objects.
Syntax:
class numpy.memmap
Version: 1.15.0
Parameter:
Name | Description | Required / Optional |
---|---|---|
filename | The file name or file object to be used as the array data buffer. str, file-like object, or pathlib.Path instance |
Required |
dtype | The data-type used to interpret the file contents. Default is uint8. data-type |
Optional |
mode | The file is opened in this mode: 'r' Open existing file for reading only. 'r+' Open existing file for reading and writing. 'w+' Create or overwrite existing file for reading and writing. 'c' Copy-on-write: assignments affect data in memory, but changes are not saved to disk. The file on disk is read-only. Default is 'r+'. {'r+', 'r', 'w+', 'c'} |
Optional |
offset | In the file, array data starts at this offset. Since offset is measured in bytes, it should normally be a multiple of the byte-size of dtype. When mode != 'r', even positive offsets beyond end of file are valid; The file will be extended to accommodate the additional data. By default, memmap will start at the beginning of the file, even if filename is a file pointer fp and fp.tell() != 0. int |
Optional |
shape | The desired shape of the array. If mode == 'r' and the number of remaining bytes after offset is not a multiple of the byte-size of dtype, you must specify shape. By default, the returned array will be 1-D with the number of elements determined by file size and data-type. tuple |
Optional |
order | Specify the order of the ndarray memory layout: row-major, C-style or column-major, Fortran-style. This only has an effect if the shape is greater than 1-D. The default order is 'C'. {'C', 'F'} |
Optional |
Notes:
The memmap object can be used anywhere an ndarray is accepted.
Given a memmap fp, isinstance(fp, numpy.ndarray) returns True.
Memory-mapped files cannot be larger than 2GB on 32-bit systems.
When a memmap causes a file to be created or extended beyond its current size in the filesystem, the contents of the new part are unspecified.
On systems with POSIX filesystem semantics, the extended part will be filled with zero bytes.
>>> data = np.arange(16, dtype='float32')
>>> data.resize((4,4))
>>> from tempfile import mkdtemp
>>> import os.path as path
>>> filename = path.join(mkdtemp(), 'newfile.dat')
This example uses a temporary file so that doctest doesn’t write files to your directory.
You would use a 'normal' filename.
>>> from tempfile import mkdtemp
>>> import os.path as path
>>> filename = path.join(mkdtemp(), 'newfile.dat')
Create a memmap with dtype and shape that matches our data:
>>> import numpy as np
>>> from tempfile import mkdtemp
>>> import os.path as path
>>> filename = path.join(mkdtemp(), 'newfile.dat')
>>> fpath = np.memmap(filename, dtype='float32', mode='w+', shape=(4,4))
>>> fpath
Output:
memmap([[0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.]], dtype=float32)
Write data to memmap array:
>>> import numpy as np
>>> from tempfile import mkdtemp
>>> import os.path as path
>>> data = np.arange(16, dtype='float32')
>>> data.resize((4,4))
>>> filename = path.join(mkdtemp(), 'newfile.dat')
>>> fpath = np.memmap(filename, dtype='float32', mode='w+', shape=(4,4))
>>> fpath[:] = data[:]
>>> print(fpath)
Output:
[[ 0. 1. 2. 3.] [ 4. 5. 6. 7.] [ 8. 9. 10. 11.] [12. 13. 14. 15.]]
>>> fpath.filename == path.abspath(filename)
Output:
True
Deletion flushes memory changes to disk before removing the object:
del fpath
Load the memmap and verify data was stored:
>>> newfp = np.memmap(filename, dtype='float32', mode='r', shape=(4,4))
>>> newfp
Output:
memmap([[ 0., 1., 2., 3.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.], [12., 13., 14., 15.]], dtype=float32)
Read-only memmap:
>>> fpi = np.memmap(filename, dtype='float32', mode='r', shape=(4,4))
>>> fpi.flags.writeable
Output:
False
Copy-on-write memmap:
>>> fpr = np.memmap(filename, dtype='float32', mode='c', shape=(4,4))
>>> fpr.flags.writeable
Output:
True
It’s possible to assign to copy-on-write array, but values are only written into the memory copy of the array, and not written to disk:
>>> fpr
Output:
memmap([[ 0., 1., 2., 3.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.], [12., 13., 14., 15.]], dtype=float32)
>>> fpr[0,:] = 0
>>> fpr
Output:
memmap([[ 0., 0., 0., 0.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.]], dtype=float32)
File on disk is unchanged:
>>> fpr
Output:
memmap([[ 0., 0., 0., 0.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.], [12., 13., 14., 15.]], dtype=float32)
Offset into a memmap:
>>> fp1 = np.memmap(filename, dtype='float32', mode='r', offset=16)
>>> fp1
Output:
memmap([ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15.], dtype=float32)
Python - NumPy Code Editor:
Previous: format_float_scientific() function
Next: set_printoptions() function
It will be nice if you may share this link in any developer community or anywhere else, from where other developers may find this content. Thanks.
https://www.w3resource.com/numpy/input-and-output/memmap.php
- Weekly Trends and Language Statistics
- Weekly Trends and Language Statistics