Nested Filestore

This is a simple filestore that stores files in a nested directory structure.

Install

pip install "git+https://github.com/0xidm/nested-filestore"

Usage

The following example creates a nested filestore in /tmp/files with 2 nested levels. The leaf nodes will be grouped into containers containing 10^2 files each. There will be subdirectories of 10^1 containers each. Finally, the root node will be grouped into containers containing 10^1 files each.

filestore = NestedFilestore("/tmp/files", [2, 1, 1])
filestore.put(0, "tests/data/12345678.bin")
assert filestore.exists(0)
with filestore.get(0) as f:
   print(f.read())

Online resources

API

NestedFilestore

class nested_filestore.NestedFilestore(root_path, hierarchy_order, pad_character='0', base=10)[source]

Bases: object

NestedFilestore is a filestore that stores files in a nested directory structure. This is a thin wrapper around the Index class.

exists(identifier)[source]: does the specified identifier exist as a file? If it exists, return True

get(identifier)[source]: given an identifier, return a file handle pointing to the file if it exists

ingest_filesystem(filestore_path)[source]: low-level filesystem scan of filestore_path for .bin files, which it imports

put(identifier, filename=None, filehandle=False, move=False, overwrite=False)[source]: given the path to an existing file, and given an identifier, copy the file to the file store and put it in the right place, creating directories as needed.

writer(identifier, overwrite=False)[source]: given an identifier, return a writable file handle pointing to the file UNLESS it exists