Collection.read_delta#

classmethod Collection.read_delta(
source: str | Path | deltalake.DeltaTable,
*,
validation: Validation = 'warn',
**kwargs: Any,
) → Self[source]#

Read all collection members from Delta Lake tables.

This method reads each member from a Delta Lake table at the provided source location. The source can be a path, URI, or an existing DeltaTable object. Optional members are only read if present.

Parameters:

  • source – The location or DeltaTable to read from.

  • validation –

    The strategy for running validation when reading the data:

    • "allow": The method tries to read the schema data from the parquet files. If the stored collection schema matches this collection schema, the collection is read without validation. If the stored schema mismatches this schema no metadata can be found in the parquets, or the files have conflicting metadata, this method automatically runs validate() with cast=True.

    • "warn": The method behaves similarly to "allow". However, it prints a warning if validation is necessary.

    • "forbid": The method never runs validation automatically and only returns if the metadata stores a collection schema that matches this collection.

    • "skip": The method never runs validation and simply reads the data, entrusting the user that the schema is valid. Use this option carefully.

  • kwargs – Additional keyword arguments passed directly to polars.read_delta().

Returns:

The initialized collection.

Raises:

  • ValidationRequiredError – If no collection schema can be read from the source and validation is set to "forbid".

  • ValueError – If the provided source does not contain Delta tables for all required members.

  • ValidationError – If the collection cannot be validated.

Attention

Schema metadata is stored as custom commit metadata. Only the schema information from the last commit is used, so any table modifications that are not through dataframely will result in losing the metadata.

Be aware that appending to an existing table via mode=”append” may result in violation of group constraints that dataframely cannot catch without re-validating. Only use appends if you are certain that they do not break your schema.

Be aware that this method suffers from the same limitations as serialize().