tecton.declarative.data_source.SparkStreamConfig¶

class tecton.declarative.data_source.SparkStreamConfig(data_source_function)¶

Configuration used to define a stream source using a Data Source Function.

The SparkStreamConfig class is used to configure a stream source using a user defined Data Source Function.

This class is used as an input to a StreamSource’s parameter stream_config. This class is not a Tecton Object: it is a grouping of parameters. Declaring this class alone will not register a data source. Instead, declare as part of StreamSource that takes this configuration class instance as a parameter.

Do not instantiate this class directly. Use tecton.declarative.spark_stream_config instead.

Methods

__init__

Instantiates a new SparkBatchConfig.

__init__(data_source_function)¶

Instantiates a new SparkBatchConfig.

Parameters: data_source_function (Callable[[SparkSession], DataFrame]) – User-defined Data Source Function that takes in a SparkSession and returns a streaming DataFrame.
Returns: A SparkStreamConfig class instance.