Skip to main content
Version: 0.5

Creating & Managing Features

Defining feature pipelines​

What programming languages do you support for defining features?​

For each feature in Tecton, you will create a python-based feature definition file that includes all of the metadata and logic you want Tecton to manage for you.

Tecton's transformation logic is managed in Spark, using PySpark or SQL transformations. If your model requires request-time transformations, those are managed in python.

See Feature Views for more details.

What data types are supported for feature values?​

Tecton supports the following feature data types:

  • Int64: LongType in Tecton on Spark (Databricks or EMR). NUMBER in Tecton on Snowflake.
  • Float64: DoubleType in Tecton on Spark (Databricks or EMR). FLOAT in Tecton on Snowflake.
  • String: StringType in Tecton on Spark (Databricks or EMR). STRING in Tecton on Snowflake.
  • Bool: BooleanType in Tecton on Spark (Databricks or EMR). BOOLEAN in Tecton on Snowflake. Bool is not supported as an element of an Array.

In Tecton on Spark (Databricks or EMR), the following data types are also supported:

  • Array with Int64, Float32, Float64, and String elements.
  • Struct which can contained named fields of any other feature types.
    • Structs are only supported as On-Demand Feature View outputs.
    • See this documentation for usage examples.

Feature materialization and lineage​

What happens when the definition of a feature changes?​

If a feature's definition changes, Tecton automatically detects all the dependencies on that feature, surfaces that for you, and asks you to confirm if you want to go forward with the changes. If you would like to roll back the changes or see the feature lineage, these definitions are backed by git. So you can always track the state of the world of your feature store, at all times.

How far back does Tecton support time-travel?​

You set your features' backfill start date in Tecton. Time-travel can be performed as far back as feature data exists.

What support do you provide for time travel?​

Tecton performs time travel on a row level basis - our granularity of time travel can be quite specific. If you have event driven ML models where you're regularly making predictions and you need to go back to every single specific point in time, and get the feature values as of that point in time, Tecton will handle that full travel query as opposed to just being able to get all feature values at a single point in time.

Does Tecton provide the functionality to replay and fix a backfill if the underlying data source is updated?​

Yes, it is possible to kick off an "overwrite backfill" for a particular time range through the Tecton UI.

When scheduling materializations, does Tecton only materialize new data? Or does Tecton re-materialize all data?​

Generally speaking, Tecton only reads and computes new data. There may be instances in which more historical data is required (eg, computing a one month average at materialization time requires knowing the full window of information).

What does Tecton do for data lineage? Does it support the entire data flow?​

For data lineage, we consider both how features are created and how they are consumed. For feature creation, we show you the entire data flow - from your raw data sources, to the different transformations being ran, to where the data is being stored. For feature consumption, we have concept of a FeatureService which maps to the features for a model that is running. For any feature, you can see which services are using it and, likewise for any service, what are all the features that are inside of it - there is bidirectional tracking.

Does Tecton have an Airflow or Prefect integration?​

Tecton has open-sourced an Airflow provider for coordinating orchestration between Tecton and upstream or downstream pipelines.

If you don't use Airflow, you can implement similar functionality with the Tecton SDK.

Sharing Features​

Can users inspect features?​

Both the Tecton SDK and Web UI enable teams to inspect existing features in their feature store. They can review the actual code that produces the feature, see the status of materialization jobs, or query the actual feature data.

Can users register and discover features in Tecton?​

Yes, with Tecton, you register the entire transformation logic, plus metadata around owners, custom tags, and more. The Tecton Web UI then allows users to access, browse and discover different features.

How can users ensure there are no duplicate features ingested?​

The Tecton Feature Store manages feature dependencies through the names of the objects that are configured for Tecton (eg, data sources, feature views, and services). It is possible to have users submit similar features with different names; we would recommend users first look to reuse features that exist in the feature store.

Handling Nulls​

Does Tecton support null feature values?​

Yes. Tecton supports nulls for feature values and for request data fields. Null values may be returned when data is missing (e.g. for a brand new user), when a materialized feature view column computes null (e.g. SELECT NULLIF(a, b)), or when an On-Demand Feature View returns None.

Nulls may also be members of arrays, e.g. ["foo", null], or members of structs.

Numeric null inputs in Spark Pandas On-Demand Feature Views​

info

If you expect to use numeric nulls, Python mode (mode="python") is strongly recommended.

Spark offline Pandas-mode (mode="pandas") On-Demand Feature Views inputs have special handling for numeric (i.e. Integer or Float) null values; numeric null inputs are cast to NaN.

By contrast, when running online (i.e. serving a production HTTP request), in Python mode, or on Tecton on Snowflake, numeric nulls are provided as None like all other data types.

See the following example given the request input to this feature view are provided as {input_int: null, input_float: null}:

from tecton import RequestSource, on_demand_feature_view
from tecton.types import Int64, Float64, Field

request = RequestSource([Field("input_int", Int64), Field("input_float", Float64)])


@on_demand_feature_view(
    sources=[request],
    mode="pandas",
    schema=[Field("output_int", Int64), Field("output_float", Float64)],
)
def numeric_null_example(request_df):
    import pandas

    print(request_df["input_int"][0])  # `nan` in Spark offline. `None` in all other cases.
    print(request_df["input_float"][0])  # `nan` in Spark offline. `None` in all other cases.

    # `None` values in the output features are correctly handled as `null` in all cases.
    return pandas.DataFrame.from_records([{"output_int": None, "output_float": None}])

The ttl (time-to-live) parameter in Feature Views​

The value of ttl affects the availability of feature data in the online store, the generation of training feature data, and the deletion of feature values from the online store.

ttl is a Batch and Stream Feature View parameter, as well as a Feature Table parameter.

For more information, refer to this page.

How can users delete feature values from the offline store?​

To delete values from the offline store, use the delete_keys() method.

Was this page helpful?

Happy React is loading...