Connecting CI/CD

Overview

We recommend creating a new CI/CD pipeline to automatically run tecton plan and tecton apply upon changes to your Tecton feature repo.

Example

For example, in GitHub actions, you can add the following:

name: Tecton Feature Repo CI/CD

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      TECTON_API_KEY: ${{ secrets.TECTON_API_KEY }}
      API_SERVICE: https://<YOUR CLUSTER SUBDOMAIN>.tecton.ai/api
      FEATURE_REPO_DIR: ./feature_repo
      WORKSPACE: <Your-workspace>
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up Python 3.7.10
        uses: actions/setup-python@v2
        with:
          python-version: 3.7.10

      - name:
          Install pypandoc (for pyspark bug
          https://stackoverflow.com/questions/51500288/unable-to-install-pyspark)
        run: pip install pypandoc

      - name: Install the Tecton CLI
        run: pip install --no-cache-dir tecton

      - name: Select the workspace
        run: tecton workspace select ${WORKSPACE}

      - name: Run tecton plan
        run: cd ${FEATURE_REPO_DIR} && tecton plan --no-safety-check

      - name: Run tecton apply
        if: ${{ github.event_name == 'push' }}
        run: cd ${FEATURE_REPO_DIR} && tecton apply --no-safety-check

note

If you are using Spark and you will be running unit tests, replace pip install --no-cache-dir tecton, as specified above, with one of the following commands:

• To install with Pyspark 3.1: pip install --no-cache-dir 'tecton[pyspark]'
• To install with Pyspark 3.2: pip install --no-cache-dir 'tecton[pyspark3.2]'
• To install with Pyspark 3.3: pip install --no-cache-dir tecton pyspark==3.3

To authenticate your Github Action, you'll need to create a Service Account to obtain a Tecton API key, and assign the Service Account the Editor role for the appropriate workspace:

1. Create the Service Account to obtain the API key.

tecton service-account create \
      --name "cicd-service-account" \
      --description "A cicd example"

Output:

Save this API Key - you will not be able to get it again.
API Key:            <Your-api-key>
Service Account ID: <Your-Service-Account-Id>

2. Assign the Editor role for the workspace to the Service Account.

tecton access-control assign-role --role editor \
      --workspace <Your-workspace> \
      --service-account <Your-Service-Account-Id>

Output:

Successfully updated role.

Validate Plan with JSON output

You can output a JSON version of a to-be-applied diff using the --json-out flag.

tecton plan --json-out <path>

This can be useful in a CI/CD pipeline to prevent applying unintended changes by running a custom script on the output.

Example json file output:

{
  "objectDiffs": [
    {
      "transitionType": "DELETE",
      "objectMetadata": {
        "name": "transaction_user_has_good_credit",
        "objectType": "FEATURE_VIEW",
        "owner": "john@doe.com",
        "description": "Whether the user had a good credit score (over 670) as of the time of a transaction."
      }
    },
    {
      "transitionType": "RECREATE",
      "objectMetadata": {
        "name": "continuous_feature_service",
        "objectType": "FEATURE_SERVICE",
        "owner": "john@doe.com",
        "description": "A FeatureService providing continuous features."
      }
    }
  ]
}

See Types of Repository Changes doc to help understand the plan output.

Apply Generated Plan

When a plan is successfully generated with tecton plan, an ID for that plan is printed to the console after the plan contents.

...
↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
Generated plan ID is a25e9516ebde475690ef3806e1f12e1e

If tecton plan was run with the --json-out flag, the plan ID is also included as a field in the JSON file:

{
  "objectDiffs": [
    ...
  ],
  "planId": "a25e9516ebde475690ef3806e1f12e1e"
}

After this plan is approved through your team's workflow (whether automated or manual), you can directly apply the plan by passing the plan ID through the --plan-id parameter:

tecton apply --plan-id=a25e9516ebde475690ef3806e1f12e1e

This will apply the plan directly without recomputing a new plan.

If any changes have been made to the feature repo since the plan was generated (i.e. someone ran tecton apply), then you will get an error and must generate a new plan on top of the current repo state.

Protecting Critical Objects from Destruction

Your repo may have critical Tecton objects that you would like to prevent from being destroyed, for example a large feature view which would be costly to rematerialize. It is possible that future state updates may accidentally delete the feature view, or trigger a destructive update (e.g. if a data source for the feature view is updated).

To protect your feature view and other critical objects from unintentionally being destroyed or recreated, you can set the prevent_destroy parameter for Feature Views or Feature Services.

For example, this Feature View is protected with the prevent_destroy parameter:

Spark

@batch_feature_view(
    sources=[FilteredSource(transactions_batch)],
    entities=[user],
    mode="spark_sql",
    online=True,
    offline=True,
    feature_start_time=datetime(2021, 5, 20),
    batch_schedule=timedelta(days=1),
    ttl=timedelta(days=30),
    description="Last user transaction amount (batch calculated)",
    prevent_destroy=True,
)
def critical_feature_view(transactions):
    pass

Snowflake

@batch_feature_view(
    sources=[FilteredSource(transactions_batch)],
    entities=[user],
    mode="snowflake_sql",
    online=True,
    offline=True,
    feature_start_time=datetime(2021, 5, 20),
    batch_schedule=timedelta(days=1),
    ttl=timedelta(days=30),
    description="Last user transaction amount (batch calculated)",
    prevent_destroy=True,
)
def critical_feature_view(transactions):
    pass

Note that the prevent_destroy has no effect in development workspaces, as there are no processes to protect from destruction.

Destroying protected objects

If at some point in the future, you want to destroy or recreate this object, this must be done in 2 steps:

1. Remove the prevent_destroy parameter from the object or set it to False, and apply this repo update. Do not combine this change with any other changes.
2. The object is unprotected at this point, so you can apply any destructive updates as you normally would.

Attempting to do the 2 steps above in a single step and running apply will return an error.