Retrying and Canceling Materialization

Automatic Retry Strategy

Tecton will automatically retry failing jobs with different policies based on the failure:

• Failures due to expected reasons such as an AWS spot instances being reclaimed are retried immediately.
• Otherwise, the job will be retried after 5 minutes, with exponential backoff for each successive failure.

Tecton will not retry jobs that fail for reasons that are expected to be permanent and that won't disappear by retrying.

Jobs that will be retried are shown in the Web UI as Retrying (in X minutes).

If a job fails too many times, auto-retries will stop and jobs will be labeled Failed in the Web UI. This happens after 100 attempts for spot instance-related failures or 10 attempts for other failures.

Manually Retrying Failed Jobs

Materialized Jobs that fail repeatedly will eventually stop retrying. At this stage they can be manually retried. When you are ready to retry the failed job (e.g. after fixing the cause of the failures), trigger a manual retry of the job by clicking the Retry button in the Actions column of the failed job:

A notification will appear when the retry has been successfully scheduled. This happens immediately:

A retry job gets scheduled immediately:

If the scheduling of the retry fails, the notification will give you a job ID, and you can contact support with the message:

Manually Re-running Successful Jobs

Although a materialization job may have succeeded, you may run into scenarios where you need to re-run a specific time range for your feature view.

note

When re-running a materialization job that completed successfully, it’s possible to produce incorrect results in the Online Store. However, this operation is generally safe if one of the following are true:

• Your previous job completed and did not output any feature data; or

• Your Feature View is only materialized offline. (The Feature View is configured with offline=True and online=False).

If you are unsure of the impact of re-running a materialization job that completed successfully, please contact Tecton Support before proceeding.

Trigger a re-run of the job by clicking the Overwrite button in the Actions column of the Success job:

A re-run job gets scheduled immediately:

Cancelling Running Jobs

You may cancel a Running job or a job that's been continuously retrying by clicking the Cancel button in the Actions column of the job.

The job gets marked Manual Cancellation Requested immediately:

Within minutes the jobs will transition to Manually Cancelled. You may need to refresh the page to see this change. At this point, you will have the option to manually Retry the job.