Connecting to a Databricks Data Source

Overview

Tecton can use Databricks as a source of batch data (via Amazon S3) for feature materialization, provided that Databricks uses the AWS Glue Data Catalog as the Hive metastore. This page explains how to set up Tecton to use Databricks as a data source.

Limitations

Hive datastores have the following limitations:

• You cannot dynamically switch between different Glue Data Catalogs. You must restart the cluster for new Spark configurations to take effect.
• For other considerations when using Glue Data Catalogs, see the AWS Spark documentation.

Configure access to the Glue Data Catalog

info

The instructions that follow apply only to cases where Tecton needs access to a Glue Data Catalog from an account other than the Tecton data plane account.

Prerequisites

Before you begin, you must have:

• A deployed Tecton cluster
• AWS administrator access to IAM roles and policies for the Tecton deployment AWS account.
• A Target Glue Data Catalog ID.
• AWS administrator access to IAM roles and policies for the Glue Data Catalog AWS account.

Procedure

To configure access to the Glue Data Catalog, follow these steps:

Grant Glue Data Catalog Access to Tecton's Role

Log in to the AWS account of the target Glue Data Catalog and go to the Glue Console. In Settings, paste the following policy into the Permissions box. Set the "AWS" ARN to the Spark role (the instance profile role) used by Databricks. For "Resource", set the <aws-region-target-glue-catalog> and <aws-account-id-target-glue-catalog>.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Example permission",
      "Effect": "Allow",
      "Principal": {
        "AWS": <TECTON ROLE ARN>
      },
      "Action": [
        "glue:BatchGetPartition",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetUserDefinedFunction",
        "glue:GetUserDefinedFunctions"
      ],
      "Resource": "arn:aws:glue:<aws-region-target-glue-catalog>:<aws-account-id-target-glue-catalog>:*"
    }
  ]
}

Grant IAM Permissions to the Spark Role

Log in to your Tecton AWS account and go to the Iam Console. Create the following policy and attach it to your tecton spark role (created in Databricks Setup).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GlueAccess",
            "Effect": "Allow",
            "Action": [
                "glue:BatchGetPartition",
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetPartition",
                "glue:GetPartitions",
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetUserDefinedFunction",
                "glue:GetUserDefinedFunctions",
            ],
            "Resource": "arn:aws:glue:<aws-region-target-glue-catalog>:<aws-account-id-target-glue-catalog>:*",
        }
    ],
}

Specify the Target Glue Data Catalog ID

If the target Glue Data Catalog region is the same as the AWS region, you can set the aws-account-id-target-glue-catalog, as specified in the previous step, in the Web UI; navigate to <cluster name>.tecton.ai/app/data-platform/ and set the value in the AWS Glue Data Catalog id field.

Otherwise, Tecton needs the aws-account-id-target-glue-catalog and the aws-region-target-glue-catalog values that you set in the previous step. Your deployment specialist will request these values.

Validate Permissions

The preceding steps set up the Glue Data Catalog policies, which grant Tecton access only to the metadata. The related S3 bucket and object-level access permissions are defined separately by S3, and can be more restrictive if required. For more information, see this AWS blog post.

To validate the cross-account permissions with Databricks, launch a cluster with the *-spark-node instance profile:

1. Create a cluster.

2. Click the Instances tab on the Cluster Creation page.

3. In the Instance Profiles drop-down list, select the instance profile.

4. Set the Spark configuration spark.databricks.hive.metastore.glueCatalog.enabled=true

5. If the target Glue Data Catalog is in a different AWS account from the Databricks deployment, set the Spark configuration spark.hadoop.hive.metastore.glue.catalogid to the target Glue Data Catalog id.

6. If the target Glue Data Catalog is in a different region from the Databricks deployment, set the Spark configuration spark.hadoop.aws.region to the target region.

7. Verify that you can access the Glue Data Catalog by running the following command in a notebook:

   show databases;

   If the command succeeds, the Tecton cluster is configured to use Glue.