Effective governance of data catalogs requires more than simple queries; automated dashboards are essential for data stewards and compliance teams. These dashboards help identify issues such as undocumented assets, missing ownership, and outdated metadata.
This article builds upon previous insights into querying Amazon SageMaker Catalog metadata using SQL, demonstrating how to create governance dashboards with Amazon Quick.
Introduction to Amazon Quick
Amazon Quick is a powerful digital workspace that integrates analytics and automation. With Amazon Quick Sight, users can develop interactive dashboards and visualizations, benefiting from automatic chart suggestions and machine learning insights.
Connecting Amazon Quick Sight to SageMaker Catalog
To visualize catalog health metrics, users must connect Amazon Quick Sight to their Athena metadata tables. The Amazon Quick Sight service role requires permissions to access Amazon S3 Tables and the AWS Glue catalog. This can be achieved by adding an inline policy in the IAM console.
Additionally, both the Amazon Quick Sight service role and the admin user need AWS Lake Formation permissions on the S3 Tables catalog. The full S3 Tables catalog identifier must be selected in the Lake Formation console to access the asset_metadata database.
Creating Datasets and Analyzing Data
To access S3 Tables data, users can create a Quick Sight dataset using an Amazon Athena data source and the custom SQL option. A sample SQL command to query the data is:
SELECT * FROM "s3tablescatalog/aws-sagemaker-catalog".asset_metadata.assetBuilding Dashboards with Natural Language Prompts
Amazon Quick allows users to create governance dashboards using natural language prompts, eliminating the need for manual configuration. This method is faster and more intuitive than traditional dashboard creation.
To start building a dashboard, follow these steps:
- Create each visualization using natural language prompts.
- For each recommended visualization, enter the prompt, select Build, and then choose ADD TO ANALYSIS.
Recommended Visualizations
- Asset Inventory by Type:
Show count of asset_id by resource_type_enum as a pie chart - Documentation Completeness:
Show count of asset_id where business_description is not null as a KPI - Monthly Registration Trends:
Show count of asset_id by asset_created_time month as a line chart - Asset Count by Account:
Show count of asset_id by account_id as a bar chart - Namespace Distribution:
Show count of asset_id by namespace as a treemap - Resource Type by Namespace:
Show count of asset_id by resource_type_enum and namespace as a heat map
Automated Governance Reports
Once dashboards are published, users can query governance data, with Amazon Quick providing insights and visualizations. Automated governance reports can be generated, summarizing key metrics such as:
- Total asset counts and growth trends
- Documentation completeness metrics
- Ownership coverage statistics
- Classification distribution analysis
Sharing and Managing Access
Governance dashboards include critical metadata like ownership and classification details. Access should be restricted to authorized users. In the Amazon Quick Sight console, users can manage sharing settings to grant access to specific individuals or groups.
Conclusion
This guide illustrates how to connect Amazon Quick Sight to Amazon SageMaker Catalog metadata and build governance dashboards. This approach enhances visibility into catalog health through various visualizations that monitor asset inventory, documentation completeness, registration trends, and more.
For further exploration of Amazon SageMaker Catalogs, consult the official documentation. For more information on Amazon Quick, the Amazon Quick Sight documentation is available for review.
