Analytics

AWS Glue

Focused AWS Solutions Architect Associate notes from the Analytics domain.

AWS Glue#

AWS Glue Overview#

  • Definition: AWS Glue is a serverless data integration service that automates the process of discovering, cataloging, cleaning, transforming, and moving data between data stores, enabling ETL workflows and data lake management.
  • Key Features:
    • Data Catalog: Centralized metadata repository for databases, tables, and schemas.
    • Crawlers: Automatically discover data schemas in S3, RDS, DynamoDB, and other sources.
    • ETL Jobs: Generate or customize Python/Scala code (Spark-based) for data transformation.
    • Serverless: Auto-scales compute resources (Data Processing Units, DPUs) without infrastructure management.
    • Integrates with S3, Athena, Redshift, Lake Formation, and more for data lakes and analytics.
    • Supports scheduling, orchestration, and monitoring via CloudWatch and CloudTrail.
  • Use Cases: Build data lakes, perform ETL for analytics, unify data from disparate sources, enable data discovery for Athena/Redshift.
  • Key Updates (2024–2025):
    • Enhanced ETL Performance: Optimized Spark engine for faster jobs (October 2024).
    • Lake Formation Integration: Improved fine-grained access control (January 2025).
    • FIPS 140-2 Compliance: Enhanced for GovCloud (October 2024).
    • Security Hub Integration: Compliance monitoring for catalog and jobs (January 2025).

1. AWS Glue Core Concepts#

Components#

  • Data Catalog:
    • Metadata store for databases, tables, partitions, and schemas.
    • Used by Glue ETL, Athena, EMR, and Redshift Spectrum.
    • Explanation: E.g., catalog table sales_data for S3 data.
  • Crawlers:
    • Automated processes to scan data sources (e.g., S3, RDS) and infer schemas.
    • Populates Data Catalog with metadata.
    • Explanation: E.g., crawler scans s3://sales-data/ to create table.
  • ETL Jobs:
    • Spark-based jobs (Python/Scala) to transform and move data.
    • Auto-generated or custom code via Glue Studio (visual editor).
    • Explanation: E.g., job converts CSV to Parquet and loads to Redshift.
  • Triggers:
    • Schedule or event-based job execution (e.g., on-demand, cron, or job completion).
    • Explanation: E.g., trigger ETL job daily at 2 AM.
  • Workflows:
    • Orchestrates crawlers, jobs, and triggers for complex pipelines.
    • Explanation: E.g., workflow: crawl S3 → transform data → load to Redshift.
  • Connections:
    • Defines access to data stores (e.g., RDS, Redshift, JDBC).
    • Supports VPC endpoints for private access.
    • Explanation: E.g., connect to MySQL RDS for ETL.
  • Development Endpoints:
    • Environments for testing and debugging ETL scripts.
    • Explanation: E.g., test Python script in Zeppelin notebook.
  • Glue Studio:
    • Visual interface for creating and managing ETL jobs and workflows.
    • Explanation: E.g., drag-and-drop to build ETL pipeline.

Key Concepts#

  • Data Processing Unit (DPU):
    • Unit of compute (4 vCPUs, 16 GB RAM) for crawlers and ETL jobs.
    • Minimum 2 DPUs; scales up for performance.
    • Explanation: E.g., 10 DPUs for large ETL job.
  • Data Formats:
    • Supports CSV, JSON, Parquet, ORC, Avro, and more.
    • Optimized formats (Parquet, ORC) improve performance.
    • Explanation: E.g., Parquet reduces ETL processing time.
  • Partitioning:
    • Organizes S3 data (e.g., year=2025/month=04) for efficient querying.
    • Managed by crawlers or manually in Data Catalog.
    • Explanation: E.g., partition by date for faster Athena queries.
  • Glue Schema Registry:
    • Centralizes schema management for streaming data (e.g., Kinesis, Kafka).
    • Explanation: E.g., enforce schema for Kinesis stream.
  • Serverless Architecture:
    • No infrastructure management; auto-scales DPUs.
    • Explanation: E.g., Glue scales to 20 DPUs for peak load.
  • Security Configurations:
    • Encryption, IAM roles, and VPC settings for jobs and crawlers.
    • Explanation: E.g., KMS encryption for S3 data.

Key Notes:#

  • Exam Relevance: Understand Data Catalog, crawlers, ETL jobs, integrations, and cost optimization.
  • Mastery Tip: Compare Glue vs. EMR vs. Data Pipeline for ETL.

2. Glue Performance Features#

Glue optimizes data integration.

Low Latency#

  • Purpose: Fast ETL and catalog operations.
  • Features:
    • Enhanced Spark engine for faster ETL (2024).
    • Crawlers process metadata in minutes.
    • Serverless startup in seconds.
  • Explanation: E.g., ETL job transforms 1 GB in <1 minute.
  • Exam Tip: Highlight serverless speed for ETL.

High Throughput#

  • Purpose: Handle large datasets.
  • Features:
    • Parallel processing with multiple DPUs.
    • S3 integration for high-throughput data access.
  • Explanation: E.g., process 1 TB with 20 DPUs concurrently.
  • Exam Tip: Use for big data ETL.

Scalability#

  • Purpose: Support growing data lakes.
  • Features:
    • Auto-scales DPUs based on workload.
    • Data Catalog supports millions of tables/partitions.
    • Lake Formation enables cross-account access (2025).
  • Explanation: E.g., scale to 100 DPUs for 10 TB ETL job.
  • Exam Tip: Emphasize serverless scalability.

Key Notes:#

  • Performance: Low latency + high throughput + scalability = efficient ETL.
  • Exam Tip: Optimize with Parquet and DPUs.

3. Glue Resilience Features#

Resilience ensures reliable data integration.

Multi-AZ/Region Redundancy#

  • Purpose: Survive failures.
  • Features:
    • Glue is a Regional service with multi-AZ compute.
    • Data Catalog metadata stored durably.
    • S3 provides 11 9s durability for data.
  • Explanation: E.g., ETL job continues if us-east-1a fails.
  • Exam Tip: Highlight S3 durability for resilience.

Continuous Processing:#

  • Purpose: Uninterrupted ETL.
  • Features:
    • Serverless architecture eliminates downtime.
    • Automatic retries for transient job failures.
  • Explanation: E.g., job retries after network glitch.
  • Exam Tip: Use for 24/7 pipelines.

Monitoring and Recovery:#

  • Purpose: Detect and resolve issues.
  • Features:
    • CloudWatch metrics for job execution (e.g., GlueETLJobDuration).
    • CloudTrail logs Glue API calls (e.g., RunJob).
    • Security Hub detects misconfigured jobs/catalogs (2025).
    • Workflow logs track pipeline failures.
  • Explanation: E.g., alarm on high GlueETLJobErrors.
  • Exam Tip: Use CloudWatch and CloudTrail for monitoring.

Data Durability:#

  • Purpose: Protect data and metadata.
  • Features:
    • S3 for persistent data storage.
    • Data Catalog metadata backed up across AZs.
  • Explanation: E.g., recover input data from S3 after job failure.
  • Exam Tip: Highlight S3 and Catalog resilience.

Key Notes:#

  • Resilience: Multi-AZ + serverless + monitoring + S3 = reliable ETL.
  • Exam Tip: Design resilient pipelines with S3 and workflows.

4. Glue Security Features#

Security is a core focus for Glue in SAA-C03.

Access Control#

  • IAM Policies:
    • Restrict Glue actions (glue:CreateJob, glue:GetTable).
    • Scope to catalogs, jobs, or S3 buckets.
    • Example: {“Effect”: “Allow”, “Action”: “glue:RunJob”, “Resource”: “arn:aws:glue:::job/sales-etl”}.
  • Lake Formation:
    • Fine-grained access (row/column-level) for Data Catalog (2025).
    • Cross-account data sharing.
    • Explanation: E.g., restrict sales_data to analysts.
  • Resource Policies:
    • Control access to Data Catalog resources.
    • Explanation: E.g., limit catalog access to specific IAM roles.
  • Exam Tip: Practice IAM and Lake Formation policies.

Encryption#

  • In Transit:
    • HTTPS for API calls and data transfer.
    • Explanation: E.g., secure RunJob call.
  • At Rest:
    • S3: SSE-S3, SSE-KMS, or CSE-KMS.
    • Data Catalog: KMS encryption for metadata.
    • Job bookmarks: KMS encryption.
    • Explanation: E.g., KMS-encrypted S3 output.
  • Exam Tip: Highlight KMS for compliance.

Compliance:#

  • Purpose: Meet regulatory standards.
  • Features:
    • Supports HIPAA, PCI, SOC, ISO, GDPR, FIPS 140-2 (GovCloud).
    • Lake Formation ensures compliant data access (2025).
    • Security Hub detects non-compliant configurations (2025).
  • Explanation: E.g., process HIPAA-compliant data in S3.
  • Exam Tip: Use Lake Formation for compliance.

Auditing:#

  • Purpose: Track Glue activity.
  • Features:
    • CloudTrail logs API calls.
    • CloudWatch Logs for job execution details.
    • Security Hub monitors compliance (2025).
  • Explanation: E.g., audit CreateCrawler for unauthorized access.
  • Exam Tip: Use CloudTrail and CloudWatch for auditing.

Key Notes:#

  • Security: IAM + Lake Formation + encryption + auditing = secure ETL.
  • Exam Tip: Configure Lake Formation, KMS, and CloudTrail for secure Glue.

5. Glue Cost Optimization#

Cost efficiency is a key exam domain.

Pricing#

  • Crawlers: $0.44/DPU-hour, minimum 10 minutes.
  • ETL Jobs: $0.44/DPU-hour, minimum 10 minutes.
  • Data Catalog:
    • Storage: $1/100 objects/month (first 1M free).
    • Requests: $1/100K requests.
  • Other Costs:
    • S3: $0.023/GB/month.
    • Development Endpoints: $0.44/DPU-hour.
  • Example:
    • ETL job: 10 DPUs, 1 hour/day, 30 days.
    • Crawler: 2 DPUs, 10 min/day, 30 days.
    • Catalog: 1K objects, 10K requests.
    • S3: 1 TB storage.
      • ETL: 10 × $0.44 × 30 = $132.
      • Crawler: 2 × $0.44 × (10/60) × 30 = $4.40.
      • Catalog Storage: 1K × $1/100K = $0.01.
      • Catalog Requests: 10K × $1/100K = $0.10.
      • S3: 1,000 GB × $0.023 = $23.
      • Total: $132 + $4.40 + $0.01 + $0.10 + $23 = ~$159.51/month.
  • Free Tier: 1M catalog objects, 1M requests/month.

Cost Strategies#

  • Optimize DPU Usage:
    • Use minimal DPUs for small jobs; scale for large jobs.
    • Explanation: E.g., reduce from 10 to 5 DPUs, saving $66/month.
  • Shorten Job Duration:
    • Optimize Spark code, use Parquet/ORC.
    • Explanation: E.g., cut job from 1 hour to 30 min, saving $66/month.
  • Schedule Crawlers Efficiently:
    • Run crawlers only when data changes.
    • Explanation: E.g., reduce daily to weekly, saving $3.70/month.
  • Compress Data:
    • Use GZIP/Snappy for S3 data to reduce storage.
    • Explanation: E.g., compress 1 TB to 200 GB, saving $18.40/month.
  • Limit Catalog Objects:
    • Consolidate tables/partitions to stay within free tier.
    • Explanation: E.g., reduce objects to save $1/month.
  • Tagging:
    • Tag jobs, crawlers, and S3 buckets for cost tracking.
    • Explanation: E.g., tag job with “Project:Analytics”.
  • Monitor Usage:
    • Use CloudWatch and Cost Explorer to optimize DPU and job runtime.
    • Explanation: E.g., optimize jobs to save $50/month.

Key Notes:#

  • Cost Savings: Optimize DPUs + Parquet + scheduling + tagging = lower costs.
  • Exam Tip: Calculate DPU costs and optimize with Parquet.

6. Glue Advanced Features#

Enhanced ETL Performance:#

  • Purpose: Faster data processing.
  • Features:
    • Optimized Spark engine for ETL jobs (2024).
    • Explanation: E.g., 2x faster CSV-to-Parquet conversion.
  • Exam Tip: Know for high-performance ETL.

Lake Formation Integration:#

  • Purpose: Secure data lakes.
  • Features:
    • Row/column-level access, cross-account sharing (2025).
    • Explanation: E.g., restrict sales_data to specific columns.
  • Exam Tip: Use for compliance.

Glue Schema Registry:#

  • Purpose: Manage streaming schemas.
  • Features:
    • Enforces schemas for Kinesis/Kafka streams.
    • Explanation: E.g., validate JSON schema for Kinesis.
  • Exam Tip: Know for streaming ETL.

Security Hub Integration:#

  • Purpose: Compliance monitoring.
  • Features:
    • Detects misconfigured jobs/catalogs (2025).
    • Explanation: E.g., flag unencrypted S3 output.
  • Exam Tip: Use for compliance.

Glue Studio:#

  • Purpose: Simplify ETL development.
  • Features:
    • Visual drag-and-drop for jobs and workflows.
    • Explanation: E.g., build ETL pipeline without coding.
  • Exam Tip: Know for ease of use.

Key Notes:#

  • Flexibility: Lake Formation + Schema Registry + Studio = advanced ETL.
  • Exam Tip: Master Lake Formation and Glue Studio.

7. Glue Use Cases#

Understand practical applications.

Data Lake Creation#

  • Setup: Crawlers for S3, ETL jobs for transformation.
  • Features: Catalog metadata, convert to Parquet.
  • Explanation: E.g., build lake from raw CSV data.

ETL for Analytics#

  • Setup: Jobs to transform and load to Redshift/Athena.
  • Features: Spark-based processing, scheduling.
  • Explanation: E.g., load sales data to Redshift daily.

Streaming ETL#

  • Setup: Glue Schema Registry with Kinesis.
  • Features: Real-time data processing.
  • Explanation: E.g., process clickstream data from Kinesis.

Data Unification#

  • Setup: Connections to RDS, DynamoDB, S3.
  • Features: Combine disparate data sources.
  • Explanation: E.g., join RDS customer data with S3 logs.

8. Glue vs. Other ETL Services#

FeatureGlueEMRData Pipeline
TypeServerless ETLManaged Big DataOrchestration
FocusETL, Data CatalogSpark, HadoopPipeline scheduling
ComputeServerless DPUsEC2 clustersEC2, Lambda
Cost$0.44/DPU-hourEC2 + EMR fee$1–$2.50/pipeline/month
Use CaseData lake ETLComplex big dataLegacy pipelines

Explanation:#

  • Glue: Serverless ETL and catalog for data lakes.
  • EMR: Flexible big data with multiple frameworks.
  • Data Pipeline: Legacy orchestration for simple pipelines.

9. Detailed Explanations for Mastery#

  • Enhanced ETL Performance:
    • Example: Faster CSV-to-Parquet job.
    • Why It Matters: Scalable ETL (2024).
  • Lake Formation Integration:
    • Example: Secure catalog with row-level access.
    • Why It Matters: Compliant data lakes (2025).
  • Security Hub Integration:
    • Example: Flag unencrypted job output.
    • Why It Matters: Compliance monitoring (2025).

10. Quick Reference Table#

FeaturePurposeKey DetailExam Relevance
Data CatalogMetadata managementDatabases, tables, schemasCore Concept
CrawlersSchema discoveryScans S3, RDS, etc.Core Concept
ETL JobsData transformationSpark-based, Python/ScalaCore Concept
Lake FormationSecure data lakesRow/column access (2025)Security
Glue StudioVisual ETL developmentDrag-and-drop interfaceFlexibility
Security HubCompliance monitoringMisconfigured jobs (2025)Security, Resilience
PartitioningOptimize queriesS3 data by keysCost, Performance

Found an issue? Improve this note in the GitHub repo.