Amazon Managed Service for Apache Flink — Stream Processing

Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics for Apache Flink) is the AWS managed runtime for stateful stream processing, and on the DEA-C01 exam it is the highest-yield service for Task 1.2 (Transform and process data) in the streaming context. Community study guides — Tutorials Dojo, ExamCert.App, and the popular Medium walkthroughs — all flag the same pain point: candidates from a batch background underestimate stream processing depth, especially the windowing semantics, watermark handling for late-arriving data, and the checkpointing requirements for fault-tolerant stateful processing. The exam plants scenarios where the wrong answer is "Lambda" because Lambda cannot maintain state across invocations, or "Kinesis Data Firehose" because Firehose cannot do windowed aggregations. Getting Managed Flink right means understanding the programming models, windowing types, state management, and the canonical traps cold.

This study note is built for the data engineer perspective. It covers what Managed Flink is and why stateful stream processing exists, the three programming models (DataStream API, Table API, Flink SQL), the four window types (tumbling, sliding, session, global), stateful processing with keyed state and operator state, checkpointing and savepoints for fault tolerance, sources and sinks (KDS, MSK, S3, Redshift, OpenSearch), parallelism and KPU scaling, late-arriving data and watermarks, Studio notebooks for interactive development, the cost model, common integration patterns, and the exam traps that catch most candidates. By the end the stream processing decision should be a structured choice you can defend in any architecture review or DEA-C01 scenario.

What Is Amazon Managed Service for Apache Flink?

Apache Flink is the open-source distributed processing engine purpose-built for stateful computations over unbounded data streams. Managed Service for Apache Flink runs Flink applications without you provisioning or managing the underlying compute cluster. AWS handles JobManager and TaskManager provisioning, checkpointing storage, automatic scaling, and version upgrades; you write Flink application code (Java, Scala, Python via PyFlink, or SQL) and submit it.

Why a Managed Stream Processing Engine Exists

A streaming pipeline that just writes events to S3 (Firehose) or invokes a Lambda per record (Lambda+Kinesis) is stateless — each record is processed independently. Real analytics requirements quickly demand state: "rolling 5-minute average temperature per sensor," "session-based user behavior aggregation," "fraud score computed from the last 100 transactions per customer." None of these can be answered by stateless processing because the answer depends on prior records.

Flink's value is built-in distributed state management plus event-time processing plus exactly-once semantics. The runtime maintains keyed state (per partition key) and operator state (per task) in a fault-tolerant backend, replays state from checkpoints on failure, and handles late-arriving data with watermarks. Building this from scratch in Lambda or Kinesis Client Library is months of engineering work; Managed Flink gives it as a service.

From Kinesis Data Analytics to Managed Flink

Originally launched as Kinesis Data Analytics with three flavors (SQL, Apache Flink, Apache Beam). AWS deprecated the SQL-on-KDA flavor in 2024 and renamed the Apache Flink flavor to Amazon Managed Service for Apache Flink. Older study guides may reference "Kinesis Data Analytics" — on the DEA-C01 exam treat it as Managed Flink with the new name and feature set.

Apache Flink Programming Models

Flink offers three distinct programming abstractions. Choose by team skill, complexity, and use case.

DataStream API — Lowest-Level, Most Powerful

The DataStream API is Flink's foundational stream processing API in Java/Scala (or PyFlink for Python). You define sources, transformations (map, filter, keyBy, window, aggregate), and sinks programmatically. Use for complex stateful logic, custom windowing, and integration with custom data sources. The trade-off is the most code to write and the highest skill barrier.

Table API — Declarative, Strongly Typed

The Table API is a relational abstraction over the DataStream API. You define tables (logical views over streams), apply select/filter/groupBy/window operations, and Flink translates to optimized DataStream code under the hood. Less code than DataStream API; same expressiveness for most analytical patterns. Good for teams comfortable with relational thinking.

Flink SQL — Pure SQL on Streams

Flink SQL is ANSI SQL extended with streaming primitives (window functions on time, MATCH_RECOGNIZE for pattern detection). You write SQL queries against logical streams; Flink translates to the runtime. Lowest skill barrier for SQL-fluent teams. Use for windowed aggregations, joins, simple pattern detection, and analytics-style transformations. The Studio notebook (Zeppelin-based) makes Flink SQL interactive.

Choosing the Programming Model

Flink SQL: simple aggregations, windowing, joins, analytics-style transformations. Table API: similar use cases with stronger typing and IDE-friendly Java/Scala. DataStream API: complex stateful logic, custom windowing, integration with external systems, ML inference inside the pipeline. The DEA-C01 exam asks scenario questions like "team is SQL-fluent and needs windowed aggregation" (Flink SQL) vs "complex CEP pattern detection with custom state" (DataStream API).

Windowing — The Heart of Stream Processing

Stream processing requires bounding unbounded streams into finite groups for aggregation. Flink supports four window types.

Tumbling Windows — Fixed, Non-Overlapping

Tumbling windows divide time into fixed-size, non-overlapping chunks. A 5-minute tumbling window emits one result per 5-minute block: 00:00-00:05, 00:05-00:10, 00:10-00:15. Each event belongs to exactly one window. Use for periodic aggregations: "transactions per minute," "daily revenue," "hourly active users."

Sliding Windows — Fixed Size with Overlap

Sliding windows have a fixed size and a slide interval; consecutive windows overlap. A 5-minute window sliding every 1 minute emits a result every minute that covers the last 5 minutes: 00:00-00:05, 00:01-00:06, 00:02-00:07. Each event belongs to multiple windows. Use for rolling aggregates: "rolling 5-minute average response time," "moving 10-minute fraud rate."

Session Windows — Activity-Based

Session windows group events by activity gaps. A session ends when no events arrive for a configured timeout (e.g., 30 minutes). Different keys can have different session boundaries. Use for user session analytics: "events per user session," "average session duration," "actions before churn."

Global Windows — Manually Triggered

Global windows assign all events with the same key to a single window with no automatic emission. You attach a custom trigger that decides when to emit. Use for advanced patterns where built-in windows do not fit.

Event-Time vs Processing-Time

Flink supports two notions of time, and the distinction is exam-critical.

Processing-Time — Wall-Clock at the Operator

Processing-time uses the wall-clock time at the Flink operator. Simpler to implement; results depend on operator timing rather than event semantics. A 5-minute tumbling window in processing-time covers events that arrived at the operator in that wall-clock window, regardless of when the events actually happened.

Event-Time — Timestamp Embedded in the Event

Event-time uses a timestamp embedded in each event (extracted by a TimestampAssigner). A 5-minute tumbling window in event-time covers events whose embedded timestamp falls in that range, regardless of when they arrived at the operator. Event-time produces deterministic results across runs and is required when ordering matters or when late-arriving events must be handled correctly.

Why Event-Time Wins for Most Real Use Cases

A user-clickstream pipeline must group "events that happened during user session 14:00-15:00" — not "events the operator received at 14:00-15:00." Network delays, retry storms, or producer outages can delay events by minutes; processing-time would put them in the wrong window. Event-time with watermarks correctly attributes them to the original session.

Watermarks — Handling Late-Arriving Data

Watermarks are Flink's mechanism for declaring "all events with timestamp ≤ X have probably arrived."

What a Watermark Is

A watermark is a special event injected into the stream that signals "no events older than this timestamp will arrive (with high probability)." The Flink runtime uses watermarks to decide when to close event-time windows and emit results.

Bounded Out-of-Orderness Strategy

The most common watermark strategy is bounded out-of-orderness: assume events can arrive up to N seconds late, generate watermarks at (current event timestamp - N seconds). For a clickstream with up to 30 seconds of delay, watermark = current event timestamp - 30 seconds. Windows close when the watermark exceeds the window end.

Late-Arriving Events

Events arriving after the watermark passes their event-time window are "late." Flink offers three handling strategies: drop (default; events are silently discarded), side output (route late events to a separate stream for later analysis), allowed lateness (extend window closing by N seconds before final close). Pick by use case: drop for high-volume non-critical, side output for audit, allowed lateness when you can wait.

Watermark Generation Frequency

Watermarks should be generated frequently enough to keep pipeline latency low (every 200ms is typical) but not so often that overhead dominates. Misconfigured watermarks are the most common Flink bug: too aggressive (close windows before late events arrive, lose data), too lazy (windows never close, results never emit, memory grows unboundedly).

Stateful Stream Processing

Flink's distinguishing feature is built-in distributed state.

Keyed State — Per Partition Key

When you keyBy a stream, Flink partitions records by key and gives each operator instance access to a state backend keyed by that partition. Common state types: ValueState (one value per key, e.g., last seen timestamp), ListState (list of values per key, e.g., recent transactions), MapState (map per key, e.g., feature dictionary), ReducingState/AggregatingState (incrementally aggregated value).

Operator State — Per Task Instance

Operator state is scoped to a task instance, not to a partition key. Common use: source connector tracking which Kinesis shard offsets have been read. Less commonly used in application code than keyed state.

State Backends

Flink stores state in a state backend. Managed Flink defaults to RocksDB (on-disk plus in-memory for hot state), which scales to TBs of state. Memory-only backends are available for small-state workloads with tighter latency requirements.

Checkpoints — Fault-Tolerant State

Flink takes periodic snapshots of all state to a durable store (S3 in Managed Flink). On failure, Flink restarts the application and restores state from the last completed checkpoint. The Flink runtime guarantees exactly-once semantics across the checkpoint boundary as long as sources support replay (KDS, MSK do) and sinks support transactional commit or idempotent writes (S3, Iceberg, Kafka with transactions).

Savepoints — Manual State Snapshots

Savepoints are like checkpoints but explicitly triggered (typically before code deployment). Operators upgrade Flink jobs by stopping with savepoint, deploying new code, and restarting from savepoint — preserving state across the upgrade.

Sources and Sinks

Managed Flink applications read from sources and write to sinks. Connector ecosystem is broad.

Common Sources

Amazon Kinesis Data Streams (built-in connector, KCL-based with EFO support). Amazon MSK (Kafka source connector). Self-managed Kafka (Kafka source connector). Amazon S3 (file source for batch reprocessing or hybrid pipelines). Amazon Kinesis Data Firehose is not a source — Firehose only delivers, never reads.

Common Sinks

Amazon S3 (writes Parquet, ORC, JSON, CSV partitioned by event time). Amazon Kinesis Data Streams (forwards to another stream). Amazon Kinesis Data Firehose (delivery to S3/Redshift/OpenSearch). Amazon Redshift (via Firehose or via custom JDBC sink). Amazon OpenSearch (direct connector or via Firehose). Amazon DynamoDB (DynamoDB sink connector). Apache Iceberg (writes to Iceberg tables on S3 with ACID semantics).

Connector Configuration

Each connector requires configuration in the application code: stream/topic name, deserialization schema, watermark strategy (for sources), serialization schema, batching parameters (for sinks). Common patterns are documented in the AWS Managed Flink GitHub samples.

Parallelism and KPU Scaling

Managed Flink scales by parallelism, billed in KPUs.

What a KPU Is

A Kinesis Processing Unit (KPU) is the billing unit for Managed Flink. One KPU = 1 vCPU + 4 GB memory + a fixed amount of state storage. Pricing is per KPU-hour ($0.11/hr in us-east-1 baseline). Applications use a configurable number of KPUs based on parallelism and memory needs.

Parallelism and Task Slots

Flink applications are partitioned into parallel tasks. Parallelism is the number of parallel instances of each operator. Each KPU provides task slots that host parallel tasks. A higher parallelism processes more events per second but consumes more KPUs.

Auto-Scaling

Managed Flink supports auto-scaling: AWS adjusts parallelism (and thus KPUs) based on observed CPU and memory utilization. Set min and max KPU bounds; the service scales within them. Auto-scaling reacts to sustained load — short bursts may not trigger scale-out.

Sizing Guidelines

Throughput-bound workloads: parallelism roughly equals input partition/shard count. State-bound workloads: more KPUs to fit state in memory. Network-bound workloads (lots of cross-task shuffling): more KPUs to handle inter-task data movement. The DEA-C01 exam asks scenario questions like "Flink application has high CPU utilization" (more parallelism / more KPUs) vs "checkpoint duration is high" (more memory / smaller checkpoint intervals).

Managed Flink vs Self-Managed Flink on EMR

Managed Flink is one of two ways to run Flink on AWS.

Managed Flink Advantages

Zero cluster management. Managed checkpointing storage. Auto-scaling. Simple deployment via Java JAR or Python file upload. Pay-per-KPU-hour with no idle cluster cost beyond minimum.

EMR Flink Advantages

Custom Flink version selection. Custom networking and IAM beyond Managed Flink defaults. Co-location with Hive, Presto, Hudi for hybrid batch-streaming pipelines. Cheaper at very high sustained throughput due to EC2 reserved-instance pricing.

When to Pick Each

Managed Flink: most teams, most workloads, especially when operational simplicity matters. EMR Flink: when custom configuration or co-location with other big data frameworks is essential. The DEA-C01 exam plants this decision occasionally — the default answer is Managed Flink unless the requirement explicitly calls for EMR's customization or co-location.

Studio Notebooks — Interactive Flink SQL

Managed Flink Studio is a Zeppelin-based notebook environment for interactive Flink SQL development.

What Studio Provides

Interactive Flink SQL execution in a Zeppelin notebook. Live preview of streaming results. Promotion of validated SQL queries to production Managed Flink applications. Built-in connectors to Kinesis, MSK, and Glue Data Catalog tables.

Studio vs Production Application

Studio is for development and exploration. Production applications run as standalone Managed Flink applications (Java JAR or PyFlink file). Studio's "deploy as application" feature converts a tested SQL query into a production application.

Common Integration Patterns

Real Flink architectures combine sources, processing, and sinks in standard patterns.

Pattern 1 — KDS to Flink to Firehose to S3

Kinesis Data Streams ingests events. Flink reads from KDS, applies windowed aggregation or enrichment, writes results to a downstream Kinesis Data Stream. Kinesis Data Firehose tee'd off the downstream stream delivers to S3 in Parquet partitioned by event time. The output S3 path is queryable from Athena.

Pattern 2 — MSK to Flink to OpenSearch

MSK Kafka topics carry events. Flink reads, applies session windowing or pattern detection, writes results to OpenSearch via the OpenSearch sink connector. Real-time dashboards in OpenSearch Dashboards display the live aggregates.

Pattern 3 — KDS to Flink to Iceberg

Kinesis Data Streams ingests CDC events from a database. Flink reads, applies upsert logic, writes to an Apache Iceberg table on S3. Downstream Athena and EMR query the Iceberg table with ACID semantics.

Pattern 4 — Two-Stream Join

Two Kinesis Data Streams (orders and inventory updates) join on product_id with event-time semantics. Flink maintains keyed state for both streams, joins them in a windowed interval (last 10 minutes), and emits enriched order events.

Cost Model

Managed Flink billing is per KPU-hour with checkpoint storage charges.

KPU Pricing

$0.11 per KPU-hour for application processing (us-east-1 baseline). One KPU = 1 vCPU + 4 GB memory.

Application Storage

50 GB of persistent application storage is included per KPU; additional storage (running applications and durable application backups) is charged per GB-month.

Bandwidth

Data transfer between Managed Flink and other AWS services in the same region is free; cross-region or internet egress incurs standard data transfer fees.

Cost Compared to Alternatives

For low-throughput stateful processing, Lambda may seem cheaper but cannot maintain state — total cost when adding DynamoDB or ElastiCache for external state often exceeds Managed Flink. For high-throughput sustained workloads, EMR Flink on reserved instances may be cheaper but the operational cost of running EMR clusters is non-trivial. Managed Flink wins on operational simplicity for most stateful streaming workloads.

Plain-Language Explanation: — Managed Service for Apache Flink

Stream processing with state is the kind of system where naming alone misses the engineering reality. Three concrete analogies make the structure stick.

Analogy 1 — The Air Traffic Controller

Picture an air traffic controller at a busy airport. Aircraft transmit position updates continuously (the input stream). The controller maintains a mental map of every active flight — altitude, heading, speed, destination, last contact time (that mental map is keyed state, one entry per aircraft tail number). The controller groups aircraft by approach window — "all aircraft landing in the next 5 minutes" — to issue clearance instructions (a tumbling window). When a pilot reports late or radio drops out, the controller does not panic; the system has bounded tolerance for delayed messages (watermark with bounded out-of-orderness) before declaring an aircraft missing. If the controller's console crashes, a backup console restores the mental map from the last full snapshot exchanged with central tracking (checkpoint).

The controller's mental map is exactly Flink's keyed state. The 5-minute approach window is exactly a tumbling window. The tolerance for delayed reports is exactly a watermark. The backup console restoration is exactly checkpointing. Trying to do air traffic control without state — answering each aircraft transmission independently with no memory of prior ones — is impossible. Trying to do real-time fraud scoring or session aggregation without Flink's state is the same impossibility, just less visible.

Analogy 2 — The Restaurant Bartender

Picture the bartender during a busy Friday night service. Every drink order arrives one at a time (the input stream). The bartender mentally tracks each customer's tab (keyed state per customer). The bartender batches drinks for delivery in waves — every 90 seconds the runner takes whatever is ready (a processing-time tumbling window, except food service usually wants event-time when the customer ordered, not when the bartender finished). When a customer disappears for a while and comes back later (gap in activity), they restart a new tab (session window with a 30-minute inactivity timeout). If the bartender hands off mid-shift to a relief bartender, they pass written notes about every active tab (savepoint for state migration across shifts).

Stateful stream processing is the bartender's mental model. Stateless processing — no memory between drinks, no tabs, no sessions — is a vending machine. Both have their place; for analytics on continuous customer activity, you need the bartender model, which is what Flink provides.

Analogy 3 — The Hospital Cardiac Monitor

Picture the cardiac monitor at the bedside of an ICU patient. The monitor continuously receives heart rate readings (the input stream). It computes a rolling 5-minute average updated every 30 seconds (a sliding window, 5-minute size, 30-second slide). It maintains the patient's baseline rhythm in memory (keyed state). When a reading arrives 10 seconds late from a transient sensor glitch, the monitor accepts it and recomputes (bounded out-of-orderness with allowed lateness). If a reading arrives 5 minutes late after the window has long closed, the monitor flags it for manual review (late event side output). If the monitor reboots, it restores the baseline and recent readings from the central monitoring system snapshot (checkpoint).

The cardiac monitor cannot work without state — knowing only the latest single heart rate is meaningless without context. Stream processing in fraud, IoT, recommendations, and operational analytics is the same. Managed Flink is the off-the-shelf cardiac monitor for streaming data; building the equivalent in Lambda plus DynamoDB plus custom code is the equivalent of building a cardiac monitor from raw transistors.

Common Exam Traps for Managed Flink

The DEA-C01 exam plants a consistent set of traps. Memorize all five.

Trap 1 — Lambda for Stateful Stream Processing

A scenario describes "rolling 5-minute average per device" or "session-based aggregation" and offers Lambda as a serverless answer. Wrong. Lambda is stateless — the rolling average requires either external state (DynamoDB, ElastiCache) plus complex consistency logic, or Flink's built-in keyed state. Right answer: Managed Flink.

Trap 2 — Firehose for Aggregation

A scenario asks for "per-minute event count delivered to S3" and offers Kinesis Data Firehose as a managed answer. Wrong. Firehose delivers raw records; it does not aggregate. The right answer is Managed Flink for the aggregation, then Firehose tee for S3 delivery, or Managed Flink directly writing to S3.

Trap 3 — Processing-Time When Event-Time Was Required

A scenario describes "events with timestamp X must be in window for X's hour even if they arrive late." Wrong answer: processing-time windowing. Right answer: event-time windowing with watermarks. Processing-time puts late events in the wrong window.

Trap 4 — Forgetting Checkpointing Equals No Fault Tolerance

A scenario describes "Flink application restarted after a TaskManager failure but lost the last hour of aggregations." The cause is checkpointing was disabled or misconfigured. The right answer is enable checkpointing with appropriate interval — usually 1 minute by default in Managed Flink.

Trap 5 — Late Events Silently Dropped

A scenario describes "downstream dashboard shows fewer events than the source — investigation shows events arriving 30 seconds late." The cause is the default Flink behavior of dropping late events past the watermark. The right answer is configure side output for late events or use allowed lateness, not increase parallelism or shard count.

Key Numbers and Must-Memorize Facts

Managed Flink Pricing

• $0.11 per KPU-hour (1 vCPU + 4 GB memory per KPU)
• 50 GB application storage included per KPU
• Studio notebook: same KPU-hour pricing for development environment

Window Types

• Tumbling: fixed size, non-overlapping
• Sliding: fixed size, slide interval, overlapping
• Session: activity-based with inactivity gap timeout
• Global: manually triggered with custom trigger

Time Semantics

• Event-time: timestamp embedded in event (deterministic, late-event safe)
• Processing-time: wall-clock at operator (simpler, non-deterministic)
• Ingestion-time: timestamp at source connector entry

Checkpoint Defaults (Managed Flink)

• Checkpoint interval: 1 minute default
• Min pause between checkpoints: 5 seconds
• Checkpoint storage: AWS-managed S3
• Savepoints: manual, used for code upgrades

Programming Model Selection

• Flink SQL: simplest, SQL-fluent teams, windowed aggregations
• Table API: relational with stronger typing, Java/Scala
• DataStream API: full power, complex stateful logic, custom integrations

FAQ — Managed Service for Apache Flink Top Questions

Q1 — When should I pick Managed Flink over Lambda or Firehose?

Pick Managed Flink when the requirement involves state across events: rolling aggregations, session-based grouping, windowed computations, joins between two streams, pattern detection, stateful enrichment, or exactly-once stream processing. Lambda is stateless — every invocation starts fresh, with state living only as long as the invocation. Firehose is delivery-only — it transforms individual records via Lambda but cannot aggregate across them. The DEA-C01 exam plants scenarios where the requirement word is "rolling," "windowed," "session," "aggregate over time," or "exactly-once" — all of these point at Flink. Lambda fits stateless per-record transformation; Firehose fits managed delivery with no aggregation.

Q2 — How do I handle late-arriving events in Flink?

Three options. Drop (default): events past the watermark are silently dropped. Cheapest, but loses data — only acceptable for non-critical aggregations. Side output: route late events to a separate stream for offline reprocessing or audit. Useful when late events must be accounted for but do not need to update the main result. Allowed lateness: extend window closing by a configured duration, accepting and re-emitting updated results for events that arrive within the lateness window. Most expressive, but downstream consumers must handle update semantics. The DEA-C01 exam asks scenarios where "events from a slow source are missing in aggregates" — the fix is one of these three handling strategies, depending on whether you can tolerate drop, want audit, or need updated results.

Q3 — What is the difference between event-time and processing-time?

Event-time uses a timestamp embedded in each event (extracted by a TimestampAssigner). Windows are defined over event-time, so a "5-minute window starting at 14:00" includes events whose embedded timestamp is between 14:00 and 14:05 regardless of when they arrived at the operator. Processing-time uses the wall-clock at the operator. Same window definition includes events that arrived at the operator between 14:00 and 14:05 wall-clock. Event-time produces deterministic results across replays and handles late-arriving data correctly; processing-time is simpler but non-deterministic. For most analytical use cases (clickstream, IoT, fraud, financial), event-time is the right choice. The DEA-C01 exam asks scenarios where the requirement is "group events by when they happened" (event-time) vs "group events by when we received them" (processing-time).

Q4 — How does checkpointing make Flink fault-tolerant?

Flink takes periodic snapshots of all state (keyed state, operator state, source positions) to a durable store (S3 in Managed Flink). On failure, the runtime restarts the application and restores state from the last completed checkpoint. Sources are repositioned to the offsets captured in the checkpoint, so processing resumes exactly where it left off. Combined with sources that support replay (KDS, MSK) and sinks that support transactional or idempotent writes (S3, Iceberg, Kafka with transactions), Flink achieves end-to-end exactly-once semantics. Without checkpointing, a failure loses all in-memory state and either skips the lost work or restarts from the source's earliest available offset — neither is acceptable for production stateful processing.

Q5 — How do I scale a Managed Flink application?

Two scaling axes. Parallelism: number of parallel instances of each operator. Increase parallelism for throughput-bound workloads. Set parallelism explicitly in application code or via the Managed Flink configuration. KPUs: total compute capacity (parallelism * KPUs-per-task-slot ratio). Increase KPUs for memory-bound or state-bound workloads. Auto-scaling adjusts KPUs based on observed utilization within configured min/max bounds. Diagnose scaling needs from CloudWatch metrics: high CPU utilization implies more parallelism, high checkpoint duration implies more memory, high backpressure (records-in-buffer) implies a downstream operator is the bottleneck. The DEA-C01 exam asks scenarios where the symptom is "Flink job lagging behind input rate" — the answer hinges on which metric is the bottleneck signal.

Q6 — Can Managed Flink read from MSK?

Yes. Managed Flink includes a Kafka source connector that reads from MSK or self-managed Kafka. Configure the connector with the bootstrap servers, topic name, consumer group, and deserialization schema. The connector tracks offsets per partition and integrates with Flink checkpointing for exactly-once. The same application can have multiple sources — KDS for one stream, MSK for another — and join them in keyed-stream operations. The DEA-C01 exam includes scenarios where the source is MSK and the sink is S3 (Parquet) or OpenSearch — Managed Flink handles both endpoints natively.

Q7 — When should I use Managed Flink Studio versus a production application?

Studio is for interactive development, prototyping Flink SQL queries, and validating windowed aggregations against live streams. The notebook environment uses Apache Zeppelin with Flink SQL, Python, Scala, and Java cells. Studio is not for production — it is single-user, less optimized for sustained workloads, and lacks production application features like alarms, automated deployment, and stable application IDs. Promote validated Studio queries to production by exporting them as Flink applications. The DEA-C01 exam plants scenarios where the requirement is "data engineer wants to interactively explore stream patterns" (Studio) vs "production pipeline computing real-time aggregates 24x7" (Managed Flink application).

Further Reading — Official AWS Documentation

The authoritative AWS sources for Managed Service for Apache Flink are: the Managed Service for Apache Flink Developer Guide (concepts, programming models, deployment, monitoring); the Windowing documentation (tumbling, sliding, session, global windows with code examples); the Checkpoints and Snapshots documentation (configuration, savepoints, fault tolerance semantics); the Scaling documentation (parallelism, KPUs, auto-scaling); the Studio Notebooks documentation (Zeppelin-based SQL development); and the broader Apache Flink documentation for deep-dive on DataStream and Table APIs.

The AWS Big Data Blog has multiple deep-dive posts on Flink stateful enrichment, late-event handling patterns, and Iceberg sink integration. The AWS Skill Builder course Exam Prep Standard Course: AWS Certified Data Engineer – Associate (DEA-C01) has Domain 1 modules covering Managed Flink scenarios. The AWS Managed Flink GitHub samples repository contains production-ready code for KDS sources, MSK sources, S3 sinks, OpenSearch sinks, and join patterns. The Apache Flink documentation at flink.apache.org is the upstream reference for the engine; the Managed Flink-specific behaviors layer on top of that base.