Kinesis Data Streams, Firehose, and Amazon MSK - DEA-C01 Data Engineer Study Notes

Amazon Kinesis Data Streams, Amazon Data Firehose, and Amazon Managed Streaming for Apache Kafka (MSK) form the streaming ingestion core of any modern AWS data engineering architecture, and on the DEA-C01 exam they are the single highest-density topic in Domain 1 (Data Ingestion and Transformation, 34 percent weight). Community study guides from Tutorials Dojo, Digital Cloud Training, ExamCert.App, and the popular Medium walkthroughs all flag the same pain point — candidates conflate Kinesis Data Streams (true real-time) with Kinesis Data Firehose (near-real-time), and they confuse MSK (managed Apache Kafka with the full open-source ecosystem) with Kinesis (AWS-native simplicity but no Kafka API). Getting these distinctions wrong on the exam is the same wrong choice that breaks production: pick Firehose for sub-second fraud scoring and your model misses its SLA; pick Kinesis Data Streams when the consumer team expected Kafka Connect connectors and you build the connector ecosystem from scratch.

This study note is built for the data engineer perspective. It covers the four Kinesis services overview, Kinesis Data Streams shards and partition keys and consumer types and on-demand mode, KDS throughput math, Kinesis Data Firehose delivery destinations and buffering and format conversion, KDS vs Firehose differences, MSK Provisioned vs MSK Serverless, MSK Connect for source/sink connectors, ordering guarantees, encryption and security, and the canonical exam traps that catch most candidates. By the end the streaming service selection should be a structured choice you can defend cleanly.

What Is Amazon Kinesis — The Four-Service Family

Amazon Kinesis is the umbrella name for AWS's streaming data services. The family has four distinct services that the DEA-C01 exam tests as separate decisions.

Kinesis Data Streams (KDS)

Real-time data stream with shards, custom consumers, retention up to 365 days. The lowest-latency option (sub-second end-to-end). Use for fraud scoring, real-time bidding, IoT alerting, and anywhere you need consumer code that does more than write to S3.

Kinesis Data Firehose (now Amazon Data Firehose)

Managed delivery stream that writes to S3, Redshift, OpenSearch, HTTP endpoints, Splunk, and Snowflake. Buffers records by size or time before delivery. Near-real-time only (60-900 second buffering). Use for log archival, analytics destination loading, and any scenario where managed delivery beats custom consumer code.

Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics)

Managed Flink for stateful stream processing — windowing, aggregations, joins, pattern detection. Reads from KDS and MSK; writes to S3, Redshift, OpenSearch. Covered in detail in the kinesis-managed-flink-stream-processing topic.

Kinesis Video Streams

Video-specific streaming for cameras, IoT video, and ML video analytics. Out of scope for DEA-C01 in any depth — recognize the name and move on.

Kinesis Data Streams — Shards and Partition Keys

Kinesis Data Streams is the foundational real-time service. Understanding shards is mandatory for the exam.

What a Shard Is

A shard is the unit of capacity in a Kinesis Data Stream. Each shard supports 1 MB/s or 1000 records/second of writes and 2 MB/s of reads (shared across all standard consumers of that shard). A stream is composed of one or more shards; throughput scales linearly with shard count.

Partition Key — Routing Records to Shards

When a producer calls PutRecord, it supplies a partition key. Kinesis hashes the partition key (MD5) and routes the record to the shard whose hash range contains the result. Records with the same partition key always land on the same shard, which guarantees ordering within a key. Choose partition keys with high cardinality — using a low-cardinality key like "country" with 200 values across millions of users creates hot shards where all USA traffic crowds onto one shard.

Retention Period

Default retention is 24 hours. You can extend to 7 days at no extra cost or up to 365 days for an additional fee (long-term retention). Long retention is useful for replay scenarios, late-arriving consumers, and compliance archival of stream data.

Sequence Numbers and Iterator Types

Every record in a shard has a monotonically increasing sequence number. Consumers track their position with a sequence number and request records via shard iterators. Iterator types: TRIM_HORIZON (oldest record in retention), LATEST (only new records), AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER, AT_TIMESTAMP. The DEA-C01 exam tests "consumer was offline for an hour, how does it resume" — the answer is checkpoint the last processed sequence number and resume with AFTER_SEQUENCE_NUMBER.

A Kinesis shard supports 1 MB/s or 1000 records/sec write and 2 MB/s read shared across standard consumers; partition key hashes determine which shard a record lands on. This is the one capacity formula to memorize for DEA-C01. Total stream write capacity equals shard count times 1 MB/s. Total stream read capacity for a single consumer equals shard count times 2 MB/s. With multiple standard consumers, they share the 2 MB/s per shard — Enhanced Fan-Out gives each consumer a dedicated 2 MB/s pipe per shard. Hot shards happen when partition key cardinality is too low and one shard exceeds its limits while others sit idle. The exam plants scenarios where the fix is either a higher-cardinality key or splitting shards.

KDS Throughput — Per-Shard Limits and Scaling

Throughput math is exam-critical.

Write Limits

Per shard: 1 MB per second OR 1000 records per second (whichever hits first). Exceeding either triggers ProvisionedThroughputExceededException, and the producer should back off and retry. PutRecords (batched) is more efficient than PutRecord (single) for throughput.

Read Limits

Per shard, standard consumers: 2 MB per second OR 5 GetRecords calls per second, shared across all standard consumers reading that shard. Per shard, Enhanced Fan-Out (EFO) consumers: 2 MB per second per consumer with sub-200ms propagation. EFO is the right answer when more than two consumers read the same stream.

Shard Splitting and Merging

To increase throughput, split a shard into two child shards (each gets half the original hash range). To decrease throughput and save cost, merge two adjacent shards into one. Both operations are online — the original shard remains readable for the retention period (so consumers can finish processing in-flight data) but accepts no new writes after split/merge.

On-Demand Capacity Mode

On-demand mode automatically scales shard count based on observed throughput, up to 200 MB/s write or 200,000 records/second by default (extendable). You pay per GB ingested and per GB retrieved instead of per shard-hour. Use for unpredictable workloads, prototypes, and avoiding capacity planning. Provisioned mode is cheaper for sustained predictable throughput.

Kinesis Data Firehose — Managed Delivery to S3 and More

Firehose is the no-code managed delivery option.

Supported Destinations

S3 (the most common), Amazon Redshift (via S3 staging plus COPY), Amazon OpenSearch Service, HTTP endpoints (with optional API Gateway), Splunk, Datadog, MongoDB Atlas, New Relic, and Snowflake. Each destination has destination-specific configuration (Redshift cluster credentials, OpenSearch index name, HTTP endpoint URL).

Buffering Hints

Records are buffered before delivery. Buffer size: 1-128 MB (default 5 MB). Buffer interval: 60-900 seconds (default 300 seconds). Whichever limit is hit first triggers delivery. The minimum 60-second buffer is the floor for end-to-end latency, which is why Firehose is near-real-time only.

Format Conversion

Firehose can convert records from JSON to Parquet or ORC during delivery using a Glue Data Catalog table for the schema. Parquet and ORC are columnar formats that downstream Athena and Redshift Spectrum queries scan more efficiently. Format conversion adds a per-GB charge but saves significant downstream query cost.

Dynamic Partitioning

Firehose can dynamically partition delivered records into S3 prefixes based on field values from each record. Example: deliver records into s3://bucket/year=2026/month=05/day=02/customer_segment=A/. Dynamic partitioning improves Athena partition pruning and reduces query scan cost. Charged per GB processed.

Buffering Errors and Backup

Firehose can be configured to back up source records to a separate S3 bucket before transformation, useful for replay if downstream delivery fails. Failed records (after retry) land in an error S3 prefix.

Kinesis Data Firehose is near-real-time only — minimum 60-second buffer means end-to-end latency cannot drop below one minute. This is the most-tested distinction on the DEA-C01 exam. If a scenario says "near-real-time delivery to S3 with managed buffering and format conversion," Firehose is the right answer. If a scenario says "real-time fraud scoring within 1 second" or "react to events within 5 seconds," Firehose is the wrong answer regardless of how convenient the managed delivery sounds. Even with the lowest 60-second buffer setting, Firehose adds delivery latency on top of that. The right answer for true real-time is Kinesis Data Streams plus a Lambda or Flink consumer, with Firehose only acceptable as a tee path to S3 archive.

KDS vs Firehose — The Decision Cheat Sheet

The exam plants this comparison repeatedly. Memorize the distinctions.

Latency

KDS: sub-second end-to-end with custom consumers. Firehose: 60+ seconds minimum.

Custom Code

KDS: requires consumer code (Lambda, KCL, Flink, Spark Streaming, custom). Firehose: zero consumer code, just configure destinations.

Destinations

KDS: anywhere you can write consumer code. Firehose: S3, Redshift (via S3), OpenSearch, HTTP, third-party SaaS.

Pricing Model

KDS: per shard-hour plus per million PUT records (provisioned), or per GB ingested plus per GB retrieved (on-demand). Firehose: per GB ingested with surcharges for format conversion, dynamic partitioning, and VPC delivery.

Retention

KDS: 24 hours default, up to 365 days. Firehose: no retention (records flow through, not stored in Firehose itself).

Replay

KDS: yes, by setting iterator to TRIM_HORIZON or AT_TIMESTAMP. Firehose: no replay; records are consumed once and delivered.

Use Cases

KDS: real-time fraud, IoT, recommendations, anywhere with custom consumer logic. Firehose: log archival, analytics warehouse loading, OpenSearch indexing, third-party SaaS integration.

Firehose cannot deliver to custom destinations without a Lambda transformation hop — and even with Lambda, it cannot make synchronous decisions like real-time fraud scoring. A common exam trap: the scenario describes a custom database (DynamoDB, Aurora, on-premises Postgres) and offers Firehose as a managed delivery answer. Firehose's native destinations are limited to S3, Redshift, OpenSearch, HTTP, and a few SaaS targets. To reach DynamoDB or Aurora, you must use Firehose's Lambda transformation feature plus a Lambda that writes to the target — at which point Lambda+Kinesis Data Streams is usually the cleaner architecture. Firehose's strength is managed buffering and managed delivery to its native destinations, not "any destination with Lambda."

KDS Scaling — Splitting, Merging, and On-Demand

Operating a Kinesis Data Stream means managing shards or letting AWS manage them.

Manual Resharding (Provisioned Mode)

Operators run SplitShard to double a shard's capacity (creates two child shards, each covering half the parent's hash range) or MergeShards to halve cost (combines two adjacent shards). Manual resharding requires monitoring CloudWatch metrics (IncomingBytes, IncomingRecords, WriteProvisionedThroughputExceeded) and reacting before producers start throttling.

On-Demand Mode — Hands-Off Scaling

On-demand mode automatically doubles shard count when sustained traffic exceeds capacity, up to the account default of 200 MB/s. There is no manual shard management. You pay for what you use ($0.04/GB ingested, $0.04/GB retrieved at us-east-1 baseline pricing). On-demand is the right answer when traffic is unpredictable, when the team does not have capacity to monitor and reshard, or when prototype workloads do not justify capacity planning.

Provisioned vs On-Demand Cost Trade-Off

Provisioned: pay per shard-hour ($0.015/hr us-east-1) plus per million PUTs ($0.014). Cheaper for sustained predictable throughput. On-demand: pay per GB. Cheaper for spiky or unknown workloads; ~5-10x more expensive than provisioned at sustained high throughput. The DEA-C01 exam asks "predictable 50 MB/s for years" (provisioned) vs "unpredictable spiky workload" (on-demand).

Amazon MSK — Managed Apache Kafka

MSK is the managed Apache Kafka service. The DEA-C01 exam tests when MSK beats Kinesis and vice versa.

When MSK Beats KDS

The team has existing on-premises Kafka and wants AWS migration with API compatibility. The team uses Kafka Connect, Kafka Streams, Schema Registry, or other Kafka ecosystem tools. The team needs Kafka-specific features like compacted topics, exactly-once transactions, or larger message size (Kafka supports up to 1 MB messages by default vs Kinesis 1 MB record limit but with different semantics). Multi-tenancy with per-topic access control via Kafka ACLs.

MSK Cluster Architecture

Brokers run on EC2 instances in private subnets across multiple Availability Zones. AWS manages broker patching, replacement on failure, and storage scaling. You configure broker instance type (kafka.m5.large to kafka.m5.24xlarge or kafka.t3.small for dev), number of brokers per AZ, EBS volume size, and Kafka version.

Topic Partitions and Replication

A Kafka topic is divided into partitions; producers write to partitions and consumers read from partitions. Each partition is replicated across multiple brokers (default replication factor 3) for durability. Ordering is guaranteed within a partition, not across partitions — same as Kinesis shards.

MSK Provisioned Pricing

Pay per broker-hour ($0.21/hr for kafka.m5.large at us-east-1) plus per GB-month of EBS storage. The cluster runs 24x7 regardless of traffic. Cost scales with cluster size, not with throughput.

MSK Connect — Managed Kafka Connect

MSK Connect is managed Kafka Connect for source/sink connector workloads.

What Kafka Connect Does

Kafka Connect is the open-source framework for moving data between Kafka and external systems. Source connectors pull data from databases (Debezium for CDC), files, or APIs into Kafka topics. Sink connectors push data from Kafka topics to S3, Redshift, OpenSearch, JDBC databases, or other targets.

MSK Connect vs Self-Managed

MSK Connect runs Kafka Connect workers as a managed service. You upload the connector plugin, configure source/target endpoints, and MSK Connect handles worker provisioning, scaling, and recovery. Pricing is per worker-hour. Self-managed Kafka Connect on EC2 is cheaper at scale but requires operational ownership.

Common MSK Connect Patterns

Debezium CDC: PostgreSQL or MySQL writes into Kafka topics in real-time, downstream consumers (Flink, Lambda) process the change stream. S3 sink: Kafka topic continuously delivered to S3 partitioned by date. JDBC sink: Kafka topic written to Aurora or RDS for operational use.

MSK Serverless — No Brokers to Manage

MSK Serverless is the on-demand Kafka equivalent.

What MSK Serverless Provides

Fully managed Kafka with no broker provisioning, no capacity planning, no broker tuning. Auto-scales partitions and throughput. Pay per partition-hour and per GB ingress/egress. Topics can scale to higher partition counts than provisioned MSK.

Provisioned vs Serverless Cost Trade-Off

MSK Serverless is cheaper for variable or low-volume workloads. MSK Provisioned is cheaper for sustained high-throughput workloads where the broker fleet cost amortizes well across constant traffic. The DEA-C01 exam asks "low operational overhead with unpredictable Kafka traffic" (Serverless) vs "1 GB/s sustained Kafka traffic with cost optimization" (Provisioned).

Pick MSK Serverless when traffic is unpredictable, when the team does not want broker management, or when partition count needs to grow above provisioned cluster limits. MSK Serverless removes the entire broker provisioning, scaling, and tuning surface area. The trade-off is per-partition and per-GB pricing that becomes more expensive than provisioned at very high sustained throughput. The right pattern: start with Serverless for new Kafka workloads, migrate to Provisioned only when sustained traffic and cost analysis justify the broker management overhead. The DEA-C01 exam asks scenario questions where "minimum operational overhead" plus "unpredictable Kafka traffic" points at Serverless directly.

MSK vs Kinesis — When to Choose Which

Both are streaming services. Different decision criteria.

Pick MSK When

Existing Kafka ecosystem (Kafka Connect, Kafka Streams, Schema Registry, kSQL). Migration from on-premises Kafka with API compatibility. Multi-tenant Kafka clusters with per-topic ACLs. Compacted topics for keyed event streams (latest value per key). Larger throughput per partition than Kinesis shard limits when partition count is high.

Pick Kinesis When

AWS-native simplicity (no Kafka knowledge required). Direct integration with Lambda, Firehose, Flink, and AWS-native consumers. On-demand capacity mode for unpredictable workloads. Lower operational overhead than even MSK Serverless. Long retention (up to 365 days) without manual log retention management.

Common Hybrid Pattern

Some architectures use both: MSK as the central event bus across services, Kinesis for AWS-native ingestion (CloudWatch Logs subscription, AWS service event capture), with cross-stream replication via MSK Connect or custom Lambda.

Ordering Guarantees

Both Kinesis and MSK guarantee ordering, but only within a unit.

Kinesis Ordering Within a Shard

All records with the same partition key route to the same shard, and within a shard records are strictly ordered by sequence number. Across shards, ordering is not guaranteed. A consumer reading a stream of customer-123 transactions sees them in order if customer_id is the partition key.

MSK Ordering Within a Partition

All records to the same partition (assigned by partition key hash or producer-specified partition) are strictly ordered. Across partitions, ordering is not guaranteed. Same logical model as Kinesis shards.

Cross-Shard or Cross-Partition Ordering

If global ordering across all records is required, the only safe options are: use a single shard or partition (limits throughput to per-shard maximums) or implement application-level ordering with global sequence numbers (complex, defeats the point of partitioning). The DEA-C01 exam plants scenarios like "preserve transaction order across the entire stream" — the answer is "use a single shard" or "partition by global ordering key" depending on the constraint.

Encryption and Security

Both services support encryption at rest and in transit.

Kinesis Data Streams Encryption

Server-side encryption with AWS KMS (AWS-managed key or customer-managed CMK). Enabled per stream. Encryption applies to data at rest in stream storage; in-transit encryption is enforced by HTTPS API endpoints.

Kinesis Data Firehose Encryption

Server-side encryption for delivery destinations (SSE-S3 or SSE-KMS for S3 destinations). In-transit encryption to destinations is enforced. Source records can be encrypted by KMS if the producer uses a Kinesis stream as the source (Firehose decrypts via the stream's KMS key).

MSK Encryption

In-transit: TLS for client-broker, broker-broker, and broker-ZooKeeper. Configurable as TLS-only, TLS-plus-plaintext, or plaintext-only. At rest: server-side encryption with KMS for broker storage volumes.

Authentication

KDS: IAM (signed API calls). Firehose: IAM. MSK: SASL/SCRAM (username-password), IAM authentication (AWS-native auth on Kafka API), TLS mutual auth (client certificate), or unauthenticated (in private subnets only).

Plain-Language Explanation: — Kinesis Streams, Firehose, and MSK

The streaming service trio is the kind of decision where naming alone misses the architectural reality. Three concrete analogies make the structure stick.

Analogy 1 — Express Train, Postal Truck, and Freight Railway

Picture three ways to move goods between two cities. The express train (Kinesis Data Streams) departs every minute, carries passengers and cargo at high speed, and arrives at a precise platform — but you must coordinate a person at the destination platform to receive the goods, and you must build that receiving station yourself. The postal truck (Kinesis Data Firehose) waits at the loading dock for either a full truck or 5 minutes (whichever first), then drives to a fixed set of postal warehouses — you do not manage the truck or the warehouse, but you cannot send to a destination that does not have a postal warehouse, and the wait time means urgent packages do not get there in seconds. The freight railway (Amazon MSK) is the industrial-scale version of the train — many cars per train, dedicated tracks, every shipping company in the country has built tools that work with the railway standard, but operating a freight railway requires staff, scheduling, and infrastructure investment.

For real-time deliveries to a custom destination: the express train (KDS plus custom consumer). For batched deliveries to a standard destination: the postal truck (Firehose). For industrial-scale shipping with the full railway ecosystem: the freight railway (MSK). The DEA-C01 exam asks scenario questions where the requirement narrows to one of these three; recognizing the parallel speeds the choice.

Analogy 2 — Phone Line, Voicemail Service, and Office PBX

Picture three ways to handle business communication. A direct phone line (Kinesis Data Streams) connects you to the recipient in real-time — when they pick up, you talk; if they are away, you hang up and try later, but the line itself is always live. A voicemail service (Kinesis Data Firehose) records messages for at least a minute then plays them to the recipient when convenient — you cannot interrupt the recipient mid-task, but you also do not need them to be at the desk. An office PBX (Amazon MSK) is a full corporate phone system with extensions, call routing, conference bridges, voicemail, recording, and integrations to every CRM in the industry — installing it is a project, but once installed every department can use it.

Real-time conversation: phone line. Recorded delivery to a known recipient: voicemail. Full corporate communication infrastructure: PBX. The decision is exactly the same as KDS vs Firehose vs MSK on the exam.

Analogy 3 — Restaurant Counter, Drive-Through, and Catering Operation

Picture three ways a restaurant takes orders. The counter (Kinesis Data Streams) takes each customer's order in real-time, the kitchen processes it immediately, the customer waits 60 seconds and walks out with food. The drive-through (Kinesis Data Firehose) batches orders at the speaker, the kitchen processes them in waves, the customer pulls forward and waits 3 minutes — fewer staff needed but slower per customer. The catering operation (Amazon MSK) handles multi-course events for hundreds of guests with menu planning, equipment rental, dedicated catering chefs, and the full restaurant ecosystem deployed offsite — overkill for a single sandwich, the only option for a corporate event.

Counter for individual real-time service. Drive-through for batched managed delivery. Catering for industrial-scale operation with the full ecosystem. The DEA-C01 exam asks the same architectural question with technical names.

Common Exam Traps for Kinesis and MSK

The DEA-C01 exam plants a consistent set of traps. Memorize all five.

Trap 1 — Firehose for Real-Time

A scenario says "real-time within 1 second" and offers Firehose as a managed answer. Wrong. Firehose has a 60-second minimum buffer. The right answer is KDS plus Lambda or Flink.

Trap 2 — KDS Without a Consumer

A scenario describes data being put into KDS with no consumer mentioned and asks where the data goes. KDS does not auto-deliver — it is a buffer waiting for consumers. If the requirement is "deliver to S3 without writing code," the answer is Firehose, not KDS.

Trap 3 — Hot Shards from Low-Cardinality Partition Keys

A scenario describes a stream with high write traffic but skewed across shards (one shard at 100% capacity, others idle). The cause is a low-cardinality partition key. The fix is choosing a higher-cardinality key — not adding shards. Adding shards without fixing the key just creates more underutilized shards while the hot one stays hot.

Trap 4 — Confusing Enhanced Fan-Out with Adding Shards

A scenario describes four consumers competing for shared 2 MB/s read throughput per shard. The fix is Enhanced Fan-Out (each consumer gets dedicated 2 MB/s per shard), not adding shards. Adding shards increases write capacity but does not change per-consumer read bottleneck on shared reads.

Trap 5 — MSK Provisioned When Serverless Was the Right Answer

A scenario describes "unpredictable Kafka traffic with low operational overhead." Wrong answer: MSK Provisioned with manual broker sizing. Right answer: MSK Serverless. Provisioned is the correct choice only when sustained throughput is predictable and cost optimization justifies broker management.

Key Numbers and Must-Memorize Facts

Kinesis Data Streams

1 MB/s or 1000 records/s write per shard
2 MB/s read per shard (shared) or 2 MB/s per consumer per shard (Enhanced Fan-Out)
Default retention 24 hours, max 365 days
On-demand mode default cap: 200 MB/s write, 200,000 records/s
Sub-200ms propagation with EFO; ~70ms median end-to-end

Kinesis Data Firehose

Buffer size 1-128 MB (default 5 MB)
Buffer interval 60-900 seconds (default 300 seconds, minimum 60)
Native destinations: S3, Redshift (via S3), OpenSearch, HTTP, Splunk, Datadog, Snowflake, MongoDB Atlas, New Relic
Format conversion to Parquet/ORC via Glue table
Dynamic partitioning supported

Amazon MSK

Provisioned: per broker-hour pricing (kafka.m5.large $0.21/hr us-east-1)
Serverless: per partition-hour plus per GB ingress/egress
Default replication factor 3 across AZs
Authentication: SASL/SCRAM, IAM, TLS mutual auth, plaintext (private only)
MSK Connect: per worker-hour managed Kafka Connect

Ordering

KDS: ordered within a shard
MSK: ordered within a partition
No cross-shard or cross-partition ordering without single shard/partition

DEA-C01 exam priority — Kinesis Data Streams, Firehose, and Amazon MSK. This topic carries weight on the DEA-C01 exam. Master the trade-offs, decision boundaries, and the cost/performance triggers each AWS service exposes — the exam will test scenarios that hinge on knowing which service is the wrong answer, not just which is right.

Key facts to memorize. Pin the service-level limits, default behaviours, and SLA promises related to this topic. The exam often tests recall of specific defaults (durations, limits, retry behaviour, free-tier boundaries) where memorising the exact number is the difference between right and 'almost right'.

FAQ — Kinesis and MSK Top Questions

Q1 — When should I pick Kinesis Data Streams over Kinesis Data Firehose?

Pick KDS when latency must be sub-second, when you need custom consumer logic (filtering, multi-target routing, stateful processing), when you want long retention for replay (up to 365 days), or when consumers are written by your team. Pick Firehose when 60+ seconds latency is acceptable, when the destination is S3/Redshift/OpenSearch and you do not want consumer code, or when format conversion (JSON to Parquet) and dynamic partitioning are needed. Many architectures use both: KDS for the real-time consumer path plus Firehose tee'd off the same KDS stream for S3 archive.

Q2 — How do I diagnose a hot shard problem?

Watch CloudWatch metrics IncomingBytes and IncomingRecords per shard. If one shard sits at the 1 MB/s or 1000 record/s ceiling while peers sit idle, the partition key cardinality is too low. The fix is changing the partition key to something higher-cardinality (user_id instead of country, transaction_id instead of customer_segment). Splitting the hot shard alone does not help — the new child shards inherit the same hash range split, and if the partition key still maps overwhelmingly to one half, the problem persists. The DEA-C01 exam asks scenarios where the right fix is partition key redesign, not capacity scaling.

Q3 — What is Enhanced Fan-Out and when do I need it?

Enhanced Fan-Out (EFO) gives each registered consumer a dedicated 2 MB/s read pipe per shard with sub-200ms propagation. Without EFO, all standard consumers share the per-shard 2 MB/s read budget, which means three consumers each get ~666 KB/s and they slow each other down. Use EFO when more than two independent consumer applications read the same stream, when sub-second propagation matters, or when consumers should not interfere with each other's throughput. EFO costs more per consumer (per shard-hour-consumer plus per GB retrieved) but is the right answer for multi-consumer streams. The exam plants this pattern as "four consumers reading the same stream, one falling behind."

Q4 — When does MSK make sense over Kinesis?

MSK is the right choice when the team has existing Kafka skills, when migration from on-premises Kafka requires API compatibility, when Kafka ecosystem tools (Kafka Connect, Kafka Streams, Schema Registry, kSQL) are required, when compacted topics are needed for keyed-state event streams, or when multi-tenant Kafka clusters with per-topic ACLs are required. Kinesis is the right choice when AWS-native simplicity beats the Kafka ecosystem, when on-demand capacity mode beats broker management, or when consumers are written from scratch using Lambda/Firehose/Flink. The DEA-C01 exam includes scenario questions like "migrate on-premises Kafka with minimal application changes" (MSK) and "AWS-native streaming with no Kafka background" (Kinesis).

Q5 — How does MSK Connect compare to running Kafka Connect on EC2?

MSK Connect is a managed runtime for Kafka Connect workers. You upload the connector plugin, configure connector properties, and MSK Connect handles worker provisioning, auto-scaling, recovery on failure, and patching. Pricing is per worker-hour. Self-managed Kafka Connect on EC2 is cheaper at scale (you control instance types and reserved instance pricing) but requires operational ownership of the worker fleet. Pick MSK Connect for managed simplicity and standard connectors. Pick self-managed when you have Kafka Connect operational expertise and cost optimization at scale matters. Common connectors: Debezium for CDC, S3 sink, Snowflake sink, JDBC source/sink.

Q6 — What are ordering guarantees in Kinesis and MSK?

Kinesis guarantees ordering within a shard — all records with the same partition key route to the same shard and are strictly ordered by sequence number within that shard. Across shards, no ordering guarantee. MSK guarantees ordering within a partition — all records with the same partition key (or producer-specified partition) are strictly ordered. Across partitions, no ordering guarantee. For global ordering across all records, the only options are a single shard/partition (limits throughput) or application-level reordering. The DEA-C01 exam tests scenarios where customer ordering is required — partition by customer_id, ordering is preserved per customer; cross-customer ordering is not.

Q7 — When should I pick on-demand vs provisioned mode for Kinesis Data Streams?

Pick on-demand when traffic is unpredictable, when capacity planning is impractical, when you do not want to monitor and reshard, or for prototypes and dev environments. Pricing is per GB ingested and per GB retrieved. Pick provisioned when traffic is predictable and sustained, when cost optimization at high throughput matters, or when on-demand caps (200 MB/s default) are insufficient. Pricing is per shard-hour plus per million PUTs. Crossover point: at sustained ~5 MB/s and above with predictable patterns, provisioned typically wins on cost. The DEA-C01 exam asks "spiky traffic, no time for capacity planning" (on-demand) vs "steady 50 MB/s for years, optimize cost" (provisioned).