Data ingestion and analytics
Ingesting, transforming and analyzing data at the right speed and size, from streams to data lakes and warehouses. Part of Domain 3: Design High-Performing Architectures, 24% of the scored content.
Kinesis Data Streams
Amazon Kinesis Data Streams
Real-time stream that stores records in shards so several consumers can read, and re-read, them in order.
Key points
- Shard: writes up to 1 MB/s or 1,000 records/s, reads up to 2 MB/s shared by the consumers.
- Enhanced fan-out: each registered consumer gets its own 2 MB/s per shard.
- Retention: 24 hours by default, up to 365 days. Consumers can replay data.
- Ordering: the partition key picks the shard, records with the same key stay in order.
- Capacity modes: on-demand (scales automatically) or provisioned (you manage shards).
On the exam
"Real-time, multiple consumers, replay, ordering per key" -> Kinesis Data Streams. Throttled writes -> more shards or a better partition key.
Fully managed delivery of streaming data to storage and analytics destinations, with buffering, transformation and format conversion. No shards to manage.
Key points
- Destinations: S3, Redshift (staged in S3, then COPY), OpenSearch Service and Serverless, Splunk, Snowflake, Apache Iceberg tables and HTTP endpoints.
- Buffering: delivers when a size (1 to 128 MiB for S3) or time (0 to 900 s) threshold is reached, so it is near real time.
- Format conversion: JSON to Parquet or ORC using a schema from the Glue Data Catalog.
- Transformation: a Lambda function can transform records, for example CSV to JSON before conversion.
- Sources include direct PUT and Kinesis data streams.
On the exam
"Load streaming data into S3 or Redshift with no code, near real time" -> Data Firehose. "Custom consumers, replay, milliseconds" -> Kinesis Data Streams.
MSK and video streams
Amazon MSK and Amazon Kinesis Video Streams
Managed Apache Kafka for teams already on Kafka, and a dedicated service for video from devices.
Key points
- Amazon MSK: runs open-source Apache Kafka, so existing producers and consumers work without code changes.
- MSK options: provisioned (Standard or Express brokers), MSK Serverless, MSK Connect for connectors and MSK Replicator.
- Kinesis Video Streams: ingests, stores and indexes video from cameras for playback (HLS, MPEG-DASH), analytics and ML.
- WebRTC in Kinesis Video Streams supports two-way real-time media.
On the exam
"Migrate an on premises Kafka cluster with minimal changes" -> Amazon MSK, not Kinesis. "Camera streams for analysis with Rekognition" -> Kinesis Video Streams.
Glue and Lake Formation
AWS Glue and AWS Lake Formation
Glue discovers, catalogs and transforms data, and Lake Formation governs who can read which parts of the data lake in S3.
Key points
- Crawlers infer schemas and populate the Glue Data Catalog, a Hive metastore compatible catalog used by Athena, EMR and Redshift Spectrum.
- Glue ETL: serverless Apache Spark jobs (batch or streaming). Job bookmarks avoid reprocessing old data. A typical job converts CSV to Parquet.
- AWS Glue DataBrew: visual data preparation without code.
- Lake Formation: permissions at database, table, column, row and cell level, LF-Tags for tag-based access and cross-account sharing.
On the exam
"Convert CSV in S3 to Parquet on a schedule, serverless" -> Glue ETL job. "Column-level access to the data lake for different teams" -> Lake Formation, not bucket policies.
Athena and Quick
Amazon Athena and Amazon Quick
Athena queries S3 with SQL without servers, and Amazon Quick turns the results into dashboards.
Key points
- Athena: serverless SQL on S3, billed per TB scanned. Uses the Glue Data Catalog for tables.
- Cut cost and time: columnar formats (Parquet, ORC), compression and partitioning reduce the data scanned.
- Federated Query: Lambda-based connectors query sources outside S3 (relational, NoSQL).
- Amazon Quick: evolved from Amazon QuickSight. Its BI feature, Amazon Quick Sight, builds dashboards with the SPICE in-memory engine and natural language questions.
On the exam
"Ad hoc SQL on logs in S3, no infrastructure" -> Athena. "Athena too slow and costly on CSV" -> convert to Parquet and partition. "Interactive BI dashboards" -> Amazon Quick.
Big data frameworks, the data warehouse and search and log analytics answer different questions about the same data.
Key points
- Amazon EMR: managed Spark, Hive, Trino, Flink and Hadoop on EC2, EKS or Serverless. Primary, core and task nodes. Spot is common on task nodes, avoid it on core nodes (HDFS).
- Amazon Redshift: columnar warehouse for SQL analytics. Redshift Serverless bills by RPU usage. Spectrum queries S3 without loading.
- Redshift features: concurrency scaling, data sharing and zero-ETL from Aurora, RDS and DynamoDB.
- Amazon OpenSearch Service: full-text search and log analytics with OpenSearch Dashboards. UltraWarm and cold tiers, OpenSearch Serverless and Ingestion.
On the exam
"Existing Spark or Hadoop jobs, full control of the cluster" -> EMR. "Complex SQL joins across terabytes for BI" -> Redshift. "Full-text search over logs or a catalog" -> OpenSearch.
Which analytics service
| Service | Best for |
|---|---|
| Athena | Ad hoc SQL on S3, serverless, pay per query. |
| Redshift | Recurring BI and complex SQL on structured data. |
| EMR | Spark, Hadoop and other frameworks at large scale. |
| Glue | Serverless ETL and the Data Catalog. |
| OpenSearch Service | Search, log analytics, dashboards over text. |
| Amazon Quick | Visualization and BI dashboards. |
Online services move data over the network on a schedule or protocol, physical options move it when the network is too slow.
Key points
- AWS DataSync: online copies from NFS, SMB, HDFS, object storage and other clouds to S3, EFS and FSx, with encryption, integrity checks and scheduling. An agent is needed for on premises sources.
- AWS Transfer Family: managed SFTP, FTPS, FTP and AS2 endpoints into S3 or EFS, with your existing identity provider. FTP only on internal VPC endpoints.
- AWS Snowball Edge: since November 7, 2025, available only to existing customers, and support in commercial Regions ends on December 31, 2026.
- AWS Data Transfer Terminal: physical locations where you bring your own storage devices and upload over a high-bandwidth connection.
On the exam
"Partners send files over SFTP into S3" -> Transfer Family. "Recurring sync of an NFS share to S3 or EFS" -> DataSync. Exam questions with petabytes and little bandwidth still point to Snowball Edge.
Transfer option by need
| Need | Choice |
|---|---|
| Online copy or recurring sync | DataSync |
| SFTP, FTPS, FTP, AS2 partners | Transfer Family |
| Offline, limited bandwidth | Snowball Edge (existing customers) or Data Transfer Terminal |
| SaaS application data | Amazon AppFlow |
| Third-party data sets | AWS Data Exchange |
AppFlow and Data Exchange
Amazon AppFlow and AWS Data Exchange
Managed ways to bring data from SaaS applications and from third-party providers into AWS.
Key points
- Amazon AppFlow: no-code flows between SaaS apps (Salesforce, ServiceNow, Zendesk, Slack) and S3, Redshift or Snowflake, in both directions.
- AppFlow triggers: on demand, on a schedule or on an event. Transformations include filtering, masking, mapping and validation. Supports PrivateLink.
- AWS Data Exchange: find, subscribe to and use third-party data delivered as files, APIs, Redshift datashares or S3 data access.
On the exam
"Import Salesforce data into S3 daily without code" -> AppFlow, not a custom Lambda. "Subscribe to a commercial data set" -> Data Exchange.