Weekly Digest

AWS Certified Data Engineer

AWS data engineering certification — The AWS Certified Data Engineer (Associate) exam is now available in beta. The certification focuses on four domains: (1) ingestion and transformation, (2) storage and management, (3) operations and (4) security and governance. The exam guide provides more details.

General Availability of Docker Scout

Docker — Docker Scout is ready to roll! 🐳 Docker Scout provides continuous evaluation of images against pre-defined policies, giving real-time security and management insights to developers. 💵 The free plan allows analysis of up to three image repositories.

Generative Agents

LLM AI simulation — The paper Generative Agents: Interactive Simulacra of Human Behavior describes simulations involving generative agents, computational entities that mimic human behaviour and interactions. The agents are powered by Large Language Models (LLMs). The simulations have demonstrated the emergence of individual and social behaviours.

Quant Research at Scale

quant Spark AWS — A walk through of using Spark (via the Python API) on Amazon EKS for large scale quant research. Uses data provided by Refinitiv via AWS Marketplace. Source code is provided so that you can replicate it yourself.

Spark on AWS Lambda

Spark AWS — It’s now possible to use Spark on AWS Lambda, another way to process large volumes of data using a serverless architecture. There’s a walk through and source code, so you can (and should) try this at home.

How Loaded is Taylor Swift?

visualisation — FlowingData published a novel visualisation of Taylor Swift’s earnings. Inspiring to see such a creative way of presenting data.