Weekly Digest — 2023/11/03

Highlights from this week (some cloud, a bit of Docker, Spark and AI):

AWS Certified Data Engineer

— The AWS Certified Data Engineer (Associate) exam is now available in beta. The certification focuses on four domains: (1) ingestion and transformation, (2) storage and management, (3) operations and (4) security and governance. The exam guide provides more details.

General Availability of Docker Scout

Docker Scout is ready to roll! 🐳 Docker Scout provides continuous evaluation of images against pre-defined policies, giving real-time security and management insights to developers. 💵 The free plan allows analysis of up to three image repositories.
Docker Scout landing page.

I tried it out on one of my images. Looks like I have some work to do!

Docker Scout analysis of a Gatsby Docker image.

Generative Agents

— The paper Generative Agents: Interactive Simulacra of Human Behavior describes simulations involving generative agents, computational entities that mimic human behaviour and interactions. The agents are powered by Large Language Models (LLMs). The simulations have demonstrated the emergence of individual and social behaviours.
Generative agents simulation.

Quant Research at Scale

— A walk through of using Spark (via the Python API) on Amazon EKS for large scale quant research. Uses data provided by Refinitiv via AWS Marketplace. Source code is provided so that you can replicate it yourself.

Spark on AWS Lambda

— It’s now possible to use Spark on AWS Lambda, another way to process large volumes of data using a serverless architecture. There’s a walk through and source code, so you can (and should) try this at home.

How Loaded is Taylor Swift?

FlowingData published a novel visualisation of Taylor Swift’s earnings. Inspiring to see such a creative way of presenting data.