Skip to content

Research & explainers

Deep Dives · Latest

Explaining how systems work

12 selected reportsIn the last 30 days 12 reportsIndexed 37 reports

RSS
updated

Selected: Deep Dives

Oct 29ThuItems 1–12
  1. AWS Machine Learning BlogFirst-party

    AWS Releases Ray Train Deep Learning Container for Scalable PyTorch Training on EKS

    AWS has published a pre‑built Ray Train Deep Learning Container (DLC) that bundles CUDA, PyTorch, NCCL, the Elastic Fabric Adapter (EFA) userspace libraries, the AWS OFI NCCL plugin, and a compatible Ray version. The DLC is stored in Amazon ECR Public and is designed to run on Amazon Elastic Kubernetes Service (EKS) with the KubeRay operator. A single RayCluster manifest can launch a head node and multiple worker pods, each attaching to a GPU (e.g., eight NVIDIA L4 GPUs on two g6.12xlarge instances). Because the DLC already includes the EFA stack, inter‑node collective communication automatically uses EFA, eliminating the need to build and maintain a custom container image. The sample workflow demonstrates fine‑tuning the Qwen2.5‑1.5B model with PyTorch Fully Sharded Data Parallel (FSDP) across the cluster, using only the published DLC image and external configuration files.

    Read source aws.amazon.com
Oct 28Wed
  1. Google ResearchFirst-party

    Google Research introduces sleep framework for LLMs to retain knowledge in continual learning

    Google Research proposes a sleep framework for continual learning in large language models, enabling them to retain knowledge beyond the context window. The framework operates in alternating active and offline sleep phases, during which the model consolidates short-term context into stable long-term parameters using Mixture-of-Experts memory blocks. This approach allows LLMs to continuously acquire new facts, retain long-context knowledge up to 10M tokens, and improve complex reasoning without altering the core architecture.

    Read source research.google
Oct 27Tue
  1. AWS Machine Learning BlogFirst-party

    AWS launches Managed Amazon Bedrock Knowledge Base with AgenticRetrieveStream API for multi-turn retrieval

    AWS introduces a Managed Amazon Bedrock Knowledge Base that transforms retrieval into an agentic capability, replacing single embed-and-fetch calls with multi-turn planning and tool-driven retrieval. The AgenticRetrieveStream API enables a reasoning agent to decide whether to retrieve, which knowledge base to consult, and whether to iterate before composing a grounded, cited answer. This addresses limitations of ordinary RAG, such as answers spanning multiple corpora or requiring follow-up lookups after initial results. The service is now available as a first-class feature, eliminating the need to manually manage OpenSearch Serverless collections. AWS CDK users can now provision the knowledge base and AgenticRetrieveStream as stacks, though observability remains critical due to the non-deterministic paths agents take during retrieval.

    Read source aws.amazon.com
Oct 23Fri
  1. AWS Machine Learning BlogFirst-party

    Amazon SageMaker training session APIs let users customize LLM training loops without managing GPUs

    Amazon SageMaker AI introduces training session APIs that let users drive GPU-backed sessions from local Python code. The service handles distributed execution, gradient synchronization, and fault recovery, while users retain control over training procedures such as custom loss functions and reinforcement learning reward functions. A new cookbook provides ready-to-run SFT and RL recipes as blueprints.

    Read source aws.amazon.com
  2. AWS Machine Learning BlogFirst-party

    AWS SageMaker AI Uses Two Measurements to Set TTS Auto‑Scaling Targets

    AWS deployed the 82‑million‑parameter Kokoro‑82M text‑to‑speech model on a SageMaker AI real‑time endpoint and replaced guesswork with two live‑endpoint measurements. The maximum concurrency a single instance can handle before breaching the service‑level objective (Q_max) was measured at 50 requests, and the end‑to‑end time for a scale‑out (T_total) was 320.2 seconds. These values were used to calculate precise scale‑out and scale‑in thresholds and cooldown periods, eliminating the hidden risk of over‑ or under‑provisioning that traditional auto‑scaling metrics miss.

    Read source aws.amazon.com
Oct 22Thu
Oct 21Wed
  1. AWS Machine Learning BlogFirst-party

    i14 Trains and Serves Laplace 3B Linear‑Attention Model on AWS Trainium2

    i14’s Laplace 3B, a 3‑billion‑parameter linear‑attention language model, was trained on a single trn2.48xlarge AWS Trainium2 instance at roughly 120,000 tokens per second. During inference, the model delivered 11,544 output tokens per second across 1,280 concurrent sequences while keeping a fixed 29.5 MB recurrent state per sequence, independent of context length. The work leveraged TorchNeuron Native to port i14’s Gated Linear Attention (GLA) kernels to Trainium2, preserving the model’s Muon optimizer and custom architecture.

    Read source aws.amazon.com
  2. AWS Machine Learning BlogFirst-party

    Amazon Quick and Bedrock AgentCore Enable Conversational Supply‑Chain Decision‑Making

    Amazon Quick and Bedrock AgentCore let supply‑chain managers ask plain‑language questions and receive real‑time, cross‑system answers from live ERP, monitoring, procurement, and email data. By exposing enterprise Model Context Protocol (MCP) servers as composable tools, the system eliminates manual stitching of data and reduces a 45‑minute multi‑tool workflow to a few conversational prompts. The solution demonstrates how conversational AI can directly access live operational data and generate actionable requests such as emergency RFQs.

    Read source aws.amazon.com
  3. AWS Machine Learning BlogFirst-party

    AtScale Builds a Dimensional Semantic Layer for Agentic AI on AWS

    AtScale introduces a dimensional semantic model on AWS, replacing graph‑based layers used in earlier posts. The model, defined in the open Semantic Modeling Language (SML) and versioned in Git, is exposed via SQL, XMLA, and the Model Context Protocol (MCP). A Strands agent on Amazon Bedrock AgentCore queries this layer, with identity and role‑based access control propagated through AgentCore’s identity features. The solution is deployed on Amazon EKS using AtScale’s Helm chart.

    Read source aws.amazon.com
  4. AWS Machine Learning BlogFirst-party

    Strategy Mosaic Enables Agentic AI on AWS by Providing a Unified Semantic Layer for Bedrock AgentCore

    Strategy Mosaic is used as a semantic layer on AWS to support agentic AI via Amazon Bedrock AgentCore. The solution models a Customer‑360 domain across Amazon Redshift and Amazon Aurora with two linked models, avoiding ETL. Mosaic’s dimensional model exposes a Model Context Protocol that lets a Strands agent reach the semantic layer, while row‑level and object‑level security propagate end‑to‑end. The same governed definitions can be shared with Power BI, Tableau, notebooks, and custom apps through JDBC/ODBC, XMLA, and REST.

    Read source aws.amazon.com
Oct 20Tue
  1. AWS Machine Learning BlogFirst-party

    Deepgram Boosts Speech‑to‑Text Efficiency 11‑70% Using AWS Trainium2

    Deepgram ported its Nova‑3 speech‑to‑text model to AWS Trainium2, deploying eight replicas on a single trn2.3xlarge EC2 instance. The optimized configuration processed 11–70 % more audio per dollar on batch transcription workloads compared to previous compute options. The team used TorchNeuron for native PyTorch integration, performed kernel‑level tuning, and built a Rust‑based data‑parallel runtime to eliminate Python overhead. These gains enable Deepgram to serve more audio for the same infrastructure budget and lower costs for customers who self‑host on SageMaker AI.

    Read source aws.amazon.com
Oct 2Fri