Skip to content

HyperPod

SageMaker HyperPod is a standing fleet of accelerated instances for ML training and inference. It checks each node’s health, including deep checks on its GPUs, replaces nodes that fail, and stays up between jobs. Where ECS and EKS run application containers, HyperPod runs distributed training.

Clusters, instance groups and lifecycle scripts

Section titled “Clusters, instance groups and lifecycle scripts”

A cluster is the fleet. It’s divided into instance groups, each a set of identical instances with one type, count, role and lifecycle script: the script every node runs when it boots, which installs the scheduler, mounts shared storage and wires up monitoring. A cluster usually has a controller group and one or more worker groups.

Something has to decide which job runs on which nodes. That’s the orchestrator, and a cluster has one of two. Slurm, the default, is the classic HPC scheduler: you submit jobs with sbatch from a login node. EKS makes the nodes part of a Kubernetes cluster you attach, and you run Deployments and Jobs on them.

Choose Slurm for HPC-style training queues. Choose EKS when your platform is already Kubernetes, or to share the fleet between teams with task governance: a scheduler policy of priority classes, and compute quotas that reserve capacity per team.

In Alchemy, the fleet is SageMaker.Cluster. Governance is ClusterSchedulerConfig and ComputeQuota. On EKS, plain Kubernetes.Deployment and Kubernetes.Job workloads run on HyperPod nodes by naming the group’s nodeSelector, and the quota’s namespace and queueName.

Create a Slurm cluster

Section titled “Create a Slurm cluster”

A Slurm cluster needs three things: an S3 bucket holding the lifecycle script, an execution role the nodes assume, and the cluster itself. The script must exist in S3 before the cluster creates — a deploy-time Action uploads it, and the data flow (bucket → script → cluster) orders the deploy:

const bucket = yield* AWS.S3.Bucket("LifecycleScripts", {
  forceDestroy: true,
});


const script = yield* UploadLifecycleScript({
  bucketName: bucket.bucketName,
});


const role = yield* AWS.IAM.Role("HyperPodInstanceRole", {
  assumeRolePolicyDocument: {
    Version: "2012-10-17",
    Statement: [
      {
        Effect: "Allow",
        Principal: { Service: "sagemaker.amazonaws.com" },
        Action: ["sts:AssumeRole"],
      },
    ],
  },
  managedPolicyArns: [
    "arn:aws:iam::aws:policy/AmazonSageMakerClusterInstanceRolePolicy",
  ],
});


const cluster = yield* AWS.SageMaker.Cluster("TrainingCluster", {
  instanceGroups: {
    controller: {
      InstanceType: "ml.t3.medium",
      InstanceCount: 1,
      ExecutionRole: role.roleArn,
      LifeCycleConfig: {
        SourceS3Uri: script.sourceS3Uri,
        OnCreate: script.onCreate,
      },
    },
  },
});

Instance groups reconcile in place: change a count or add a group and the cluster updates; remove a group from instanceGroups and it is deleted from the cluster. Changing the VPC replaces the cluster. Provisioning takes ~5 minutes for small CPU groups and 10–25 minutes for large GPU groups.

The full working stack is examples/aws-hyperpod.

Submit Slurm jobs

Section titled “Submit Slurm jobs”

Slurm has no remote submission API — jobs are submitted on the cluster. Every node is an SSM target named sagemaker-cluster:<cluster-id>_<instance-group>-<instance-id>:

Terminal window
aws sagemaker list-cluster-nodes --cluster-name <clusterName>
aws ssm start-session \
  --target sagemaker-cluster:<cluster-id>_controller-<instance-id>
# then, on the node:
sbatch --nodes=4 train.sbatch

Create an EKS-orchestrated cluster

Section titled “Create an EKS-orchestrated cluster”

Under EKS orchestration, HyperPod nodes join an EKS cluster you provide and workloads are ordinary Kubernetes objects. The API enforces a few constraints, all encoded in the example:

  • The EKS cluster must use the API (or API_AND_CONFIG_MAP) authentication mode — EKS’s own CONFIG_MAP default is rejected.
  • HyperPod trails the newest Kubernetes version; pin one it supports (1.28–1.35 today).
  • LifeCycleConfig is required for EKS-orchestrated instance groups too.
  • The HyperPod dependencies Helm chart (aws/sagemaker-hyperpod-cli) must be installed on the EKS cluster before the HyperPod cluster attaches — SageMaker validates it. The example fetches the chart with an Action and applies it with Kubernetes.HelmChart.
const eks = yield* AWS.EKS.Cluster("Orchestrator", {
  roleArn: eksRole.roleArn,
  version: "1.34",
  resourcesVpcConfig: { subnetIds: network.privateSubnetIds },
  accessConfig: {
    authenticationMode: "API",
    bootstrapClusterCreatorAdminPermissions: true,
  },
});


const hyperpod = yield* AWS.SageMaker.Cluster("HyperPod", {
  orchestrator: { Eks: { ClusterArn: eks.clusterArn } },
  vpcConfig: {
    SecurityGroupIds: [clusterSecurityGroupId],
    Subnets: network.privateSubnetIds,
  },
  instanceGroups: {
    workers: {
      InstanceType: "ml.t3.medium",
      InstanceCount: 1,
      ExecutionRole: role.roleArn,
      LifeCycleConfig: {
        SourceS3Uri: script.sourceS3Uri,
        OnCreate: script.onCreate,
      },
    },
  },
});

HyperPod nodes must live in private subnets — the Network helper with nat: "single" builds a suitable VPC.

Run workloads on HyperPod nodes

Section titled “Run workloads on HyperPod nodes”

HyperPod nodes are ordinary EKS nodes carrying well-known labels. Kubernetes.Deployment and Kubernetes.Job opt onto them with each instance group’s nodeSelector attribute — health-checked nodes of that group. The instance-group keys carry through to the cluster’s attributes as types, so the reference is typed per key (a typo’d group name is a compile error) and the workload is connected to the fleet through the resource graph:

const train = yield* Kubernetes.Job(
  "Train",
  {
    cluster: eks,
    main: import.meta.url,
    podTemplate: {
      spec: {
        nodeSelector: hyperpod.instanceGroups.workers.nodeSelector,
      },
    },
  },
  Effect.gen(function* () {
    return {
      run: Effect.gen(function* () {
        // training / eval logic; bindings land IAM on pod identity
      }),
    };
  }),
);

For anything the typed surface doesn’t cover — a Kubeflow PyTorchJob, a custom operator — apply a raw Kubernetes.Manifest pinned by the node labels directly:

nodeSelector: {
  "sagemaker.amazonaws.com/node-health-status": "Schedulable",
  "sagemaker.amazonaws.com/instance-group-name": "workers",
},

Task governance

Section titled “Task governance”

Task governance arbitrates the fleet between teams. It ships as the amazon-sagemaker-hyperpod-taskgovernance EKS add-on (Kueue under the hood), a scheduler policy with priority classes, and per-team compute quotas:

yield* AWS.EKS.Addon("TaskGovernance", {
  clusterName: eks.clusterName,
  addonName: "amazon-sagemaker-hyperpod-taskgovernance",
});


// One policy per cluster — a second create fails with the typed
// ClusterSchedulerConfigAlreadyExists error.
const policy = yield* AWS.SageMaker.ClusterSchedulerConfig("Scheduler", {
  clusterArn: hyperpod.clusterArn,
  schedulerConfig: {
    PriorityClasses: [
      { Name: "inference", Weight: 100 },
      { Name: "training", Weight: 75 },
    ],
    FairShare: "Enabled",
  },
});


// Creates the hyperpod-ns-research namespace + its Kueue queue.
const quota = yield* AWS.SageMaker.ComputeQuota("ResearchQuota", {
  clusterArn: hyperpod.clusterArn,
  computeQuotaTarget: { TeamName: "research", FairShareWeight: 10 },
  computeQuotaConfig: {
    ComputeQuotaResources: [{ InstanceType: "ml.t3.medium", Count: 1 }],
  },
});

A workload submits through governance by referencing the quota’s attributes — the governed namespace and the team’s Kueue queue — which also orders the workload after the quota:

namespace: quota.namespace, // hyperpod-ns-research
labels: {
  [AWS.SageMaker.KUEUE_QUEUE_NAME_LABEL]: quota.queueName,
  [AWS.SageMaker.KUEUE_PRIORITY_CLASS_LABEL]: "training-priority",
},

Where next

Section titled “Where next”