Ray: Distributed Python and ML

By Khimananda Oli | Last reviewed: August 2026

Scaling Python workloads often hits a wall where single-machine memory or compute becomes the bottleneck, forcing teams to rewrite logic for specialized frameworks. Ray: Distributed Python and ML solves this by providing a unified runtime that scales standard Python code from a local laptop to multi-node GPU clusters with minimal refactoring. Whether you are fine-tuning LLMs, running batch inference, or processing terabytes of data, Ray abstracts the distributed systems complexity so you can focus on application logic rather than infrastructure plumbing.

What is Ray: Distributed Python and ML and how does it work?

At its core, Ray transforms your Python functions and classes into distributed tasks and actors. Unlike traditional HPC schedulers or MapReduce frameworks, Ray uses a dynamic task graph and a shared-memory object store (Plasma) to minimize serialization overhead. This architecture makes it uniquely suited for modern AI workloads that mix CPU-heavy preprocessing with GPU-accelerated training and low-latency serving.

When you initialize Ray, it starts a head node containing the Global Control Store (GCS), scheduler, and driver. Worker nodes connect to this head, registering their resources (CPUs, GPUs, memory). The GCS maintains cluster metadata and fault tolerance state, while the distributed scheduler places tasks based on resource availability and locality. For teams familiar with MLOps workflows, Ray bridges the gap between experimental notebooks and production pipelines by keeping the same API surface across environments.

This shared-memory design is critical for ML. When multiple workers need the same dataset or model weights, they read directly from local shared memory instead of copying over the network. In practice, this reduces data loading bottlenecks by 5–10x compared to frameworks that serialize everything through a central coordinator. For Nepali teams working with limited inter-node bandwidth on hybrid setups, this locality-aware execution can be the difference between a viable training run and one that stalls on I/O.

How do you set up a Ray cluster for production ML workloads?

Setting up Ray for production requires more than pip install ray. You need reproducible environments, proper resource isolation, and observability. Most production deployments in 2026 use Kubernetes via the KubeRay operator, though bare-metal and cloud VM deployments remain valid for specific compliance or cost scenarios.

Step-by-step KubeRay deployment

1. Install KubeRay Operator: Deploy the operator using Helm to manage RayCluster CRDs. Ensure your Kubernetes version supports the required API groups and that RBAC policies allow pod creation in target namespaces.
2. Define RayCluster YAML: Specify head and worker group configurations including resource requests/limits, GPU tolerations, and autoscaling bounds. Always set explicit resource limits to prevent noisy-neighbor issues.
3. Configure Object Store Size: Set --object-store-memory to ~30% of node RAM. Too small causes spilling to disk; too large risks OOM kills. On GPU nodes, account for CUDA memory separately.
4. Enable Dashboard & Metrics: Expose the Ray Dashboard via Ingress for debugging. Configure Prometheus scraping endpoints to feed into your existing monitoring stack.
5. Validate with Smoke Test: Submit a simple ray job submit that verifies GPU access, object store connectivity, and cross-node communication before running real workloads.

# Example: Submitting a distributed training job
ray job submit --address=http://ray-head-svc:8265 \
  --working-dir ./training_code \
  --runtime-env-json='{"pip": ["torch==2.5", "transformers==4.46"]}' \
  -- python train_llm.py --num-gpus=8 --dataset=s3://my-bucket/data

A common mistake is under-provisioning the head node. The head runs the GCS, scheduler, and driver—under heavy load (thousands of concurrent tasks), it becomes CPU-bound. Allocate at least 4 CPUs and 8GB RAM for the head in clusters exceeding 20 workers. For larger deployments, consider dedicated head nodes with no user workloads to ensure scheduling latency stays below 10ms.

How does Ray compare to Spark and Dask for data and ML?

Choosing between Ray, Spark, and Dask depends on workload characteristics, not hype. Each has distinct strengths rooted in their original design goals. Understanding these trade-offs prevents costly migrations later.

In practice, many organizations run both Ray and Spark. Use Spark for petabyte-scale ETL and data warehousing where SQL compatibility and catalog integration matter. Use Ray when your pipeline shifts toward model-centric operations: distributed training, hyperparameter sweeps, online serving, or reinforcement learning. Ray’s ability to handle heterogeneous resources (mixing CPU preprocessors with GPU trainers in one DAG) makes it superior for end-to-end ML pipelines where Spark would require awkward handoffs.

How do you serve LLMs and ML models with Ray Serve?

Ray Serve is a scalable serving library built on Ray’s actor system. Unlike standalone serving solutions, it integrates tightly with Ray Train and Data, enabling unified pipelines from training to deployment. For LLMs specifically, Ray Serve handles batching, request routing, and multi-model composition natively.

For LLM serving, pair Ray Serve with vLLM or TGI as the backend engine. Ray Serve handles HTTP ingress, request queuing, and adaptive batching while vLLM manages KV-cache optimization and continuous batching within each replica. This separation lets you scale horizontally (more replicas) independently of vertical optimization (better per-GPU throughput).

# Ray Serve + vLLM deployment snippet
from ray import serve
from ray.serve.llm import LLMDeployment

app = LLMDeployment.bind(
    model_id="meta-llama/Meta-Llama-3.1-8B-Instruct",
    tensor_parallel_size=2,
    max_num_seqs=256,
    gpu_memory_utilization=0.9
)

serve.run(app, host="0.0.0.0", port=8000)

Set max_concurrent_queries carefully. Too high causes OOM during traffic spikes; too low wastes GPU idle time. Start with 2x your expected peak QPS and tune based on p99 latency SLOs. Always enable health checks and graceful shutdown hooks—LLLM containers take 30–60 seconds to drain, and premature kills cause dropped requests.

What are best practices for optimizing Ray performance at scale?

Performance issues in Ray usually stem from misconfigured resources, excessive serialization, or poor task granularity. Follow these battle-tested patterns to avoid common pitfalls:

• Right-size tasks: Tasks should take 100ms–10s. Shorter tasks incur scheduling overhead; longer tasks reduce fault tolerance granularity. Batch micro-tasks using @ray.remote(max_calls=100) or vectorized operations.
• Use object refs, not values: Pass ObjectRef between tasks instead of materializing results. This keeps data in shared memory and avoids redundant serialization. Only call ray.get() at the final aggregation point.
• Pin GPU memory explicitly: Set torch.cuda.set_per_process_memory_fraction() to prevent CUDA OOM when multiple actors share a GPU. Ray’s resource tags don’t enforce hard memory limits—only scheduling constraints.
• Enable placement groups: For multi-GPU training, use placement groups with STRICT_PACK strategy to co-locate workers on the same node. Avoid SPREAD for NCCL workloads—cross-node communication will kill throughput.
• Monitor object store spilling: If ray memory shows frequent disk spills, increase object store size or reduce intermediate result retention. Spilling turns fast shared-memory reads into slow disk I/O, negating Ray’s primary advantage.

For teams managing sensitive data, remember that Ray’s object store is unencrypted in transit between nodes. If operating in regulated environments (banking, healthtech in Nepal), deploy Ray within a VPC with mTLS enabled or use encrypted storage backends for intermediate artifacts. Audit trails should capture job submissions and resource allocations—integrate with your logging pipeline early to satisfy compliance requirements without retrofitting.

Making Ray Production-Ready for Your Team

Ray: Distributed Python and ML delivers genuine value when treated as infrastructure, not just a library. Invest time in proper cluster provisioning, observability integration, and security hardening before scaling workloads. Start with non-critical batch jobs to build operational familiarity, then graduate to training and serving. Document your runtime environments, resource quotas, and failure runbooks—future you (and your on-call team) will thank you. If you’re evaluating Ray for your organization’s ML platform or need help designing a compliant, cost-efficient deployment, reach out to discuss your specific requirements.