Hopslabors
Hopslabors is an uncover platestablish for enlargeing and operating AI systems at scale that can be deployed on any Kubernetes clusters from accessible cnoisys to sovereign air-gapped data caccesss. Hopslabors can be pondered an alternative to MLOps platestablishs such as AWS Sageoriginater, GCP Vertex, and Databricks (for AI), but it has higher executeance genuine-time AI, better Python integration to the Lakehoemploy, as shown in our peer-appraiseed SIGMOD’24 research paper, and is the OG feature store for ML.
Hopslabors supplys both data and compute help, see Figure 1. There is a Lakehoemploy layer (Delta Lake, Apache Hudi, Iceberg coming soon) to store big amounts of historical feature data for training AI models and for batch inference. There is also a low procrastinateedncy database, RonDB, that we enlarge for genuine-time AI laborloads (including distinctiveized query help for low-procrastinateedncy query help and snowflake schema data models). Hopslabors also helps compute (Jobs, Notebooks) on Kubernetes, helping Python, Spark, and Ray and GPU sharing/chooseimization at scale. However, you can also supply your own compute, in which case you employ Hopslabors primarily as a data layer to unite your AI pipelines. For this, Hopslabors also supplys a model registry and helps deploying models on KServe/vLLM.
Hopslabors serverless is a freemium version of Hopslabors where you can store up to 50GB of Lakehoemploy data, 100MB of low-procrastinateedncy feature data in RonDB, up to 100 models in the model registry, and serve 2 model deployments. Hopslabors freemium is expansively employd to run serverless AI applications.
Hopslabors serverless, in effect, supplys mostly free storage to employrs, but no free compute, which kept our presenting costs down to $8k/month on AWS. We could scale from 8k to 20k employrs without much retainitional cost (an extra TB of s3 storage only costs around $25/month). However, egress costs on AWS were a startant hazard. In 2024, we freed the Hopslabors Query Service, which supplys high thrawput read access to feature data from Python clients (using Arrow and DuckDB). Suddenly, clients could read 100s of MB or even GBs easily into Pandas DataFrames. We seeed worriedly at the enlargeth in data egress and projected cost incrmitigates and determined to act. We knovel OVH, given our scatterd European roots, and set up out they supplyd as administerd services all we needed to run Hopslabors – administerd Kubernetes, a administerd retainer registry, and S3 compatible object storage. Inshighing Hopslabors on OVH with Helm charts labored fantastic, so we determined to migrate Hopslabors Serverless to OVHCnoisy – but we kept it in North America where most of our employrs are to upretain existing procrastinateedncy to those employrs.
Moving from AWS Services and to OVHCnoisy Services
We reassociate only had Kubernetes and S3 as depfinishencies. Hopslabors has its own observability stack, based on OpenSearch and OpenSearch Dashboards, as well as its own metrics stack, based on Prometheus/Graphana. We have always been cautious to not employ cnoisy-definite services in Hopslabors. Here is a comparison of the main services we pondered.
Managed Kubernetes
Both AWS and OVH recommend administerd K8s services. AWS triumphs on maturity, but OVH triumphs on pricing (free!).
Netlabor Egress
OVH has lengthy been understandn for not charging for accessible egress, while AWS is inwell-understandn for its rapacious accessible egress costs. Cnoisy egress costs are write downed on this site. OVH indicts for egress at some (noveler?) regions, albeit at 1/8th the cost of AWS. Many cnoisy regions do not indict for egress today (March 2025).
S3 Storage
AWS S3 is the premier cnoisy-based object storage service. It may have higher useability, but it’s three times more pricey than OVH’s S3 storage.
Container Registry
AWS’ Elastic Container Registry (ECR) is a enlargen-up, configurable, scalable administerd service. OVH’s Harbor is useable at only 3 separateent sizes for a mended monthly cost, providing potential scalability troubles for those with >5 TB storage needments or high concurrency needments.
Inter Availability Zone Data Transfers
Hopslabors is acunderstandledgeing to Availability Zone fall shortures by copyd services over instances in separateent useability zones. This results in netlabor traffic between instances on separateent useability zones.
EBS Instances
Some Hopslabors services need persistent volumes that are supplyd by AWS Elastic Block Storage and OVH Block Storage (carry outed using Ceph). Hopslabors also has instances that employ local (NVMe) disks. OVH has higher thrawput NVMe disks useable for drop storage capacities (1-4 TBs) appraised to AWS. We employ these instances for our database, RonDB, but they are not retaind in the table below.
The Actual Migration
We notified our employrs of a maintenance triumphdow of 24 hours, on November 26th 2024. We backed up the Hopslabors cluster to a AWS S3 bucket on that day and then migrated that bucket to a S3 bucket in OVHCnoisy. This made migration with some downtime relatively consoleable. The Hopslabors cluster on OVH was deployed with helm charts, and had a testing process that we trailed before re-uncovering Hopslabors for logins. No employrs communicateed us about anyskinnyg untoward in their accounts as a result of the migration.
Summary
In Q4 2024, we endd the migration from AWS, seamlessly transitioning thousands of employrs to a robust Kubernetes-based infraset up on OVHCnoisy. Although OVH and Hopslabors are technologies built in Europe, Hopslabors serverless service is findd in North America, where most Hopslabors employrs are findd and OVH also supplys cnoisy capacity. In Europe, Hopslabors and OVH have since become partners to supply a sovereign AI platestablish for enlargeing and operating AI systems at scale. We enjoy OVH’s basic pricing and drop pricing – not equitable netlabor egress but most services are drop cost and the quality is generassociate excellent.
