TRM Labs
Senior or Staff AI Infrastructure Engineer
š° USD 200,000 - 275,000/yr
Job Description
[AI-summarized by JobStash]
You will design, build, and operate the infrastructure that supports large-scale AI and agent systems. You will create reusable CI/CD workflows for training, evaluation, and deployment; automate model versioning, approvals, and compliance checks; and assemble modular stacks including vector databases, feature stores, and model registries. You will integrate and evaluate cutting-edge LLM tools, instrument observability and monitoring, and deploy online and offline evaluation pipelines with regression testing, cost monitoring, and human-in-the-loop workflows. You will collaborate with engineers and data scientists to embed models and agents into real-time applications, provide sandboxes and reproducible environments for researchers, and continuously improve model performance, reliability, and governance.
Requirements
- āWrite high quality maintainable software primarily in Python
- āStrong background in scalable infrastructure including Docker and Kubernetes
- āExperience with infrastructure as code and deployment tools such as Terraform and CI/CD pipelines
- āFamiliarity with monitoring and logging frameworks such as Datadog Prometheus and OpenTelemetry
- āKnowledge of MLOps best practices including model versioning rollback strategies automated evaluation and drift detection
- āExperience with scalable model and agent serving infrastructure such as vLLM Triton and BentoML
- āExperience deploying and maintaining LLM and agentic workflows in production including cost latency and performance monitoring
- āExperience capturing traces for analysis debugging and optimizing prompt response flows with real time data access
- āStrong ownership pragmatism and ability to balance infrastructure design with iterative delivery
Responsibilities
- āBuild reusable CI/CD workflows for model training evaluation and deployment
- āAutomate model versioning approval workflows and compliance checks
- āBuild modular scalable AI infrastructure including vector databases feature stores model registries and observability tooling
- āPartner with engineering and data science to embed AI models and agents into real-time applications and workflows
- āContinuously evaluate and integrate state of the art AI tools and frameworks
- āDrive AI reliability and governance to ensure compliance security and uptime
- āEnsure data accuracy consistency and reliability for model training and inference
- āDeploy infrastructure to support offline and online evaluation including regression testing cost monitoring and human in the loop workflows
- āEnable researchers with sandboxes dashboards and reproducible environments
- āImprove AI and ML model performance
Benefits & Perks
- āRemote work
- āEligibility to participate in TRM's equity plan