Ruby Labs
AI Engineer
USA, CanadaRemotePosted 21 days ago$130,000 – $220,000
Full TimeSeniorRemoteCA
See how this job matches your profile
Sign in for an AI-powered fit score, breakdown, and a tailored resume.
Job Description
About usRuby Labs is a leading tech company that creates and operates innovative consumer products. We offer a diverse range of opportunities across the health, education, and entertainment industries
Key Highlights
- Advanced Prompt Engineering: Designing complex, dynamic prompt templates with conditional logic and efficiently reusing information and context within prompts to maximize generation quality and reasoning.
- Structured Outputs & Schemas: Implementing various response schemes (JSON mode, function calling, Zod/JSON schemas) to ensure AI outputs are predictable and ready for seamless integration into application logic.
- Prompt Engineering & Evaluations: Building robust evaluation pipelines and using Langfuse to collect feedback and score the quality of responses in real time.
- Tracing & Debugging: Performing deep debugging of complex LLM chains using Langfuse traces to identify bottlenecks and optimize for cost, latency, and context window usage.
- AI A/B Testing: Running systematic experiments across different models via OpenRouter (e.g., comparing Claude 3.5 Sonnet vs. GPT-4o) and analyzing results based on quantitative metrics.
Qualifications
Required Qualifications
- Node.js & Next.js: Deep knowledge of the stack to build reliable services and handle complex LLM-generated data.
- Dynamic Prompting Skills: Proven experience in building prompts where content is highly dependent on input variables and context injection.
- OpenRouter Experience: Experience working with unified APIs, managing rate limits, and selecting the most cost-effective models for specific tasks.
- Langfuse (or similar): Understanding of LLM observability principles — setting up tracing, creating test datasets, and integrating scoring systems.
- Evaluation Methodology: Experience with frameworks like RAGAS or building custom “LLM-as-a-judge” systems.
- Analytical Mindset: Ability to transform raw generation logs into actionable business metrics and technical insights.
- Iterative Mindset: Focus on continuous product improvement through constant feedback loops.
- Node.js & Next.js: Deep knowledge of the stack to build reliable services and handle complex LLM-generated data.
- Dynamic Prompting Skills: Proven experience in building prompts where content is highly dependent on input variables and context injection.
- OpenRouter Experience: Experience working with unified APIs, managing rate limits, and selecting the most cost-effective models for specific tasks.
- Langfuse (or similar): Understanding of LLM observability principles — setting up tracing, creating test datasets, and integrating scoring systems.
- Evaluation Methodology: Experience with frameworks like RAGAS or building custom “LLM-as-a-judge” systems.
- Analytical Mindset: Ability to transform raw generation logs into actionable business metrics and technical insights.
- Iterative Mindset: Focus on continuous product improvement through constant feedback loops.
Preferred Qualifications
- Fine-Tuning: Practical experience in fine-tuning models for specific domain tasks or JSON compliance.
- RAG Architecture: Understanding how to build and optimize Retrieval-Augmented Generation systems, including indexing, retrieval, and re-ranking.
- Python: Basic knowledge for working with data science scripts or AI evaluation libraries.
- Fine-Tuning: Practical experience in fine-tuning models for specific domain tasks or JSON compliance.
- RAG Architecture: Understanding how to build and optimize Retrieval-Augmented Generation systems, including indexing, retrieval, and re-ranking.
- Python: Basic knowledge for working with data science scripts or AI evaluation libraries.
Skills & Technologies
Node.jsNext.jsTypeScriptPythonRubyGo
About the Company
Ruby Labs
View company profile →
Interested in this role?
Sign in or create a free account to see how this job matches your skills, apply with one click, and let our AI tailor your resume.
Sign in to applyAI-powered resume optimization
Save and track your applications
Job Details
Employment Type
Full Time
Experience Level
Senior
Salary Range
$130,000 – $220,000
Location
USA, Canada
Work Mode
Remote
Posted
21 days ago
Country
CA