Skip to main content
REMOTERA
Back to feed
Mindrift

Technical AI Evaluator - Python, Automation or Security

RemoteUnited States only
Published
Role
QA
Experience
Senior
Employment
Contract
up to $50/hr
Check eligibility

Open to US only. Set where you work from to check your eligibility.

No BS summary

Senior software developer (5+ years) to design evaluation tasks and tests for AI coding agents. Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis. Project-based contract work, English B2+ required; US role.

Core skills

PythonFastAPI

Required skills

JavaScriptTypeScriptReactDockerPostgreSQLKafkaRedis

Optional languages

English B2+

What you'll do

  • Build realistic developer environments — a virtual company with codebase, infrastructure, and context (tickets, docs, conversations).
  • Design tasks from intermediate states of these environments: craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent.
  • Write tests that verify agent solutions — accept all valid approaches and reject incorrect ones, neither too strict nor too lenient.
  • Iterate on tasks and tests based on QA feedback: review agent solutions, analyze failures, and refine until the evaluation is fair and robust.
  • Create evaluation criteria and datasets to measure AI coding agent performance on real-world developer tasks.

What they require

  • 5+ years in software development.
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
  • Experience writing tests (functional, integration).
  • English proficiency — B2+

Benefits

  • Up to $50/hr equivalent.
  • Flexible schedule; set your own schedule.
  • Paid per task/project; tasks estimated at ~20 hours each.

Mindrift

Toloka AI supports frontier model post-training by building domain-specific reinforcement learning environments, tasks, and evaluation frameworks designed by real practitioners. Mindrift, powered by Toloka, connects top domain experts with cutting-edge AI initiatives.

🇮🇳 IndiaAI

What people say about this company

4.0/ 5

Details

Posting languageEnglish
up to $50/hr