Mindrift
Senior Software Engineer - Testing AI Coding Agents
RemotePoland only
- Role
- QA
- Experience
- Senior
- Employment
- Contract
up to $50/hr
Check eligibility
Open to PL only. Set where you work from to check your eligibility.
No BS summary
Senior software engineer to design and evaluate tasks for AI coding agents. Requires 5+ years software development and core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis. Project-based contracting role; English B2+ required.
Core skills
PythonFastAPIReact
Required skills
JavaScriptTypeScriptDockerPostgreSQLKafkaRedis
Required languages
English B2+
What you'll do
- Build realistic developer environments (virtual company with codebase, infrastructure, tickets, docs, conversations) that form a believable development history.
- Design tasks from intermediate states of these environments: craft the prompt, define success criteria, ensure solvability by an AI agent.
- Write tests that verify agent solutions: accept valid approaches and reject incorrect ones without being too strict or too lenient.
- Iterate on tasks and tests based on QA feedback: review agent solutions, analyze failures, and refine evaluations until fair and robust.
What they require
- 5+ years in software development.
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
- Experience writing tests (functional, integration).
- English proficiency - B2+.
- Please submit your CV in English and indicate your level of English proficiency.
Benefits
- Flexible schedule — you choose when and how to work.
- Project-based paid work; tasks estimated at ~20 hours each.
- Compensation up to $50/hr (equivalent), depending on level and pace.
Toloka AI supports frontier model post-training by building domain-specific reinforcement learning environments, tasks, and evaluation frameworks designed by real practitioners. Mindrift, powered by Toloka, connects top domain experts with cutting-edge AI initiatives.
🇮🇳 IndiaAI
up to $50/hr