Applied AI Engineer

At Dscout, we’re building the most flexible and powerful UX research platform on the market—trusted by the world’s top brands in finance (JP Morgan Chase, Intuit, Charles Schwab, PayPal), healthcare (Aya, Headspace), consumer goods (Keen, Verizon, Target, Northface), and tech (Google, Amazon, Facebook, Meta, Spotify, AirBnB). Our tools help teams deeply understand the humans behind their products, so they can build better ones. We are expanding our smart and driven team and would love for you to join us. AI native product is fundamentally different engineering problem than building deterministic software: the same input won't always produce the same output, and "working" means the agent behaves well across the full distribution of real world scenarios, not that it passes a fixed test suite. We're looking for an Applied AI Engineer with 2-5 years of experience building and shipping AI systems used by professionals at enterprise. You're comfortable working with modern LLM-based systems and agentic workflows, and you know how to turn powerful models into reliable product features. You have strong product judgment and think deeply about tradeoffs between LLM approaches and traditional ML when designing solutions. You care about evaluation, iteration speed, and making sure AI systems actually drive measurable business impact reliably . What you'll do Own the production improvement loop across agent behavior, customer and operator feedback, evaluation, experimentation, and verified business outcomes Instrument agent workflows so model interactions, tool use, decisions, failures, human edits, and downstream outcomes can be understood in context Define meaningful quality standards, representative evaluation datasets, regression coverage, and production monitoring. Investigate why agents underperform across context, knowledge, instructions, tools, routing, guardrails, or workflow design Design and ship targeted behavior improvements, including changes to prompting, context construction, decision logic, tool use, and human-review paths Build backend services, APIs, data models, and feedback pipelines that make agent behavior observable, steerable, and reproducible Run controlled experiments, production replays, or staged rollouts to measure whether changes improve quality and downstream business results Partner with Product, Data Science, and Sales to prioritize high-value problems and define customer and business success Ship with appropriate safeguards for privacy, security, reliability, human oversight, and safe operational rollout What you bring 2-5 years of software engineering experience, with hands-on experience building or operating LLM-powered features or agents in production - not just prototypes or demos Fluency with prompting and context engineering as an engineering discipline: you iterate on prompts, context construction, and tool definitions the way you'd iterate on code Experience building or maintaining evaluation harnesses for AI systems: offline eval sets, LLM-as-judge or human-in-the-loop scoring, regression detection Genuine comfort with non-determinism: you reason about agent behavior across a distribution of production traffic, not a fixed set of test cases, and you don't treat variance as a bug to be argued away Experience running experiments (A/B, staged rollouts, production replay) to validate whether a change actually improved outcomes, not just whether it shipped A track record of shipping features real users depended on, and owning what happened after launch A high-agency mindset: comfortable investigating an ambiguous "why is this underperforming" problem across context, tools, routing, and workflow design without a fully-scoped ticket Comfort using AI coding tools (Cursor, Claude Code, Copilot, or similar) as a real part of your workflow Nice to have Experience with voice or real-time conversational AI systems Familiarity with LLM observability/tracing tools (e.g., Braintrust, LangSmith, Datadog LLM Observability) Experience with agentic orchestration frameworks (LangChain/LangGraph or similar) Exposure to MCP-based tooling or agentic data workflows Of course, what is outlined above is an ideal set of expectations; however, business needs and other projects and tasks may shift, and additional tasks could be assigned at the discretion of your manager. If this role excites you but you're not sure you check every box, we'd still love to hear from you. Note: Some of the benefits listed below apply only to U.S.-based employees. We offer a similarly competitive benefits program in the UK with many comparable offerings, which we’ll be happy to share with you during the interview process. Of course, what is outlined above is an ideal set of expectations; however, business needs and other projects and tasks may shift, and additional tasks could be assigned at the discretion of your manager. About Dscout Dscout is a team of passionate, empathetic, and curious professionals. As a recognized leader in the Forrester Wave, we’re at the cutting edge of experience research technology. The power of research drives us - how in-context insights from real people can build more enjoyable products and services. We prioritize learning, sharing, and building. We also deeply value being a diverse and inclusive team and company and look for team members who align with that belief. Join our dynamic team and help shape product roadmaps and business strategies for the world’s most loved brands. It doesn't stop there. When you join the Dscout team, you will get: A strong and competitive compensation package with a built-in bonus and equity program. An incredible and progressive benefits package (for both you and your dependents) to support work/life balance, including flexible PTO, 15 company holidays, 12 weeks of paid parental leave, 401k match, and much more. An education stipend to support your growth & development, and a remote work stipend. A company that is open and transparent with our team. You will know what is happening and why it matters. Dscout is an equal-opportunity employer that values diversity. We do not discriminate based on identity, including race, color, religion, national origin or ancestry, sex, gender identity and expression, age, physical or mental disability, pregnancy, veteran or military status, unfavorable discharge from military service, genetic information, sexual orientation, marital status, order of protection status, citizenship status, arrest record or expunged or sealed convictions, or any other legally recognized protected basis under federal, state, or local law. If you need reasonable accommodations for any part of the employment process, please email us at accommodations@dscout.com with the nature of your request and your contact information. We'll do everything possible to ensure you're well-prepared for success during our interview process, while also upholding your privacy, including accommodating any special requests. Please note that only inquiries regarding requests for reasonable accommodation will be responded to from this email address. When you apply at Dscout, we will process your job applicant data, including your employment and education history, transcript, writing samples, and references, as necessary to consider your job application for open positions. For more information about our privacy practices, please visit our Privacy Policy . Dscout participates in the E-Verify program in certain locations, as required by law. Location Flexibility: Dscout is proud to support a remote-first workforce and enable employees to work from almost anywhere. At this time, however, we are unable to hire in the following locations: Montana, Hawaii, Alaska, and Washington DC. NOTE: DSCOUT NEVER CONTACTS JOB APPLICANTS VIA TEXT, MESSENGER, OR OTHER SIMILAR APPLICATIONS. BE AWARE OF PHISHING AND SPOOFING SCAMS, BOTH VIA TEXT AND EMAIL. ONLY RESPOND TO EMAILS FROM DSCOUT.COM