Перейти к основному содержимому
Innodata

Applied Research Scientist, LLM Evaluation & Post-Training

УдалённоCanada только
Опубликовано
Роль
AI / ML
Опыт
Синьор
Размер компании
Крупная
CAD 245k–CAD 315k/yr
Проверьте доступность

Доступно для: CA only. Укажите, откуда вы работаете, чтобы проверить доступность.

Коротко по делу

Senior applied ML/AI research scientist in Canada with 5+ years in LLMs or foundation models. Must know LLM evaluation, benchmarking/alignment/post-training, experimental design, statistics, and Python; MS/PhD required, PhD strongly preferred.

Ключевые навыки

PythonLLM evaluationPost-training

Обязательные навыки

PyTorchHugging FaceJAXTensorFlow

Чем предстоит заниматься

  • Define and execute a research agenda focused on LLM evaluation and post-training, especially evaluation-driven model improvement
  • Design rigorous experiments to study how evaluation methodologies impact fine-tuning and post-training outcomes
  • Develop and validate evaluation frameworks for LLM and multimodal systems, including benchmark/task design, scoring methods, judge/model-assisted evaluation, human evaluation protocols, and robustness/stress testing
  • Lead research on advanced evaluation domains, including long-context, cross-modal, and dynamic multi-turn evaluations
  • Study the effectiveness and limitations of existing evaluation techniques, and propose improved methodologies with clear validity and scalability tradeoffs
  • Analyze model behavior and failure patterns; generate actionable recommendations for model improvement and evaluation redesign
  • Collaborate with AI/ML Research Engineers to translate research methods into scalable evaluation and post-training pipelines
  • Collaborate with Language Data Scientists to integrate human-in-the-loop and synthetic data/evaluation strategies into research programs
  • Engage with customer technical stakeholders to understand evaluation goals, review methodologies, and provide expert recommendations
  • Contribute to internal benchmark datasets, evaluation frameworks, and reusable research assets
  • Produce high-quality technical documentation, internal research reports, and client-facing materials explaining methods, results, assumptions, and limitations
  • Contribute to thought leadership and best practices in LLM evaluation, post-training, and GenAI quality measurement

Что требуется

  • MS/PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative scientific field
  • PhD strongly preferred
  • 5+ years of relevant experience in applied research / research science in ML/AI, with substantial work in LLMs or foundation models
  • Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research
  • Strong foundation in experimental design, statistical analysis, and scientific reasoning for ML systems
  • Strong coding skills in Python for research experimentation and analysis, e.g. data processing, evaluation pipelines, statistical analysis, visualization
  • Experience working with modern ML tooling/frameworks, e.g. PyTorch, Hugging Face, JAX/TensorFlow as applicable, sufficient to design and execute model/evaluation experiments
  • Ability to evaluate and compare human and automated evaluation methods, including tradeoffs in cost, reliability, validity, and scalability
  • Experience designing evaluation studies and protocols that are reproducible across datasets, model versions, and evaluation runs
  • Ability to collaborate directly with technical stakeholders including research scientists, ML engineers, data scientists, and customer technical counterparts
  • Strong communication skills and ability to present nuanced technical conclusions, assumptions, and limitations clearly

Innodata (Nasdaq: INOD) is a global data engineering company providing data, evaluation frameworks, human expertise, solutions, platforms, and services for Generative AI / AI builders and adopters.

🇺🇸 Соединенные ШтатыAI DataКрупнаяinnodata.com/

Что говорят о компании

4.1/ 5

Детали

Способ откликаGreenhouse
CAD 245k–CAD 315k/yr