Skip to main content
Weekday AI

AI Evaluation Specialist

RemoteUnited States only
Published
Role
Research
Employment
Contract
$70/hr
Check eligibility

Open to US only. Set where you work from to check your eligibility.

No BS summary

Analytical professional with strong critical thinking and communication skills to evaluate AI-generated responses. Must identify strengths/weaknesses in reasoning and provide structured, evidence-based feedback. Ideal for detail-oriented individuals who enjoy independent, intellectually challenging tasks.

Core skills

AI evaluation

Optional skills

content evaluationresearchquality assuranceeditinganalytical reviewartificial intelligencelarge language modelsAI evaluation methodologies

Required languages

English Native-level

What you'll do

  • Evaluate AI Responses
  • Review AI-generated content for accuracy, logical reasoning, completeness, and clarity.
  • Identify factual errors, reasoning gaps, inconsistencies, and unsupported conclusions.
  • Assess responses using structured evaluation frameworks and detailed quality guidelines.
  • Provide High-Quality Feedback
  • Write clear, concise, and evidence-based rationales explaining evaluation decisions.
  • Highlight both strengths and areas for improvement in AI-generated outputs.
  • Apply consistent judgment across a wide range of evaluation tasks.
  • Maintain Evaluation Quality
  • Follow detailed project instructions and standardized assessment criteria.
  • Ensure evaluations are objective, accurate, and reproducible.
  • Complete assignments independently while maintaining high quality standards.

What they require

  • Bachelor's degree from a globally recognized university (top-ranked institutions preferred).
  • Excellent analytical thinking and problem-solving abilities.
  • Strong written communication skills with the ability to explain complex reasoning clearly and precisely.
  • Exceptional critical reading skills, including the ability to identify: Nuanced arguments Implicit meaning Logical inconsistencies Missing context Weak or unsupported reasoning
  • Strong attention to detail and ability to consistently apply structured evaluation guidelines.
  • Ability to work independently and manage assigned tasks efficiently.
  • Native-level English fluency.
  • Ability to produce thoughtful, objective, and well-supported written evaluations under defined quality standards.

Benefits

  • Contribute to the development of next-generation AI technologies.
  • Help improve the reasoning, accuracy, and reliability of advanced AI systems.
  • Work on intellectually engaging projects with real-world impact.
  • Collaborate indirectly with leading AI researchers through high-quality evaluation work.
  • Weekly payments processed through supported payment platforms.

Weekday AI is a Y Combinator-backed startup transforming the hiring landscape with outreach tools and a database of white-collar talent, delivering recruitment solutions.

RecruitmentStartup

Details

Visa sponsorshipNo
$70/hr