English (US) Audio Generalist Evaluator Expert

Mercor is seeking an English (US) Audio Generalist Evaluator Expert to contribute to a high-impact audio AI research project with a leading research lab.

This role focuses on transcription, annotation, evaluation, benchmarking, and quality assurance tasks that help train and assess advanced language models.

The position is ideal for candidates with strong analytical, writing, and language skills who can accurately translate spoken audio and video content into structured written outputs.

Job Responsibilities

Transcribe & Optimize Audio and Video

• Listen to and analyze English (US) audio and video content

• Produce accurate transcriptions following project requirements

• Create high-quality written outputs in English

• Ensure strict adherence to:

  • Formatting guidelines
  • Style requirements
  • Project instructions

• Capture nuances including:

  • Tone
  • Intent
  • Formal and informal language
  • Regional expressions
  • Spoken American English variations

Define Evaluation Standards

• Establish expectations for high-quality model responses
• Create evaluation rubrics and grading guidelines
• Document standards to ensure reviewer consistency
• Identify:
  • Linguistic nuances
  • Grammatical complexities
  • Colloquialisms
  • Dialect variations
  • American English edge cases

Conduct Model Testing & Evaluation

• Run prompts through AI models

• Evaluate outputs for:

  • Accuracy
  • Completeness
  • Fluency
  • Instruction following

• Provide structured feedback to improve model performance

Support Benchmarking & Quality Assurance

• Participate in review and QA cycles
• Validate tasks, rubrics, and outputs before benchmark inclusion
• Collaborate with project leads to improve task design and resolve ambiguities

Minimum Qualifications

• Strong writing and editing skills

• Strong critical thinking abilities

• Ability to work independently and meet deadlines

• Native or near-native fluency in English (US)

• Strong familiarity with:

  • American English
  • Regional vocabulary
  • Accents
  • Contemporary language usage

• Ability to accurately transcribe and analyze English audio content

• Availability of 10–20 hours per week

Preferred Qualifications

• College student or recent graduate

• Background in:

  • Linguistics
  • Humanities
  • Social Sciences
  • Journalism
  • Translation
  • Localization
  • Technical disciplines

• Prior experience with:

  • Transcription
  • Annotation
  • Localization
  • Evaluation
  • Research workflows

• Familiarity with American English dialects

• Interest in AI, language models, and applied research

Application Process

1. Complete a short AI-led interview (approximately 15 minutes)
2. Selected candidates will be onboarded and invited to begin project work

Contract & Payment Terms

• Independent contractor engagement
• Fully remote work
• Flexible schedule
• Weekly payments through Stripe or Wise
• Projects may be extended, shortened, or concluded depending on business needs and performance

Important Note

Mercor currently cannot support:

• H1-B candidates
• STEM OPT candidates

About Mercor

Mercor partners with leading AI labs and enterprises to train and improve frontier AI systems using human expertise.

Contributors work directly on projects that help shape the next generation of AI technologies and language models.