English (Australia) Audio Generalist Evaluator Expert

Mercor is seeking an English (Australia) Audio Generalist Evaluator Expert to contribute to a high-impact AI audio research project with a leading research laboratory.

In this role, you will work on transcription, annotation, evaluation, and quality assurance tasks that help train and benchmark advanced language models.

This is a structured short-term engagement designed for candidates with strong analytical, academic, linguistic, or research backgrounds who are fluent in Australian English and enjoy transforming complex audio and visual content into precise, well-structured written outputs.

Job Responsibilities

Transcribe and Optimize Audio & Video

• Listen to, analyze, and transcribe audio and video content in Australian English

• Produce high-quality written outputs in English

• Ensure:

  • Accuracy
  • Clarity
  • Consistency
  • Compliance with formatting requirements

• Capture linguistic nuances including:

  • Tone
  • Intent
  • Formal and informal register
  • Regional expressions
  • Slang
  • Contemporary Australian English usage

Define and Document Evaluation Standards

• Establish standards for high-quality AI responses

• Develop:

  • Evaluation rubrics
  • Grading guidelines
  • Review standards

• Document linguistic edge cases and language-specific considerations

• Identify:

  • Grammatical patterns
  • Colloquialisms
  • Cultural references
  • Regional language variations

specific to Australian English.

Conduct Model Testing and Grading

• Test AI-generated outputs

• Evaluate responses based on:

  • Accuracy
  • Completeness
  • Fluency
  • Instruction following
  • Clarity

• Provide structured feedback to improve model performance

Support Benchmarking and Quality Assurance

• Participate in:

  • QA reviews
  • Benchmark creation
  • Evaluation cycles

• Ensure consistency before datasets are integrated into official benchmarks

• Collaborate with project teams to improve task design and resolve ambiguities

Minimum Qualifications

• Strong writing, editing, and critical thinking skills

• Ability to work independently and meet deadlines

• Native or near-native fluency in Australian English

• Strong familiarity with:

  • Australian English
  • Regional vocabulary
  • Australian accents
  • Slang and colloquialisms
  • Contemporary language usage

• Ability to accurately transcribe and analyze English-language audio content

• Availability of 10–20 hours per week

Preferred Qualifications

• College student or recent graduate

• Background in:

  • Linguistics
  • Humanities
  • Social Sciences
  • Journalism
  • Translation
  • Localization
  • Technical disciplines

• Experience with:

  • Transcription
  • Annotation
  • Localization
  • AI evaluation
  • Research workflows

• Familiarity with differences between:

  • Australian English
  • New Zealand English
  • Other major English varieties

• Interest in:

  • Artificial Intelligence
  • Language Models
  • Applied Research

Application Process

1. Complete a short AI-led interview (approximately 15 minutes)
2. Successful candidates will be onboarded and invited to begin project work

About Mercor

Mercor partners with leading AI laboratories and enterprises to train frontier AI models using human expertise.

Contributors work directly on projects that improve advanced AI systems while collaborating with researchers and domain experts across multiple fields.