Data Scientist Quality Assurance Lead (QAL)

Job Summary

As a Data Scientist Quality Assurance Lead (QAL), you will oversee quality, consistency, and contributor performance across data science and machine learning AI training projects.

This role involves reviewing AI-generated data science content, evaluating trainer and QA outputs, maintaining quality standards, and ensuring training data aligns with project requirements and client expectations.

Your work will directly contribute to improving advanced AI systems by ensuring training data is statistically valid, analytically sound, reproducible, and clearly explained.

Key Responsibilities

Quality Monitoring

• Spot-check data science tasks and QA outputs
• Identify recurring quality issues and methodological gaps
• Provide actionable written feedback and escalate critical concerns

Technical Review

• Review and evaluate:

  • Data science explanations
  • Python code
  • R code
  • SQL queries
  • Machine learning workflows
  • Statistical analyses
  • Dashboards
  • Experiment designs
  • Analytical reasoning

• Assess work for:

  • Statistical validity
  • Methodological rigor
  • Reproducibility
  • Data reasoning
  • Metric interpretation
  • Business-context awareness
  • Instruction adherence

Trainer & QA Communication

• Communicate updates regarding:
  • Guidelines
  • Workflow changes
  • Data science standards
  • Quality expectations

Contributor Support

• Answer questions involving:
  • Statistical assumptions
  • Model selection
  • Validation methods
  • Data leakage
  • Metrics
  • Reproducibility
  • Coding approaches
  • Rubric interpretation

Activation Management

• Follow up with inactive contributors
• Track engagement and participation
• Report contributor availability concerns

Documentation & Onboarding

• Create and maintain:

  • Style guides
  • Documentation
  • FAQs
  • Examples
  • Calibration tasks
  • Onboarding materials

• Conduct onboarding and training sessions for contributors

Risk & Quality Review

• Identify and flag:
  • Data leakage
  • Flawed assumptions
  • Invalid statistical methods
  • Incorrect metrics
  • Weak methodologies
  • Hallucinated libraries or APIs
  • Misleading conclusions
  • Non-reproducible workflows

Process Improvement

• Improve QA workflows and review processes
• Identify recurring quality gaps and implement corrective actions

Required Qualifications

• Bachelor's, Master's, or PhD in:
  • Data Science
  • Statistics
  • Computer Science
  • Machine Learning
  • Mathematics
  • Economics
  • Engineering
  • Related quantitative fields

or equivalent professional experience.

• Strong English communication skills

• Minimum 3 years of experience in:

  • Data science
  • Analytics
  • Machine learning
  • Statistical modeling
  • Experimentation
  • Data engineering
  • Technical review
  • Data science education

• Strong understanding of:

  • Statistics
  • Probability
  • Data cleaning
  • Exploratory data analysis
  • Feature engineering
  • Supervised learning
  • Unsupervised learning
  • Model evaluation
  • Experiment design
  • Regression
  • Classification
  • Clustering
  • Validation techniques

• Ability to identify:

  • Data leakage
  • Incorrect metrics
  • Weak methodology
  • Invalid assumptions
  • Non-reproducible code
  • Hallucinated tools and libraries

Preferred Qualifications

• Experience with:

  • Python
  • pandas
  • NumPy
  • scikit-learn
  • SQL
  • Jupyter
  • matplotlib
  • R
  • Spark
  • MLflow

• Familiarity with:

  • Git
  • GitHub
  • Cloud platforms
  • Data platforms
  • Dashboarding tools

• Experience leading:

  • Data scientists
  • Analysts
  • Trainers
  • Reviewers
  • QA teams

• Experience with:

  • AI training
  • Data annotation
  • LLM evaluation
  • Rubric-based technical review

Why Join

• Help improve leading AI systems through data science quality assurance
• Flexible remote schedule
• Weekly payments
• Referral rewards and community incentives
• Access to future opportunities through SME Careers' expert network

Selection Process

1. AI Interview
2. Domain-Specific Assessment
3. Recruiter Interview

Important Note

There is currently no active project for this role. Qualified candidates will be added to the expert network and contacted when relevant opportunities become available.