Senior AI Quality Analyst

Svitla Systems Inc. is looking for a Senior AI Quality Analyst for a full-time position (40 hours per week) in Europe.

The AI Quality Analyst is a critical role responsible for ensuring the performance, safety, and reliability of our cutting-edge AI/ML models. You will be at the forefront of our development lifecycle, designing and executing comprehensive evaluation strategies to identify model weaknesses, potential biases, and critical edge cases. This role requires a blend of analytical rigor, technical aptitude, and deep curiosity about how AI models behave in real-world scenarios. You will not only find bugs but also provide actionable insights that drive model improvement and guide our research and development efforts.

Responsibilities

• Evaluation Strategy & Benchmark Development: Create and manage a comprehensive set of test cases and evaluation benchmarks. Identify potential failure points, including edge cases, adversarial inputs, and bias sources.

• Error Analysis & Failure Triage: Perform systematic error analysis to categorize failures and find underlying patterns. Triage defects, prioritize by severity and impact, and collaborate with the development team for resolution.

• Data Sourcing & Curation: Gather, curate, and manage high-quality datasets for model testing and evaluation. Conduct data annotation and validation to maintain data integrity.

• Exploratory & Adversarial Testing (Red Teaming): Conduct unscripted exploratory testing to detect unexpected model behaviors. Engage in red teaming exercises to challenge models and uncover safety and security vulnerabilities.

• Test Environment Management: Set up, maintain, and troubleshoot testing environments to ensure reliable evaluation pipelines.

• Reporting & Insights: Analyze test results and create clear, actionable reports for technical and non-technical audiences. Convert complex findings into tangible recommendations for model improvements.

• Process Improvement: Participate in review sessions and help enhance testing methodologies, tools, and quality assurance processes.

Qualifications and Skills

• Proven experience in quality assurance, testing, or data analysis, preferably in the AI/ML domain.

• A deep understanding of the machine learning lifecycle and common failure modes of AI models.

• Hands-on experience with data annotation, data validation, and large dataset management.

• Meticulous attention to detail and a methodical approach to problem-solving.

• Strong analytical skills, with the ability to identify patterns in data and draw meaningful conclusions.

• Excellent communication skills, with the ability to clearly document bugs and articulate complex technical issues.

• Experience with bug-tracking systems (e.g., Jira) and test case management tools.

Nice to have

• Basic scripting skills (e.g., Python) for test automation and data manipulation.

• Familiarity with computer vision or other specific AI domains relevant to our work.