QA Engineer

Andersen is hiring a QA Engineer to ensure quality and stability of a secure cloud platform with collaboration and analytics features, supporting continuous improvements and reliable product delivery. 

The customer is an international company delivering professional and technology-enabled solutions that support effective collaboration, structured communication, and operational efficiency for organizations. It operates in a fast-growing environment, focusing on scalability, security, and continuous improvement while developing digital platforms used by diverse clients worldwide. 

The project is focused on enhancing a secure, cloud-based board management platform with intuitive meeting tools, real-time collaboration, and advanced analytics. It also includes building and maintaining scalable AI infrastructure, orchestration patterns, and observability to ensure reliable and intelligent platform performance. 

Responsibilities: 

• Designing test cases for non-deterministic AI systems. 

• Executing evaluation runs against AI workflows using LangFuse.  

• Validating AI outputs against acceptance criteria and business rules.  

• Building and maintaining regression test suites for AI features.  

• Identifying edge cases and failure patterns in AI behavior.  

• Triaging and documenting accuracy/quality issues.  

• Monitoring production quality metrics and flag degradation.  

• Partnering with AI Engineer on test data curation and quality thresholds. 

Must-have: 

• Experience as a QA Engineer or in a similar role for 3+ years. 

• Experience testing AI/ML or other non‑deterministic systems, with an understanding of probabilistic outputs and variability in model behavior.  

• Solid knowledge of QA methodologies adapted for AI workflows, including evaluation‑based testing, scenario testing, and semantic validation.  

• Hands‑on experience designing test cases for systems without fixed expected outputs, using acceptance criteria, heuristics, and quality thresholds. Experience executing evaluation runs with 

• AI observability or evaluation tools (experience with LangFuse is a strong advantage).  

• Ability to validate LLM outputs against business rules, prompt requirements, safety guidelines, and product acceptance criteria.  

• Experience building and maintaining regression suites for AI features (prompt regressions, dataset-based regressions, workflow regressions).  

• Ability to identify edge cases, emergence patterns, and failure modes in AI behavior (e.g., hallucinations, inconsistency, bias, context loss).  

• Experience documenting issues with detailed repro steps, evaluation evidence, logs, and accuracy/quality metrics.  

• Ability to work with JSON, API tools (Postman, Swagger), and logs from AI systems. 

• Level of English – from Upper-Intermediate and above. 

Nice to have: 

• Exposure to evaluation frameworks (LangFuse evals, Ragas, TruLens, or internal evaluators).   

• Understanding of key LLM quality metrics such as accuracy, precision/recall, semantic similarity scores, or custom evaluation metrics.  

• Familiarity with production monitoring dashboards or telemetry tools to detect quality degradation.  

• Basic understanding of LLMs, embeddings, RAG workflows, and prompt-based systems.  

• Experience with test data curation, labeling, or annotation processes. 

 Reasons why this job would be interesting to you: 

• Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc.. 

• The opportunity to change the project and/or develop expertise in an interesting business domain. 

• Job conditions – you can work both fully remotely and from the office or can choose a hybrid variant. 

• Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee. 

• The opportunity to earn an additional up to 1,000 USD per month by participating in the company's activities. 

• Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated. 

• Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies). 

• Certification compensation (AWS, PMP, etc). 

• Referral program. 

• English courses. 

• Private health insurance and compensation for sports activities. 

Join us! 

Your personal data is protected in accordance with GDPR regulations. Learn more: https://andersenlab.com/privacy-policy/pl

https://people.andersenlab.com/