QA Engineer

Andersen is hiring a QA Engineer to ensure quality and stability of a secure cloud platform with collaboration and analytics features, supporting continuous improvements and reliable product delivery. 

The customer is an international company delivering professional and technology-enabled solutions that support effective collaboration, structured communication, and operational efficiency for organizations. It operates in a fast-growing environment, focusing on scalability, security, and continuous improvement while developing digital platforms used by diverse clients worldwide. 

The project is focused on enhancing a secure, cloud-based board management platform with intuitive meeting tools, real-time collaboration, and advanced analytics. It also includes building and maintaining scalable AI infrastructure, orchestration patterns, and observability to ensure reliable and intelligent platform performance. 

Responsibilities: 

- Designing test cases for non-deterministic AI systems. 

- Executing evaluation runs against AI workflows using LangFuse.  

- Validating AI outputs against acceptance criteria and business rules.  

- Building and maintaining regression test suites for AI features.  

- Identifying edge cases and failure patterns in AI behavior.  

- Triaging and documenting accuracy/quality issues.  

- Monitoring production quality metrics and flag degradation.  

- Partnering with AI Engineer on test data curation and quality thresholds. 

Must-have: 

- Experience as a QA Engineer or in a similar role for 3+ years. 

- Experience testing AI/ML or other non‑deterministic systems, with an understanding of probabilistic outputs and variability in model behavior.  

- Solid knowledge of QA methodologies adapted for AI workflows, including evaluation‑based testing, scenario testing, and semantic validation.  

- Hands‑on experience designing test cases for systems without fixed expected outputs, using acceptance criteria, heuristics, and quality thresholds. Experience executing evaluation runs with 

- AI observability or evaluation tools (experience with LangFuse is a strong advantage).  

- Ability to validate LLM outputs against business rules, prompt requirements, safety guidelines, and product acceptance criteria.  

- Experience building and maintaining regression suites for AI features (prompt regressions, dataset-based regressions, workflow regressions).  

- Ability to identify edge cases, emergence patterns, and failure modes in AI behavior (e.g., hallucinations, inconsistency, bias, context loss).  

- Experience documenting issues with detailed repro steps, evaluation evidence, logs, and accuracy/quality metrics.  

- Ability to work with JSON, API tools (Postman, Swagger), and logs from AI systems. 

- Level of English – from Upper-Intermediate and above. 

Nice to have: 

- Exposure to evaluation frameworks (LangFuse evals, Ragas, TruLens, or internal evaluators).   

- Understanding of key LLM quality metrics such as accuracy, precision/recall, semantic similarity scores, or custom evaluation metrics.  

- Familiarity with production monitoring dashboards or telemetry tools to detect quality degradation.  

- Basic understanding of LLMs, embeddings, RAG workflows, and prompt-based systems.  

- Experience with test data curation, labeling, or annotation processes. 

 Reasons why this job would be interesting to you: 

- Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc.. 

- The opportunity to change the project and/or develop expertise in an interesting business domain. 

- Job conditions – you can work both fully remotely and from the office or can choose a hybrid variant. 

- Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee. 

- The opportunity to earn an additional up to 1,000 USD per month by participating in the company's activities. 

- Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated. 

- Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies). 

- Certification compensation (AWS, PMP, etc). 

- Referral program. 

- English courses. 

- Private health insurance and compensation for sports activities. 

Join us! 

Your personal data is protected in accordance with GDPR regulations. Learn more: https://andersenlab.com/privacy-policy/pl

https://people.andersenlab.com/