Test Automation Engineer (Python+genAI)

We're searching for a Test Automation Engineer.

Your responsibilities:

• Generative AI Testing Strategy: Design and implement comprehensive testing strategies specifically tailored for LLM-based applications, including evaluation of model responses, RAG pipelines accuracy, and overall system reliability
• Quality Assurance Framework Development: Utilize Client's testing frameworks that address both traditional software quality aspects and AI-specific concerns such as output consistency, contextual accuracy, and ethical compliance; co-create and maintain such frameworks when required
• Test Automation Development: Design and implement automated testing solutions for continuous evaluation of LLM applications, including integration tests, performance tests, and specialized AI behavior tests
• Edge Case Analysis: Identify and develop test scenarios for edge cases in LLM behavior, including handling of ambiguous inputs, potential biases, and unexpected response patterns
• Bias and Fairness Testing: Design and execute tests to identify potential biases in model outputs and ensure fair treatment across different user groups and use cases
• Security Testing: Collaborate within development teams to test for potential vulnerabilities specific to LLM applications, including prompt injection, data leakage, and other AI-specific security concerns
• Test Documentation: Create and maintain comprehensive test documentation testing strategy, test cases, and testing guidelines specific to AI applications and compliant with Client practices; document the analysis of requirements and risks and develop tests accordingly
• Performance Testing: Collaborate with other engineers to conduct thorough performance testing of GenAI applications, including response time analysis, load testing, and resource utilization monitoring

We are looking for you, if you have:

• Experience: 3+ years of experience in software testing, with at least 1 year focused on genAI applications
• Technical skills: Strong proficiency in Python and testing frameworks (e.g., pytest, unittest); experience with AI testing tools and frameworks
• AI/ML knowledge: Understanding of LLM architectures, RAG systems, and common failure modes in AI applications; familiarity with evaluation metrics for generative AI
• Test Automation: Experience with test automation frameworks and CI/CD pipelines; ability to design and implement automated testing solutions for AI applications
• Ability to analyze requirements, identify test scenarios, and prioritize testing activities based on risk and impact
• Cloud Platforms: Practical experience with testing applications on cloud platforms (AWS, Azure) and working with cloud-based AI services
• Monitoring tools: Experience with monitoring and observability tools, log analysis, and performance metrics tracking (e.g. DataDog, Grafana)
• ALM tools: Experienced in ALM tools (e.g. Jira, GitLab, ServiceNow)
• Security Testing: Understanding of security testing methodologies, particularly in the context of AI applications
• Very good practical and theoretical knowledge of testing processes (ISTQB Advanced, etc.)
• The successful candidate should also hold B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent degree with strong background in testing methodologies
• Be able to communicate in English at the level of C1+

We offer:

• Participation in interesting and demanding projects
• Flexible working hours
• A great, non-corporate atmosphere
• Stable employment conditions (contract of employment or B2B contract)
• Opportunity to develop your career

We reserve the right to contact the selected candidates.