Automated QA Engineer

We are looking for an Automated QA Engineer to join our team who will focus on end-to-end quality assurance.

Your responsibilities:

● RAG Quality Evaluation: Design and curate datasets for RAG evaluation, develop and execute tests for RAG benchmarking, evaluating the accuracy and relevance of AI-generated content and information retrieval.

● Data-Driven Validation: Use data analysis techniques to evaluate test outcomes, track AI performance trends, and detect regression or drift across releases.

● Automation Test Solution Development: Design and implement automated test suites for both OutSystems front-end modules and backend microservices interacting with RaaS from the Galileo platform.

● Regression Testing: Build and maintain automated regression pipelines to ensure continuous platform stability after deployments and LLM-model updates.

● Integration Testing: Verify seamless communication between LUMIN and Galileo APIs, ensuring data integrity and correct processing across layers.

● CI/CD Integration: Embed automated tests into CI/CD pipelines (GitLab CI preferred) to enable consistent, repeatable quality gates before releases.

● Performance & Reliability Testing: Monitor and test system performance and reliability across environments, identifying performance bottlenecks or RAG degradation issues.

We are looking for you, if you have:
● Experience: 4+ years of professional experience in automated software testing, including experience with web, API, and backend systems. Strong understanding of regression and integration testing principles. Proven experience in AI-app testing, specifically with applications using LLMs and RAG.

● Automation Skills: Hands-on experience with automated testing tools (e.g., Playwright, Selenium, or OutSystems test automation frameworks). Experience developing custom test scripts in Python or JavaScript/TypeScript.

● AI Quality Testing: Experience testing LLM and RAG applications, including prompt validation, factuality testing, and output benchmarking.

● Backend & API Testing: Proven experience validating RESTful APIs, microservices, and asynchronous workflows. Familiarity with API testing tools (e.g., Postman, pytest, REST Assured).

● CI/CD & DevOps: Working knowledge of GitLab CI or equivalent systems (e.g., Github Actions), with test automation integrated into build pipelines.

We offer:
● Interesting and challenging projects

● Flexible working hours

● Friendly, non-corporate atmosphere

● Stable working conditions (CoE or B2B)

● Possibility for self-development and promotion in the company

● Rich benefits package

● Possibility to work remotely