Test Engineer – AI Voice Systems (m/f/d)

• Build and own the AI evaluation engine: discrete (e.g. 1–6) and continuous (0–100%) scoring with confidence values and partial scores, efficient evaluation via embeddings and cosine similarity, with selective LLM-as-judge only where it earns its cost and latency.
• Define and maintain the quality metric set for an automotive assistant: hallucination rate, answer faithfulness and relevance, intent-fulfilment, latency P95, BLEU / WER / CER — and defend their thresholds as release criteria.
• Develop generative test case and dialogue generation: LLM-based test case creation per domain (navigation, media, knowledge, car control) with configurable volume, complexity, scenario types and edge-case coverage; automatic generation of test descriptions, expected dialogue flows, success criteria and evaluation prompts.
• Make generated dialogue realistically human: hesitations, filler words, incomplete sentences and colloquial speech instead of rigid scripts; generative multi-turn conversations where follow-up turns are produced dynamically from the assistant's actual answer.
• Engineer test variability and reliability: randomised test parameters (language, voice characteristics, speaking rate, audio characteristics, dialects), complexity variants (short / nominal / complex) that preserve semantic intent, and automatic retry with clean separation of infrastructure failures from functional defects.
• Build prompt and model regression suites in LangFuse so prompt changes, model version upgrades and backend releases are caught before they reach vehicles — including version-over-version comparison and trend analysis.
• Contribute to audio simulation and multi-speaker testing: realistic background noise (engine, road, music, phone calls, sirens, horns, passenger conversation), mixed-language input, multi-speaker simulation from different seat positions with position-specific evaluation prompts, and comparison of synthetic against real native-speaker recordings.
• Cover conventional test engineering alongside the AI layer: unit, integration and system tests across domain, adapter and infrastructure layers (pytest, testcontainers), automated smoke and release tests, and the supplier speech-stack test suites.
• Support test execution in real vehicles and on the test rack (TISAX Level 3 environment), including device health checks, failover behaviour and triage of infrastructure versus assistant defects.
• Deliver reporting and analytics that non-engineers can act on: test statistics by batch and device, exports (PDF/CSV/Excel), version comparisons, performance trends and regression detection between software releases.
• Maintain traceability between test cases, requirements and defects, including versioned test cases with change history and diff views, and integration with the client's test specification and requirements management systems.
• Coordinate with the backend, frontend, AI and hardware teams and with the client's departments; report quality status to management in a form that supports go/no-go decisions.

Must have

• 5+ years in test automation / quality engineering with strong Python, including ownership of a test framework rather than only writing cases in someone else's.
• Hands-on experience evaluating LLM or other non-deterministic systems: building evaluation pipelines, LLM-as-judge patterns, embedding/cosine-similarity scoring, and reasoning about scoring reliability and inter-rater agreement.
• Practical command of LLM quality metrics — faithfulness, hallucination rate, relevance, plus at least one of BLEU / WER / CER — and the judgement to know what each does and does not prove.
• Experience with LangChain / LangGraph or equivalent, and with an LLM observability tool (LangFuse, Phoenix, Langsmith or comparable) for tracing, prompt versioning and evaluation runs.
• Prompt regression testing in practice: detecting quality drift across prompt edits and model version changes.
• Solid conventional test engineering: pytest, code coverage, test case design across positive, negative, boundary and regression cases; testcontainers or equivalent integration test tooling.
• Experience integrating ASR and TTS services (Azure Speech, Whisper, Cerence, Google TTS or equivalent), including streaming/real-time behaviour.
• CI/CD integration of automated test suites (Azure DevOps Pipelines, GitHub Actions) and comfort with Docker; working knowledge of REST APIs and SQL/PostgreSQL for test data and results.
• English C1, and willingness to work onsite in the Ingolstadt/Munich area, including hands-on test sessions in vehicles and on the test rack.

Nice to have

• German B2 or above — strongly preferred; project language is German and test specifications and defect discussions often are too.
• Automotive voice assistant and vehicle interface experience; familiarity with in-vehicle test benches, test racks or HiL environments.
• Audio engineering: background noise modelling, multi-speaker and mixed-language audio, speaking-rate and dialect variation, synthetic versus human speech comparison.
• Frontend skills for reporting and review tooling (Vue / React / Angular): dashboards, data visualisation, PDF/CSV/Excel export, browser-based remote test control.
• Test device and network infrastructure: remote device fleets, registration and health monitoring, failover mechanisms, reservation/scheduling systems.
• Multi-language and localisation testing, including configurable wake-words and language-specific evaluation criteria.
• Awareness of TISAX, A-SPICE, ISO/SAE 21434 and of EU AI Act documentation expectations for AI system testing.
• Experience evaluating safety-relevant AI output where a wrong answer is more than a quality defect.

We offer

Other