We are seeking an experienced LLM Quality Assurance Test Engineer to evaluate, validate, and enhance our generative AI platforms specifically for our Experience applications. These role is critical for ensuring our generative AI platforms provide accurate, helpful, and magical experiences across all our Experience touchpoints, maintaining our high standards for service and satisfaction.
Responsibilities:
Design and execute comprehensive test strategies for Large Language Models
Develop sophisticated prompting techniques to evaluate LLM responses about Disney Experience offerings
Create test cases that validate accurate knowledge of parks, resorts, and cruise operations
Identify potential vulnerabilities and edge cases in guest-facing LLM interactions
Document and track model behaviors, issues, and improvements
Collaborate with AI/ML teams to improve model performance and safety
Basic Qualifications:
3+ years of experience in Quality Assurance, with at least 1 year focused on AI/ML systems
Demonstrated expertise in LLM evaluation methodologies and prompt engineering
Strong understanding of LLM limitations, biases, and potential failure modes
Excellent analytical and problem-solving skills
Strong documentation and communication abilities
Technical Skills:
Experience with LLM testing frameworks and evaluation metrics
Knowledge of prompt engineering best practices
Understanding of AI safety and ethical considerations
Familiarity with version control systems and bug tracking tools
Basic programming skills for test automation (Python preferred)
Key Competencies:
Creative problem-solving for edge case discovery
Attention to detail in identifying subtle model behaviors
Strong analytical thinking for systematic testing approaches
Ability to think like both a guest and an adversary
Excellence in cross-functional collaboration
Compensation:
The estimated pay range for this position is USD $80.00/hr - 88.00/hr and is an Exempt role.
Exact compensation and offers of employment are dependent on circumstances of each case and will be determined based on job-related knowledge, skills, experience, licenses or certifications, and location.
Benefits:
We offer comprehensive benefit options which vary depending on role, location, and employment type. The Talent Acquisition Partner can share more details about compensation or benefits for the specific role during the hiring process.