Job Details

This is a remote, hourly-paid contractor role where you will interact with a speech-based AI model through short, natural conversations and then evaluate the models performance. Youll speak to the model in everyday scenarios (e.g., acting as a customer while the model plays a flower shop) for approximately 5 minutes per session, then answer structured questions about response quality, accuracy, helpfulness, and conversational naturalness. SME Careers is a fast-growing AI Data Services company and a subsidiary of SuperAnnotate, providing AI training data for many of the worlds largest AI companies and foundation-model AI labs. Your work directly helps improve cutting-edge speech and conversational AI systems through rigorous testing and clear, high-quality feedback. Key Responsibilities: Conduct Speech-Based Scenario Conversations: Speak with the AI model in guided real-world use cases (e.g., shopping, customer support, making appointments), maintaining a natural conversation for ~5 minutes per session. Evaluate Model Performance: Answer follow-up questions assessing the models quality (e.g., accuracy, relevance, clarity, politeness, and ability to complete the task). Provide High-Quality Written Feedback: Write concise, specific notes on what worked well and what failed (e.g., misunderstandings, incorrect assumptions, awkward phrasing, conversation breakdowns). Follow Rubrics and Guidelines: Apply evaluation criteria consistently across sessions and adhere to task instructions. Your Profile: Must be currently living in the United States. Fluent in English (spoken, reading, and writing). Strong reading and writing skills with the ability to provide clear, detailed feedback. Comfortable speaking naturally and clearly on microphone for short sessions. Detail-oriented, consistent, and able to follow task instructions and evaluation rubrics. Reliable internet connection and access to a quiet environment for speaking tasks.