Browse documentationExperiment types
Choose the judgment before configuring the job
An Experiment type defines what one reviewer sees and submits. GenMedia separates this protocol from the Dataset and from the Job that staffs or generates work. Launchable types are validated end to end; planned catalog entries remain visibly unavailable until their evaluator, result method, and controls are implemented.
| Type | Reviewer action | Current state |
|---|---|---|
| Pairwise image, video, or audio | Choose between two blind conditions, optionally with reference media | Available |
| ACR image, video, or audio | Rate one condition on an absolute category scale | Catalog only |
| Rubric image, video, or audio | Rate one condition across explicit criteria | Catalog only |
| MUSHRA | Grade several audio conditions against reference and anchors | Catalog only |
| Rank audio | Order several audio conditions | Catalog only |
| Real or generated | Make one binary authenticity judgment | Catalog only |
| Survey | Answer structured questions after inspecting media | Catalog only |
Type, Data, Experiment, and Job
- 1
Type: choose the media modality and judgment protocol.
- 2
Data: bind the main immutable Dataset version and optional Reviewer Practice and Golden Control versions.
- 3
Experiment: set the reviewer-facing title, question, instructions, criteria, reference behavior, and playback controls.
- 4
Job: choose the saved rater cohort, session target, comparisons per Session, sampling plan, budget, and launch state.
Pairwise video controls
- Presentation
Flip between candidates or show them side by side; sequential or synchronized playback is explicit.
- Decision
Allow or disallow ties and skips without changing the blind candidate order.
- Reference
Show a separate reference when fidelity matters, and decide whether a reference condition is excluded from candidate sampling.
- Playback
Set minimum display and play duration, audio state, loop, seeking, fit or crop behavior, and an optional transition mask.
- Markers
Render Dataset marker annotations on supported audio and video controls.
Job settings do not redefine the Experiment
A Job controls how many Sessions to collect and how comparisons are sampled. It does not change the Dataset, question, or scoring semantics. Starting another Job under the same frozen task preserves the Experiment contract and records a separate operational run.
Generation jobs are provider requests that produce candidates. Evaluation staffing jobs allocate reviewer Sessions. They are different operational records even when the interface presents both under one Evaluation Task.
Why unsupported cards stay disabled
A familiar name is not enough to ship a protocol. MUSHRA, ranking, authenticity, and survey tasks need specific assignment rules, reviewer controls, exports, and statistical interpretation. GenMedia exposes their intended place in the catalog without allowing an operator to launch a task whose evidence pipeline is incomplete.