Package source contexts, conditions, references, and instructions

Folder contract#

Every top-level source folder represents one comparable item. Media files inside that folder are conditions applied to the same source or context. A text-to-video Dataset might place several model outputs under one prompt folder; a compression Dataset might place several encodes under one source clip.

Every media bundle source folder must contain at least one condition. A pairwise task needs at least two usable conditions for each sampled source. Prompt-only CSV imports intentionally contain no media; they are stored as catalog records for future workflows and cannot currently launch a Dataset-bound generation or finished-media Evaluation Task. File extensions must match the bytes they contain. Hidden files, path traversal, symbolic links, executable content, and ambiguous duplicate names are rejected during import.

References, anchors, and filename patterns#

Reference and anchor files are identified by Dataset-level patterns. The defaults are reference.*, low.*, medium.*, and high.*, but an operator can configure different safe patterns before publication. These files are not silently included as ordinary candidate conditions.

A reference supplies context for fidelity or alignment review. Anchors represent known points on a quality scale and are most useful for protocols such as ACR or MUSHRA. The Evaluation Task decides which supported role each file plays in the reviewer interface.

For source-conditioned video tasks such as Ref2V, I2V, and V2V, the source image or video is part of the prompt input. It is always shown to the evaluator. Setting presentation.show_reference to false hides only optional comparison references and quality anchors; it cannot hide source-conditioning input media.

Evaluator media is delivered through an assignment-scoped opaque relay URL. The browser does not receive the upstream URL, private Storage path, filename, model identity, or provider configuration.

Per-item configuration#

An optional config.json can set the item title, question, and description. Description entries may be text or a simple key-value object; each entry is rendered as a separate, readable block. The file can also define markers for audio or video review.

Validate config files against https://docs.genmedia.dev/dataset-config.schema.json before upload. Finalization still applies authoritative path, media, secret, marker, and modality checks that JSON Schema cannot express.

Configuration is treated as data, not executable code. Unknown or unsafe fields are rejected, secrets are never accepted, and filenames are not used as trusted model identities. Candidate-to-model lineage must be supplied explicitly when a task is created.

Evaluator languages and important prompt requirements#

Dataset import records sourceLanguage and an initial targetLanguages list. The source language identifies the text already present in config.json or prompt-only CSV rows. A Dataset operator can later add, re-enable, or disable up to eight targets from Dataset settings without changing any published Dataset version. Adding Italian, for example, transactionally creates durable work for every item in every published version; future versions inherit the active language set. Disabling a target affects only new Sessions and unfinished jobs, never completed copies or existing Session pins.

Publication freezes the original prompt first. Source emphasis and selected translations then run asynchronously in a bounded, retryable queue. Every result is tied to the immutable Dataset version, item position, source fingerprint, provider, model, and translation contract version. A completed copy is never rewritten in place; a changed contract creates a different version.

An Evaluation Task records a default language. Before starting, each reviewer can choose from the Dataset languages whose exact copy and important-part analysis are ready for every item. The selected locale is pinned immutably to that Session and reused on resume and replay. Only an older Session with no Session-language contract may use its Task-time presentation for backward compatibility. Partial or mixed-language prompts are never assembled. The assignment localization object reports requestedLocale, resolvedLocale, sourceLocale, status, translationVersion, and ready.

Important requirements are stored as validated character spans for the exact displayed copy, never as generated HTML. The evaluator sees a pale red semantic mark with a dark-red label icon, border, and underline. Screen readers receive an Important requirement prefix, so importance never depends on color alone.

Draft, upload, validation, and publication#

1. 1

   Create an import with its name, purpose, retention, reference patterns, content warnings, sourceLanguage, and explicit targetLanguages. Media modality is inferred from validated files.

2. 2

   For larger media, choose a folder and upload its files directly to private Storage through resumable 6 MiB chunks. For compatibility, a .zip, .tar, .tar.gz, .tgz, or prompt-only .csv can use the bounded bundle target.

3. 3

   Finalize the import. GenMedia verifies registered byte totals and private object metadata, validates paths, media signatures, roles, configuration, markers, and source groups, then publishes one immutable Dataset version transactionally. Localization jobs are ensured idempotently after publication and never block the Dataset from reaching ready.

4. 4

   Inspect the published source rows, or correct the bundle and create a new import after a terminal validation error. A retry of a ready import returns the same Dataset version.

Read the structure preview before uploading#

Folder import inspects the selected hierarchy locally before any media is transferred. The preview reports source folders, condition files, pair-ready sources, config files, total bytes, and source-specific fixes. This is a convenience check only: authoritative media-signature, object-size, secret, path, and configuration validation still runs on the server.

A green preview means every detected source has at least two condition files. A warning means the Dataset can still be valid for a single-stimulus protocol but one or more sources cannot form a pair. An error disables publication until the displayed structure problem is corrected.

Common import errors and how to fix them#

• File outside a source folder

  Move it under source-name/file.ext. The optional outer Dataset folder is removed automatically; additional nested folders are not inferred.

• Missing candidate condition

  Add at least one non-reference media file. Pairwise Evaluation Tasks need at least two conditions for every sampled source.

• Duplicate condition

  Condition filename stems must be unique inside one source. Rename the repeated file instead of relying on path or extension differences.

• MIME mismatch

  The extension, declared MIME type, Storage metadata, and file signature disagree. Export the media again with the correct extension; renaming bytes is not enough.

• Invalid config or markers

  Validate config.json against the public schema and keep only title, question, description, tags, and bounded marker fields.

• Unsafe path or secret material

  Remove traversal segments, hidden credentials, environment files, signed URLs, API keys, executable content, and unsupported JSON files.

Import limits#

Resumable folder import accepts at most 10 GiB across 5,000 files and 2,000 source folders, with a 5 GiB per-file limit and at most 16 MiB of config.json data in total. Each file is registered before upload, written directly to a private staging bucket, verified against Storage size, MIME metadata, and media magic bytes, and copied server-side to immutable private media before publication.

Archive compatibility mode remains intentionally bounded at 200 MiB compressed, 192 MiB after expansion, and 64 MiB per entry. Prompt-only CSV is also bounded by 64 MiB. A 10 GiB archive is not supported: use folder import so the application never buffers or expands a multi-gigabyte archive. URL-manifest ingestion is not currently available.

Main, reviewer practice, and Golden Control data#

The main Dataset supplies the real evaluation items. A Reviewer Practice Dataset teaches the interface and criterion before counted work. A Golden Control Dataset contains approved answer keys for attention or calibration checks. Practice and Golden Control data are optional and remain separate from the model result.

This use of Reviewer Practice is not model training. It prepares a reviewer to use the protocol consistently. Golden answers and item-level correctness stay restricted and are never included in a normal preference export.