Metadata harmonisation: align spoken-data fields with UniDive naming conventions
Back to English ESLSpok · Back to index
Repo: https://github.com/UniversalDependencies/UD_English-ESLSpok
Cross-posting from the UniDive WG1 T1.5 (spoken language guidelines) metadata harmonisation review. We compared UD_English-ESLSpok’s current CoNLL-U metadata against the proposed naming conventions (see also the full treebank status table). This is a suggestion for maintainers to review - feel free to push back on anything that doesn’t fit the corpus. The comparison was carried out semi-automatically with the help of Claude (Anthropic); errors or misunderstandings are possible, so please double-check anything unclear.
1. Document-level (naming conventions)
No document_id exists, but sent_id already encodes document structure: e.g. file01243.txt_145 is document file01243.txt, sentence 145. 872 distinct documents across 2320 sentences (up to 22 sentences per document). Like CHILDES, the sentences are shuffled - consecutive sentences jump between documents at random. Per the README, this is “a random sample of sentences” from a spoken L2 English interview corpus, so each document is one interview session, only partially sampled here.
| Field | Suggestion |
|---|---|
| — | derive # document_id from the sent_id prefix (before _<number>); recompose by sorting within each prefix by that trailing number |
2. Speaker-level (naming conventions)
| Field | Suggestion |
|---|---|
| — | no speaker metadata exists; each document is one L2 English speaker’s interview session - could speaker_id be derived from the same filename once document_id is introduced? |
Implementation notes
- Needs a small script: deriving
# document_idfrom thesent_idprefix is mechanical and verified clean (0 sent_ids fail to match, run against all three release files):python3 workgroups/spoken-data/scripts/harmonize_metadata.py derive-document-id DIR --pattern '^(?P<doc>.+)_\d+$' --write(dry-run againsten_eslspok-ud-{dev,test,train}.conlluderives 198/200/808 doc ids respectively - the 872 distinct-document figure in the draft is the union across all three splits, since a document’s sentences can be split across dev/train/test, so the count is correctly done per-file). This only inserts the id; it does not recompose sentence order (see manual item below). - Needs manual input from maintainers: recomposing “shuffled” sentences into their original per-document order needs the true source ordering key, which isn’t in the current fields (unlike CHILDES, there’s no visible
original_sent_idequivalent here) - could you point us to what determines original order, or confirm the corpus should stay as one-sentence-per-doc-order-unknown? Same forspeaker_id: whether it can simply be derived from the (soon-to-exist)document_idneeds a maintainer confirmation before scripting it (trivialrename-comment-style copy once confirmed).
This issue was prepared as part of the UniDive WG1 T1.5 spoken language guidelines effort. Happy to help implement these changes ourselves if that’s easier than doing it on your end - just let us know.