Metadata harmonisation: align spoken-data fields with UniDive naming conventions
Back to Hebrew IAHLTknesset · Back to index
Repo: https://github.com/UniversalDependencies/UD_Hebrew-IAHLTknesset
Cross-posting from the UniDive WG1 T1.5 (spoken language guidelines) metadata harmonisation review. We compared UD_Hebrew-IAHLTknesset’s current CoNLL-U metadata against the proposed naming conventions (see also the full treebank status table). This is a suggestion for maintainers to review - feel free to push back on anything that doesn’t fit the corpus. The comparison was carried out semi-automatically with the help of Claude (Anthropic); errors or misunderstandings are possible, so please double-check anything unclear.
1. Is the spoken portion identifiable?
This treebank mixes spoken and written material but its .conllu files don’t explicitly mark which sentences are spoken. We looked for a pattern in the data (a reasonable guess):
Finding: The entire treebank may already be spoken (transcribed Knesset/parliament proceedings), rather than a partial split.
Evidence: document_id values follow <year>_<doctype>_<id> where doctype is only ever ptv (65 docs, likely ‘protocol verbatim’) or ptm (35 docs, likely ‘protocol minutes’) - both are transcribed parliamentary speech, not a spoken/written split.
Suggestion: Rather than partially tagging, please confirm whether the whole corpus should carry # modality = spoken (as transcribed parliamentary speech), or whether ptm (minutes, possibly edited/summarized) should be excluded as not verbatim spoken language.
2. Document-level (naming conventions)
| Field | Suggestion |
|---|---|
document_id |
OK - already standard |
3. Speaker-level (naming conventions)
| Field | Suggestion |
|---|---|
speaker |
change to speaker_id |
Implementation notes
Quick search & replace
speaker→speaker_id(# speaker =→# speaker_id =).
Needs a small script
- Once the modality scope is confirmed (see below), tagging is a one-line run:
python3 workgroups/spoken-data/scripts/harmonize_metadata.py tag-modality <path> --spoken-if '^\d+_pt[vm]_'(or--spoken-if '^\d+_ptv_'only, ifptmminutes are excluded).
Needs manual input from maintainers
- Whole-corpus
# modality = spokenvs. excludingptm(minutes, possibly edited/summarized) - this decides the exact regex for the script above.