Metadata harmonisation: align spoken-data fields with UniDive naming conventions
Back to Cantonese HK · Back to index
Repo: https://github.com/UniversalDependencies/UD_Cantonese-HK
Cross-posting from the UniDive WG1 T1.5 (spoken language guidelines) metadata harmonisation review. We compared UD_Cantonese-HK’s current CoNLL-U metadata against the proposed naming conventions (see also the full treebank status table). This is a suggestion for maintainers to review - feel free to push back on anything that doesn’t fit the corpus. The comparison was carried out semi-automatically with the help of Claude (Anthropic); errors or misunderstandings are possible, so please double-check anything unclear.
1. Document-level (naming conventions)
No document_id exists in the current .conllu (the field this section previously listed, _filename, is not actually present - that suggestion was stale). However, the repo’s own README documents exactly 4 distinct sources by sent_id range, each with a title, source URL, and fluency note:
sent_id range |
Proposed document_id |
Title | Fluency |
|---|---|---|---|
| 1-410 | missing_days |
Missing days / 小時光 (film) | mostly prepared dialogue |
| 411-547 | tempo_in_temple |
Tempo in Temple / 廟眾樂樂 (film) | spontaneous interview |
| 548-650 | what_day_is_today |
What day is today / 今日星期幾 (film) | mostly prepared |
| 651-1004 | legco_president_election_2016 |
Legislative Council meeting (2016-10-12) | spontaneous discussion |
| Field | Suggestion |
|---|---|
| — | add # document_id at sentences 1, 411, 548, and 651, using the proposed ids above (please confirm the slugs) |
| — | add # genre (film / legislative-proceedings) and degree_of_spontaneity (planned for the 3 films, unplanned for the legislative discussion and the “Tempo in Temple” interview) per document |
2. Speaker-level (naming conventions)
| Field | Suggestion |
|---|---|
| — | no speaker metadata exists; the legislative-council portion (sentences 651-1004) clearly involves multiple speakers - could speaker_id be recovered and tagged? |
Implementation notes
Quick search & replace
- None.
Needs a small script
sent_idis a plain sequential integer (1,2, …1004, confirmed by dry inspection of the clone) with no separator to derive document boundaries from -harmonize_metadata.py derive-document-iddoesn’t apply here since it needs a regex-extractable prefix. Instead, insert# document_id(and# genre/# degree_of_spontaneity) at the four fixedsent_idboundaries (1, 411, 548, 651) with a short bespoke script (~15 lines: iterate sentences, insert a comment block whensent_idmatches one of the four boundary values). Happy to write this once the id slugs and genre/spontaneity values are confirmed.
Needs manual input from maintainers
- Confirm the four proposed
document_idslugs (missing_days,tempo_in_temple,what_day_is_today,legco_president_election_2016). - Confirm
genreanddegree_of_spontaneityvalues per document. - Whether
speaker_idcan be recovered for the legislative-council portion (sentences 651-1004) - needs source material, not a mechanical transform.
This issue was prepared as part of the UniDive WG1 T1.5 spoken language guidelines effort. Happy to help implement these changes ourselves if that’s easier than doing it on your end - just let us know.