{"channel":"barsukas","content":" <<< This is a genuinely interesting linguistic modeling problem. After reviewing the current data model, here are my thoughts.\r\nThe Core Problem\r\nEach Lemma currently represents one concept anchored to an English label. \"hat\" and \"cap\" are two separate lemmas, each with one translation per language. But languages carve up semantic space differently:\r\nEnglish: \"hat\" (broad brim) vs \"cap\" (fitted, no brim) \u2014 distinct\r\nLithuanian: \"skrybel\u0117\" (wide brim) vs \"kepur\u0117\" (general headwear/cap) \u2014 different boundary\r\nChinese: \"\u5e3d\u5b50\" covers nearly all headwear generically\r\nFrench: \"chapeau\" (hat, broad) vs \"casquette\" (baseball cap) \u2014 yet another split\r\nThe current model has no way to say \"these two lemmas occupy overlapping semantic territory, and in language X they might collapse into one word.\" >>>\r\n\r\n--MORE--\r\n\r\n[# Design Ideas #]\r\n <<< 1. Synonym Clusters via Existing Relation Groups (Minimal Change)\r\nAdd a new relation_type = \"synonym_cluster\" to the existing LemmaRelationGroup / LemmaRelationMember system. Group \"hat\" and \"cap\" together under concept_label \"headwear\".\r\nPros: Almost no schema change. Reuses existing infrastructure and release format (lemma_relations/synonym_cluster/headwear.jsonl).\r\nCons: Too blunt. It says \"these are related\" but doesn't capture how \u2014 it doesn't tell the app what to do in Lithuanian vs Chinese. A cluster of 5 near-synonyms doesn't tell you which ones merge in which languages.\r\n\r\n2. Per-Language Acceptable Alternatives on Translations\r\nAdd an alternatives field (JSON list) to LemmaTranslation. For \"hat\" in Chinese, the primary translation is \"\u5e3d\u5b50\", and for \"cap\" in Chinese, the primary translation is also \"\u5e3d\u5b50\" (or maybe \"\u9e2d\u820c\u5e3d\"). The alternatives field would note that \"\u5e3d\u5b50\" is acceptable for both.\r\nMore concretely, a new table like:\r\nTranslationAlternative:\r\n lemma_id \u2192 FK to Lemma\r\n language_code \u2192 str\r\n alternative_text \u2192 str (the acceptable alternative translation)\r\n source_lemma_id \u2192 FK to Lemma (which lemma \"owns\" this word natively)\r\n notes \u2192 str (why it's acceptable)\r\n\r\nSo for \"hat\" (N09_X) in Chinese, you'd have: primary = \"\u5e3d\u5b50\", no alternatives needed (it's the generic). For \"cap\" (N09_Y) in Chinese, primary = \"\u9e2d\u820c\u5e3d\", alternative = \"\u5e3d\u5b50\" (source: hat's lemma).\r\nPros: Directly solves the Trakaido use case \u2014 the app knows which answers to accept. Per-language granularity. Works well for quiz/testing scenarios.\r\nCons: Doesn't capture the why \u2014 it's a flat list of \"also accept this.\" Maintenance burden grows combinatorially. Doesn't help with teaching (explaining the distinction to learners).\r\n3. Semantic Overlap Groups with Language-Specific Merge Flags\r\nA hybrid: create synonym clusters (idea 1), but annotate per language whether the cluster's members are distinct or merged in that language.\r\nSynonymCluster:\r\n id, concept_label (\"headwear\")\r\n\r\nSynonymClusterMember:\r\n cluster_id, lemma_id\r\n\r\nSynonymClusterLanguageInfo:\r\n cluster_id\r\n language_code\r\n merge_type \u2192 \"distinct\" | \"merged\" | \"partial\"\r\n dominant_lemma_id \u2192 if merged, which lemma's translation covers the group\r\n notes \u2192 free text (\"kepur\u0117 covers both unless brimmed\")\r\n\r\nFor the headwear cluster in Chinese: merge_type = \"merged\", dominant_lemma_id = hat (because \u5e3d\u5b50 covers everything). In English: merge_type = \"distinct\". In Lithuanian: merge_type = \"partial\" with a note.\r\nPros: Captures the actual linguistic reality. The app can use merge_type to decide behavior (skip testing distinctions in languages where they're merged, accept either answer for \"partial\"). Useful for pedagogy \u2014 you can teach learners \"in Chinese, there's one word for all of these.\"\r\nCons: More complex schema. \"partial\" is vague and needs the notes field to be useful. Requires human curation per language per cluster.\r\n\r\n4. Concept Hierarchy with Language-Specific Lexicalization Levels\r\nModel a hypernym/hyponym tree:\r\nheadwear\r\n\u251c\u2500\u2500 hat (brimmed headwear)\r\n\u2502 \u251c\u2500\u2500 sun hat\r\n\u2502 \u2514\u2500\u2500 top hat\r\n\u251c\u2500\u2500 cap (fitted, no brim)\r\n\u2502 \u251c\u2500\u2500 baseball cap\r\n\u2502 \u2514\u2500\u2500 beanie\r\n\u2514\u2500\u2500 helmet\r\n\r\nThen annotate per language which level has a common word. Chinese lexicalizes at \"headwear\" level (\u5e3d\u5b50). English lexicalizes at \"hat\"/\"cap\" level. Lithuanian has its own split point.\r\nPros: Linguistically the most accurate model. Powerful for curriculum design \u2014 teach the generic word first, then the specific ones at higher difficulty. Naturally handles the \"how many words does this language need for this concept space\" question.\r\nCons: Most complex to build and maintain. Over-engineered for many cases (do you really need a tree for \"big\" vs \"large\"?). The hierarchy itself may be English-biased \u2014 who says \"headwear\" is the natural top node?\r\n\r\n5. Pragmatic Recommendation: Cluster + Alternatives (Ideas 1+2 Combined)\r\nMy actual recommendation is to combine the simplest version of two ideas:\r\nA) Synonym clusters using the existing relation group system (relation_type = \"synonym_cluster\"). This groups related lemmas and is cheap to implement. It lives in lemma_relations/synonym_cluster/ in the release format.\r\nB) Acceptable alternatives as a lightweight annotation on LemmaTranslation (a JSON field or a small sibling table). This gives the app the per-language \"accept this answer too\" data it needs.\r\nThe cluster provides the human/editorial grouping for curation. The alternatives provide the app-facing behavior. They're populated semi-independently \u2014 you might have a cluster of 4 words but only 2 of them share a translation in Lithuanian. >>>","created_at":"2026-02-09T15:14:21.411980","id":745,"llm_annotations":{},"parent_id":null,"processed_content":"

This is a genuinely interesting linguistic modeling problem. After reviewing the current data model, here are my thoughts.\r

\n

The Core Problem\r

\n

Each Lemma currently represents one concept anchored to an English label. \"hat\" and \"cap\" are two separate lemmas, each with one translation per language. But languages carve up semantic space differently:\r

\n

English: \"hat\" (broad brim) vs \"cap\" (fitted, no brim) \u2014 distinct\r

\n

Lithuanian: \"skrybel\u0117\" (wide brim) vs \"kepur\u0117\" (general headwear/cap) \u2014 different boundary\r

\n

Chinese: \"\u5e3d\u5b50\" covers nearly all headwear generically\r

\n

French: \"chapeau\" (hat, broad) vs \"casquette\" (baseball cap) \u2014 yet another split\r

\n

The current model has no way to say \"these two lemmas occupy overlapping semantic territory, and in language X they might collapse into one word.\"

\n
⎯⎯⎯⎯⎯
\n

Design Ideas \r

\n

1. Synonym Clusters via Existing Relation Groups (Minimal Change)\r

\n

Add a new relation_type = \"synonym_cluster\" to the existing LemmaRelationGroup / LemmaRelationMember system. Group \"hat\" and \"cap\" together under concept_label \"headwear\".\r

\n

Pros: Almost no schema change. Reuses existing infrastructure and release format (lemma_relations/synonym_cluster/headwear.jsonl).\r

\n

Cons: Too blunt. It says \"these are related\" but doesn't capture how \u2014 it doesn't tell the app what to do in Lithuanian vs Chinese. A cluster of 5 near-synonyms doesn't tell you which ones merge in which languages.\r

\n

2. Per-Language Acceptable Alternatives on Translations\r

\n

Add an alternatives field (JSON list) to LemmaTranslation. For \"hat\" in Chinese, the primary translation is \"\u5e3d\u5b50\", and for \"cap\" in Chinese, the primary translation is also \"\u5e3d\u5b50\" (or maybe \"\u9e2d\u820c\u5e3d\"). The alternatives field would note that \"\u5e3d\u5b50\" is acceptable for both.\r

\n

More concretely, a new table like:\r

\n

TranslationAlternative:\r

\n

lemma_id \u2192 FK to Lemma\r

\n

language_code \u2192 str\r

\n

alternative_text \u2192 str (the acceptable alternative translation)\r

\n

source_lemma_id \u2192 FK to Lemma (which lemma \"owns\" this word natively)\r

\n

notes \u2192 str (why it's acceptable)\r

\n

So for \"hat\" (N09_X) in Chinese, you'd have: primary = \"\u5e3d\u5b50\", no alternatives needed (it's the generic). For \"cap\" (N09_Y) in Chinese, primary = \"\u9e2d\u820c\u5e3d\", alternative = \"\u5e3d\u5b50\" (source: hat's lemma).\r

\n

Pros: Directly solves the Trakaido use case \u2014 the app knows which answers to accept. Per-language granularity. Works well for quiz/testing scenarios.\r

\n

Cons: Doesn't capture the why \u2014 it's a flat list of \"also accept this.\" Maintenance burden grows combinatorially. Doesn't help with teaching (explaining the distinction to learners).\r

\n

3. Semantic Overlap Groups with Language-Specific Merge Flags\r

\n

A hybrid: create synonym clusters (idea 1), but annotate per language whether the cluster's members are distinct or merged in that language.\r

\n

SynonymCluster:\r

\n

id, concept_label (\"headwear\")\r

\n

SynonymClusterMember:\r

\n

cluster_id, lemma_id\r

\n

SynonymClusterLanguageInfo:\r

\n

cluster_id\r

\n

language_code\r

\n

merge_type \u2192 \"distinct\" | \"merged\" | \"partial\"\r

\n

dominant_lemma_id \u2192 if merged, which lemma's translation covers the group\r

\n

notes \u2192 free text (\"kepur\u0117 covers both unless brimmed\")\r

\n

For the headwear cluster in Chinese: merge_type = \"merged\", dominant_lemma_id = hat (because \u5e3d\u5b50 covers everything). In English: merge_type = \"distinct\". In Lithuanian: merge_type = \"partial\" with a note.\r

\n

Pros: Captures the actual linguistic reality. The app can use merge_type to decide behavior (skip testing distinctions in languages where they're merged, accept either answer for \"partial\"). Useful for pedagogy \u2014 you can teach learners \"in Chinese, there's one word for all of these.\"\r

\n

Cons: More complex schema. \"partial\" is vague and needs the notes field to be useful. Requires human curation per language per cluster.\r

\n

4. Concept Hierarchy with Language-Specific Lexicalization Levels\r

\n

Model a hypernym/hyponym tree:\r

\n

headwear\r

\n

\u251c\u2500\u2500 hat (brimmed headwear)\r

\n

\u2502 \u251c\u2500\u2500 sun hat\r

\n

\u2502 \u2514\u2500\u2500 top hat\r

\n

\u251c\u2500\u2500 cap (fitted, no brim)\r

\n

\u2502 \u251c\u2500\u2500 baseball cap\r

\n

\u2502 \u2514\u2500\u2500 beanie\r

\n

\u2514\u2500\u2500 helmet\r

\n

Then annotate per language which level has a common word. Chinese lexicalizes at \"headwear\" level (\u5e3d\u5b50). English lexicalizes at \"hat\"/\"cap\" level. Lithuanian has its own split point.\r

\n

Pros: Linguistically the most accurate model. Powerful for curriculum design \u2014 teach the generic word first, then the specific ones at higher difficulty. Naturally handles the \"how many words does this language need for this concept space\" question.\r

\n

Cons: Most complex to build and maintain. Over-engineered for many cases (do you really need a tree for \"big\" vs \"large\"?). The hierarchy itself may be English-biased \u2014 who says \"headwear\" is the natural top node?\r

\n

5. Pragmatic Recommendation: Cluster + Alternatives (Ideas 1+2 Combined)\r

\n

My actual recommendation is to combine the simplest version of two ideas:\r

\n

A) Synonym clusters using the existing relation group system (relation_type = \"synonym_cluster\"). This groups related lemmas and is cheap to implement. It lives in lemma_relations/synonym_cluster/ in the release format.\r

\n

B) Acceptable alternatives as a lightweight annotation on LemmaTranslation (a JSON field or a small sibling table). This gives the app the per-language \"accept this answer too\" data it needs.\r

\n

The cluster provides the human/editorial grouping for curation. The alternatives provide the app-facing behavior. They're populated semi-independently \u2014 you might have a cluster of 4 words but only 2 of them share a translation in Lithuanian.

","quotes":[],"subject":"in which the machine tries to design a solution for synonyms"}