Japanese: tokens and part of speech

Overview

This is a brief look at tokenization and part of speech assignment in Japanese.

Outcomes

examine linguistic data and identify differences in how tokens are defined across languages (ex. Japanese vs. English)
examine linguistic data and identify differences in how parts of speech are assigned across languages (ex. Japanese vs. English)

Prerequisites

Before starting this tutorial, ensure that …

you are familiar with tokens and types
you are familiar with parts of speech

Background

This section provides a (very) brief linguistic overview of Japanese.

Language classification

Family: Japanese is a member of the Japonic languages family
Morphological typology: The Japanese language is agglutinative. Increasingly complex words are formed by combining morphemes. The elements of these new words largely retain their original forms and meanings.

Word order

Canonical word order in Japanese is SOV (subject - object - verb):

私は寿司を食べます

S	O	V
私は	寿司を	食べます
1SG-NOM	sushi-ACC	eat
¹

Written language

No whitespace between words used in writing
there is no clear word boundary like English (see the previous example)
Mixed writing system
kanji
- logographic system of approximately 50,000 characters
- Example: 食 in 食べます ('eat')
hiragana
- syllabary for native words; all kanji can also be written (ambiguously) in hiragana
- Example: べ (IPA: /be/), ま (/ma/), and す (/su/) in 食べます ('eat')
katakana
- syllabary primarily used for foreign terms, loanwords, onomatopaeic expressions, and emphasis
- Example: ア (/a/), イ (/i/), and ス (/su/) in アイス ('ice cream' /aisu/)

Encoding, fonts, and text direction

Computers store text as numbers. A character encoding is a convention for deciding which numbers correspond to which writing symbols. Older encodings often worked well for one language, one operating system, or one region, but they made multilingual text difficult to exchange reliably. Unicode is the modern standard that gives software a shared inventory of characters across writing systems.

When we talk about writing systems, it is useful to keep three related ideas separate:

Encoding: Unicode assigns abstract characters to code points. For example, 食 has a code point, as do あ, ア, and A.
Fonts and glyphs: A font decides how a character is drawn. A font can make the same encoded character look handwritten, printed, old-fashioned, or modern without changing the underlying text.
Layout and direction: Text is usually stored in logical reading order. Direction and visual layout are handled by a combination of Unicode character properties and higher-level layout systems such as HTML/CSS, word processors, terminals, or PDF renderers.

Japanese examples make these distinctions concrete. Japanese does not have the same uppercase/lowercase distinction used by Latin scripts, so operations like lower() and casefold() do not transform kanji, hiragana, or katakana in the way they transform English letters. Japanese also appears in multiple visual layouts: modern horizontal left-to-right text is common, but traditional vertical text is still widely used in books, signs, and historical materials. That visual layout is not the same thing as reversing the stored character order.

Python exposes some of these Unicode properties through unicodedata and string methods:

import unicodedata as ud
for character in "Aあア食":
    print(
        character,
        hex(ord(character)),
        ud.name(character),
        ud.category(character),
        ud.bidirectional(character),
    )
print("ABC".lower())
print("日本語".lower())

The details matter for NLP. Tokenization, normalization, case-folding, matching, sorting, and display are connected, but they are not the same operation. A pipeline that treats “character,” “glyph,” “token,” and “word” as interchangeable will eventually run into trouble.

How is it annotated?

There are a number of annotated corpora available for Japanese that include part of speech information. Let's examine how part of speech tags are assigned according to the universal dependencies annotation guidelines for the Japanese language…

Universal POS (UPOS) categorization

The Universal Dependency (UD) annotation guidelines for assigning part of speech tags for Japanese are described in the following paper:

Overview of Universal Dependencies for Japanese

POS Tag	Explanations
ADJ	Adjective
ADV	Adverb
INTJ	Interjection
NOUN	Noun (except for nouns that can be used as VERB or ADJ)
PROPN	Proper noun
VERB	Verb (ex. 食べ "eat", 食事 "meal")
ADP	Case particles (が ga),　adverbial particles (しか shika "only"), binding particles　（は wa）
AUX	Auxiliary verb (食べ'た' "ate")
CCONJ	Conjunctions (また"also")
DET	Adnominal (あの"that")
NUM	Number
PART	Sentence-ending particles (か, "?", ), Suffix (本格'的' "genuine")
PRON	Pronoun
SCONJ	Conjunction particles (行っ'て' "go, and […]"), particle that attaches to a phrase and acts on the whole phrase (行く'の' "going")
PUNCT	Punctuation
SYM	Any punctuation except for PUNCT and X
X	Blank

Example Sentence

蔵前国技館跡地は東京都に売却し、その売却益が…

Original	蔵前	国技	館	跡地	は	東京	都	に	売却	し	、	その	売却	益	が
Romanization	Kuramae	kokugi	kan	atochi	wa	Tokyo	to	ni	baikyaku	shi		sono	baikyaku	eki	ga
Gloss	Kuramae	national sport	building	site	(topic particle)	Tokyo	capital	(direction particle)	sell			that	selling	benefit	(object particle))
Tag	PROPN	NOUN	NOUN	NOUN	ADP	PROPN	NOUN	ADP	VERB	AUX	PUNCT	DET	NOUN	NOUN	ADP
Detailed tag	Proper noun	Common noun	Suffix	Common noun	binding particle	Proper noun	Common noun	case particle	Verb ("-suru" can be attached)	Verb (cannot stand alone)	Punctuation - comma	Adnominal	Noun ("-suru" can be attached)	Noun ("-suru" can be attached)	Particle (case particle)
sent_id = dev-s49

Comment

Note that the UD annotation scheme for japanese treats the tensed element of a verb (past tense marker ta) is treated as a separate token from the verb root (ex. 食べた ('ate') → ["食べ", "た"]).

Verbal nouns are nouns that function as verbs when followed by an auxiliary verb (e.g. する suru meaning "do"). The stems of verbal nouns are tagged as verbs. For example, 売却 baikyaku ('sale') is tagged as VERB in the first instance in the previous example. As the second instance of 売却 (baikyaku) shows it as NOUN. Adjectival verbs.

Text normalization

Japanese does not have an upper case / lower case distinction. Emphasis is conveyed through other means (ex. use of katakana). Example strategies for text normalization in Japanese include …

Spelling correction (standardization) and replacement of repetitions
すげえええええええええ！！！ ('awesomeeeeeeeee') $\rightarrow$ 　すごい！
Replacing rare kanji sequences with hiragana
The hiragana syllabary is a more compact vocabulary of symbols than kanji

Challenges

Tokenization challenges

No whitespace between words
Mixed writing system: Use of katakana syllabary for emphasis or other stylistic reasons can make segmentation difficult
agglutinative: long compounds and uncommon kanji can pose a challenge to segmentation strategies

POS tagging challenges

While certainly not unique to Japanese, ambiguity is a challenge for POS assignment. Instances of the same string may be assigned different POS tags in different contexts.

For example, the particle の no can be a case marker, sentence-final particle, or a nominalizer

Misc

NOTE: Though it highlights unique linguistics challenges, this section is not strictly related to the Unit and is therefore optional.

Digitization and handwriting recognition challenges

The digitization of historic Japanese texts faces numerous challenges.

In addition to having a mixed writing system, the orientation of written text in Japanese is highly variable.

Japanese text can appear…

left-to-right (LTR) horizontally

left to right horizontally — An example of horizontal LTR text from the Japanese Wikipedia page on the Japanese language²

right-to-left (RTL) and top to bottom (TTB) (ex. newspapers)

and even …

left-to-right horizontally (uncommon and old style) ³

Older styles of cursive script bear little resemblance to modern handwritten Japanese:

kuzushiji (old cursive style)

There is a vast trove of cultural information locked away in print media.

Next steps

Practice

Using the provided linguistic data, identify some differences in how tokens are defined/delimited in English and Japanese
What patterns do you notice (if any) for tokenizing Japanese text?
Using the provided linguistic data, identify some differences in how parts of speech are assigned in English and Japanese

Additional resources

Datasets

UD Japanese BCCWJ

Footnotes

The bottom row of this table uses https://www.eva.mpg.de/lingua/pdf/Glossing-Rules.pdf

'I eat sushi' ↩
See https://ja.wikipedia.org/wiki/%E6%97%A5%E6%9C%AC%E8%AA%9E ↩
From https://ima.goo.ne.jp/column/article/3948.html (now defunct) ↩