CharacterNormalize

✖
`CharacterNormalize`

[Experimental]

✖

CharacterNormalize["text",form]

converts the characters in text to the specified normalization form.

Details

CharacterNormalize supports the following Unicode normalization forms:

	"NFD"	canonical decomposition (Form D)
	"NFC"	canonical decomposition, followed by canonical composition (Form C)
	"NFKD"	compatibility decomposition (Form KD)
	"NFKC"	compatibility decomposition, followed by canonical composition (Form KC)

In CharacterNormalize[text,…], text can be a string or a list of strings.
In "NFD" and "NFC", canonical decomposition refers to these four type of operations:

	Å  Å,…	decompose marks
	Ȱ Ȱ,…	decompose and order marks
	한 한, …	decompose Hangul and conjoining Jamo
	Ω(Ohm) Ω (Omega),…	map character to its canonical Unicode equivalent

In "NFKD" and "NFKC", compatibility decomposition refers to operations such as:

	ℌH ,ℍH,…	normalize font variants
	(NBSP)(Space), …	normalize linebreaking differences
	ﻉ ع,ﻊ ع, …	normalize positional variants
	①1, …	normalize circled variants
	ｶカ, …	normalize width variants
	︷{ ,︸} , …	normalize rotated variants
	i⁹ i9,i₉ i9, …	normalize subscripts/superscripts
	㌀アパート, …	decompose squared characters
	¼ 1/4 , …	normalize fractions
	ǆ→dž, …	other normalizations

Examples

open allclose all

Basic Examples (5)Summary of the most common use cases

Normalize string characters using canonical decomposition:

In[1]:=1

✖

https://wolfram.com/xid/05fcn43fok-f3153s

Out[1]=1

Normalize string characters using compatibility decomposition:

In[1]:=1

✖

https://wolfram.com/xid/05fcn43fok-py8wko

Out[1]=1

Normalize string characters using compatibility decomposition followed by canonical composition:

In[1]:=1

✖

https://wolfram.com/xid/05fcn43fok-sd1qqy

Out[1]=1

Normalize string characters using canonical decomposition followed by canonical composition:

In[1]:=1

✖

https://wolfram.com/xid/05fcn43fok-d8jb7v

Out[1]=1

Normalize the characters in the string using compatibility decomposition:

In[1]:=1

✖

https://wolfram.com/xid/05fcn43fok-wwcsrq

Out[1]=1

Characters with diacritics have been decomposed:

In[2]:=2

✖

https://wolfram.com/xid/05fcn43fok-820s9y

Out[2]=2

Scope (2)Survey of the scope of standard use cases

Decompose a composite character into its constituents:

In[1]:=1

✖

https://wolfram.com/xid/05fcn43fok-9a8snb

Out[1]=1

Ordering of the mark and the character has changed after normalization:

In[2]:=2

✖

https://wolfram.com/xid/05fcn43fok-3a8r0h

Out[2]=2

Obtain the "Ohm" character from its code:

In[1]:=1

✖

https://wolfram.com/xid/05fcn43fok-kwo17l

Out[1]=1

NFD maps characters to their canonically equivalent Unicode. Normalize the character using NFD:

In[2]:=2

✖

https://wolfram.com/xid/05fcn43fok-qdm8iq

Out[2]=2

Convert the output (omega) to its character code:

In[3]:=3

✖

https://wolfram.com/xid/05fcn43fok-bnul0u

Out[3]=3

Generalizations & Extensions (1)Generalized and extended use cases

CharacterNormalize threads itself elementwise over lists:

In[1]:=1

✖

https://wolfram.com/xid/05fcn43fok-5xvlby

Out[1]=1

CharacterNormalize works on strings of different scripts and letters:

In[2]:=2

✖

https://wolfram.com/xid/05fcn43fok-no7g73

Out[2]=2

Possible Issues (1)Common pitfalls and unexpected behavior

Compatibility equivalence may convert different forms of a character to a canonical form:

In[1]:=1

✖

https://wolfram.com/xid/05fcn43fok-56axxd

Out[1]=1

In[2]:=2

✖

https://wolfram.com/xid/05fcn43fok-ee364p

Out[2]=2

Compatibility equivalence may remove formatting distinctions that are not changed in canonical equivalent characters:

In[3]:=3

✖

https://wolfram.com/xid/05fcn43fok-2ikh48

Out[3]=3

In[4]:=4

✖

https://wolfram.com/xid/05fcn43fok-cd1ss9

Out[4]=4

Top